Urban water disaster entity identification method

Abstract

The invention discloses an urban water disaster entity identification method, which belongs to the technical field of natural language processing and computer vision crossing, and comprises the following steps: fusing a to-be-identified original water disaster text with an original water disaster image; the to-be-recognized multi-modal fusion features are input into a pre-trained water disaster entity recognition model for sequence decoding, and an urban water disaster entity is obtained; the pre-trained water disaster entity recognition model is obtained through the following steps: obtaining a plurality of texts based on an original water disaster text through a large language model, and obtaining a plurality of images based on the original water disaster text through a diffusion model; obtaining text features and visual features based on the text and the image, fusing the text features and the visual features into historical multi-modal fusion features, inputting the historical multi-modal fusion features into a water disaster entity recognition model, and training the water disaster entity recognition model by taking minimization of a total loss function as a target to obtain a trained water disaster entity recognition model; and the identification precision of urban water disasters on entities such as places, time, disasters and the like in complex scenes is improved.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of natural language processing and computer vision, and in particular to a method for recognizing entities in urban water disasters. Background Technology

[0002] With the intensification of global climate change, urban flooding and torrential rains are becoming increasingly frequent, posing severe challenges to urban public safety and emergency management. Named entity recognition (NAME) for urban flood disasters, as a core component of disaster information processing, can accurately extract key information such as time, location, and disaster details from events, providing crucial data support for disaster early warning, emergency response, and post-disaster reconstruction. Therefore, efficient and accurate entity recognition technology for flood disasters has become a research hotspot in the field of urban emergency management.

[0003] Currently, methods for identifying entities in urban flood disasters are mainly divided into two categories: pure text recognition and simple multimodal fusion. Pure text recognition methods rely on pre-trained language models to perform semantic analysis on disaster texts, and complete the recognition task by designing specific entity annotation rules or training sequence annotation models. For example, Peng et al. used the BERT-BiLSTM-CRF model to identify the location, time, and degree of flooding in flooded areas, and Yan et al. also used the BERT model to effectively extract the fine location and water level information of urban flood disasters. However, these methods only utilize text information and are difficult to deal with complex scenarios such as ambiguous text descriptions and semantic ambiguities.

[0004] To compensate for the shortcomings of pure text recognition, some scholars have proposed simple image-text multimodal fusion methods, which concatenate image features with text features before inputting them into the model. In the field of multimodal entity recognition, some scholars have explored the idea of ​​visual cue injection. For example, Zhang et al. proposed a named entity recognition model based on entity-related visual cue injection in the field of agricultural diseases. However, the application of such methods in urban flood disaster scenarios still has significant shortcomings: on the one hand, direct feature concatenation fails to establish a deep semantic association between text and visual information, and is prone to introducing scene noise, such as information about passersby and unaffected buildings in disaster scene photos, which interferes with the accuracy of entity recognition; on the other hand, existing methods lack targeted processing mechanisms for visual features, and cannot convert visual information into semantic cues that can be efficiently used by the text model, resulting in the failure to fully utilize visual features to assist text understanding. This is consistent with the problem of "insufficient effectiveness of feature fusion" found by Wang Zhenyu et al. in their research on multimodal named entity recognition in social media.

[0005] Furthermore, existing multimodal recognition methods do not fully utilize the knowledge reasoning capabilities of Large Language Models (LLMs), making it difficult to extract hierarchical semantic information from raw text, further limiting the model's recognition performance in complex disaster scenarios. While domestic and international scholars have conducted extensive research in named entity recognition and multimodal fusion—for example, Liu et al. proposed a multi-granularity cross-modal model to improve social media entity recognition, and Tian et al. designed a hierarchical adaptive network to optimize multimodal entity recognition performance—a comprehensive entity recognition method that balances deep semantic mining and efficient cross-modal fusion remains to be perfected for the specific scenario of urban flood disasters. How to accurately utilize visual information, effectively extract hierarchical semantics, and achieve deep interaction of multimodal features has become a key issue in improving the accuracy of entity recognition in urban flood disasters. Summary of the Invention

[0006] The purpose of this invention is to provide a method for identifying entities in urban flood disasters. This method utilizes a large language model to mine hierarchical semantic information, a diffusion model to generate semantically aligned images, and a visual cueing mechanism to achieve cross-modal deep fusion. Combined with a multi-granularity contrastive learning strategy and multi-loss function training, it solves the problems of insufficient semantic information in pure text recognition, easy introduction of noise through simple image-text splicing, and inadequate utilization of visual information in existing urban flood disaster entity identification methods. This invention is achieved through the following technical solutions.

[0007] This invention provides a method for identifying entities in urban flood disasters, comprising:

[0008] Obtain the original text of the water disaster to be identified and its corresponding original image of the water disaster;

[0009] The original text of the flood disaster to be identified is fused with its corresponding original image to obtain the multimodal fusion feature to be identified. This multimodal fusion feature is then input into a pre-trained flood disaster entity recognition model for sequence decoding to obtain urban flood disaster entities. The pre-trained flood disaster entity recognition model is obtained through the following steps:

[0010] The system extracts water disaster entity text from the original water disaster text using a large language model, generates entity explanations and sentiment information based on the original water disaster text, and generates hierarchical text based on the original water disaster text, entity explanations and sentiment information.

[0011] Sentence images and entity images are generated based on the original water disaster text and water disaster entity text, respectively, using a diffusion model.

[0012] Text features are obtained by encoding the original flood disaster text, flood disaster entity text, and hierarchical text using a text encoder; visual features are obtained by encoding the original flood disaster image, sentence image, and entity image using a visual encoder.

[0013] By fusing visual features with textual features, historical multimodal fusion features are obtained;

[0014] Historical multimodal fusion features are input into the water disaster entity recognition model. The model is trained with the goal of minimizing the total loss function, resulting in a trained water disaster entity recognition model.

[0015] In practical applications, this invention first obtains a trained flood disaster entity recognition model: Data preprocessing and semantic enhancement are performed. Upon receiving text and image data (original flood disaster text and original flood disaster images), entity information is extracted using a Large Language Model (LLM), generating entity explanations and sentiment information, which are then concatenated to form hierarchical text. An enhanced visual representation auxiliary image (sentence image and entity image) is generated through a diffusion model. Text and visual features are then obtained using a text encoder (BERT) and a visual encoder (CLIP). Subsequently, the visual features are converted into key-value pairs adapted to the BERT encoder and injected into the cross-attention layer of the BERT encoder to achieve deep feature interaction, resulting in multimodal fusion features. By employing a multi-granularity contrast and matching learning strategy, and using contrast and matching losses to construct a multi-task loss function, combined with multimodal fusion features, the recognition accuracy and generalization ability of the pre-trained flood disaster entity recognition model are improved, resulting in a trained flood disaster entity recognition model. Then, the original water disaster text to be identified and its corresponding original water disaster image are obtained. Following the above steps, the multimodal fusion features to be identified are obtained and input into the trained water disaster entity recognition model for sequence decoding and prediction. The optimal label sequence can be obtained and the water disaster entities such as time, location and disaster situation are output to complete the recognition task.

[0016] Optionally, the step of extracting flood disaster entity text from the original flood disaster text using a large language model is performed by extracting it using entity prompt templates of the large language model, and the step of generating entity explanations and sentiment information based on the flood disaster entity text is performed by generating them using entity explanation prompt templates and sentiment prompt templates of the large language model, respectively.

[0017] Optionally, the original flood disaster text, flood disaster entity text, and hierarchical text are encoded using a text encoder to obtain the following expression for the text features:

[0018] ,

[0019] ,

[0020] ,

[0021] ,

[0022] In the formula, Represents the original water disaster text Or water disaster entity text or hierarchical text , Indicates text encoder Text encoding operations, For text encoders Learnable parameters , and Text encoders Original water disaster text Flood disaster entity text Hierarchical text Encoded text features For batch size, The length of the text sequence. For feature dimensions.

[0023] Optionally, the original flood disaster image, sentence image, and entity image are encoded using a visual encoder to obtain the following expression for the visual features:

[0024] ,

[0025] ,

[0026] ,

[0027] ,

[0028] In the formula, Represents the original water disaster image or physical image or sentence image , Indicates a visual encoder Visual encoding operations, For visual encoders Learnable parameters , , Visual encoders Original water disaster images Entity-to-image generation Sentence Images The encoded visual features.

[0029] Optionally, the multimodal fusion features are calculated using the following formula:

[0030] ,

[0031] In the formula, For multimodal fusion features, Adapting text encoders to visual features Key-value pairs in the attention mechanism The mask for multimodal fusion features is calculated using the following formula:

[0032] ,

[0033] In the formula, An attention mask for visual features. Attention mask for text features, This is for mask concatenation operations.

[0034] Optional, visual feature adaptation text encoder Key-value pairs in attention mechanisms It is obtained by calculation using the following formula:

[0035] ,

[0036] In the formula, This is a splitting operation, indicating that visual cues will be... According to feature dimensions Split into an adaptive text encoder Key-value pairs in attention mechanisms Among them, visual cues It is obtained by calculation using the following formula:

[0037] ,

[0038] In the formula, Flattening operation for visual features and For linear operations, This is the activation function.

[0039] Optionally, the total loss function is calculated using the following formula:

[0040] ,

[0041] In the formula, For the total loss function, For conditional random fields, the negative log-likelihood principal loss is used. For text-image matching loss, The global contrast loss is calculated between the original flood disaster text and the original flood disaster image. For fine-grained contrast loss between water disaster entity text and entity images, This is a fine-grained contrast loss between the original flood disaster text and the sentence image.

[0042] Optionally, negative log-likelihood principal loss for conditional random fields. It is obtained by calculation using the following formula:

[0043] ,

[0044] In the formula, This represents the sequence number of the original flood disaster text or multimodal feature, ranging from 1 to B, where B is the total number of original flood disaster texts. For the first A sequence of real labels from original flood disaster text. For the first A multimodal fusion feature, For the learnable parameters of the conditional random field layer, For the first Attention mask for original water disaster text features,

[0045] Text-image matching loss It is obtained by calculation using the following formula:

[0046] ,

[0047] In the formula, The pooling result is for multimodal fusion features. Pooling features for sampled mismatched text-image pairs, Let cross-entropy be the loss function. To match the judgment header, To match tags, =1 indicates a match. =0 indicates no match.

[0048] Optionally, a global contrast loss between the original flood disaster text and the original flood disaster image. It is obtained by calculation using the following formula:

[0049] ,

[0050] In the formula, The projection result of text features after encoding the original water disaster text. The projection result of the visual features encoded from the original flood disaster image. To compare the temperature coefficient of loss, For feature projection,

[0051] Fine-grained contrast loss between flood disaster entity text and entity images It is obtained by calculation using the following formula:

[0052] ,

[0053] In the formula, The projection result of text features after encoding water disaster entity text. The projection result of the visual features encoded in the entity image.

[0054] Fine-grained contrast loss between original flood disaster text and sentence images It is obtained by calculation using the following formula:

[0055] ,

[0056] In the formula, The projection result of the visual features after the sentence image is encoded.

[0057] Optionally, the urban flood disaster entity is calculated using the following formula:

[0058] ,

[0059] In the formula, For predicted urban flood disaster entity label sequences, This is a sequence decoding operation.

[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0061] The urban flood disaster entity recognition method provided by this invention extracts semantic information such as entity generation, entity explanation, and sentiment through a large language model to construct hierarchical text, and uses a diffusion model to generate corresponding images based on the original text and entity text, which can enhance semantic features and visual representation auxiliary images; by injecting visual features into the text editor in the form of visual cues. In this way, the fusion of text features and visual features is achieved to obtain multimodal fusion features, which achieves cross-modal fusion more efficiently than traditional feature splicing. By constructing a multi-objective loss function and training the water disaster entity recognition model with the goal of minimizing the multi-objective loss function, irrelevant background noise in social media images can be effectively suppressed, the semantic alignment of text and images at different granularities and the robustness of the model can be strengthened, and the entity recognition accuracy in complex urban water disaster scenarios (such as in scenarios with ambiguous text descriptions and complex image backgrounds) can be improved. Attached Figure Description

[0062] Figure 1 The diagram shown is a flowchart of an urban flood disaster entity identification method according to one embodiment of the present invention.

[0063] Figure 2 The figure shown is a schematic diagram of entity output in one embodiment of the present invention;

[0064] Figure 3 The diagram shown is a schematic representation of entity interpretation output in one embodiment of the present invention;

[0065] Figure 4 The diagram shown is a schematic representation of the emotional information output in one embodiment of the present invention;

[0066] Figure 5 The diagram shown is a schematic diagram of the extended semantics of a large language model in one embodiment of the present invention. Detailed Implementation

[0067] The following description, in conjunction with the accompanying drawings and specific embodiments, provides further details. In this description, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature.

[0068] Example 1

[0069] This embodiment introduces a method for identifying entities in urban flood disasters, such as... Figure 1 As shown,

[0070] Obtain the original text of the water disaster to be identified and its corresponding original image of the water disaster;

[0071] The original text of the flood disaster to be identified is fused with its corresponding original image to obtain the multimodal fusion feature to be identified. This multimodal fusion feature is then input into a pre-trained flood disaster entity recognition model for sequence decoding to obtain urban flood disaster entities. The pre-trained flood disaster entity recognition model is obtained through the following steps:

[0072] The system extracts water disaster entity text from the original water disaster text using a large language model, generates entity explanations and sentiment information based on the original water disaster text, and generates hierarchical text based on the original water disaster text, entity explanations and sentiment information.

[0073] Sentence images and entity images are generated based on the original water disaster text and water disaster entity text, respectively, using a diffusion model.

[0074] Text features are obtained by encoding the original flood disaster text, flood disaster entity text, and hierarchical text using a text encoder; visual features are obtained by encoding the original flood disaster image, sentence image, and entity image using a visual encoder.

[0075] By fusing visual features with textual features, historical multimodal fusion features are obtained;

[0076] Historical multimodal fusion features are input into the water disaster entity recognition model. The model is trained with the goal of minimizing the total loss function, resulting in a trained water disaster entity recognition model.

[0077] Example 2

[0078] Based on Example 1, this example describes the specific implementation process of a method for identifying entities in urban flood disasters, including the following:

[0079] In one specific embodiment of the present invention, the extraction of flood disaster entity text from the original flood disaster text using a large language model is performed through entity prompt templates of the large language model, such as... Figure 2 As shown, the entity hint template rigorously annotates the input text with three entities: Time, Location, and Flood, and outputs the results, helping readers quickly understand the severity of urban flooding. The generation of entity explanations and sentiment information based on flood disaster entity text is achieved through entity explanation hint templates and sentiment hint templates from a large language model, respectively, as shown below. Figure 3 and Figure 4 As shown. Figure 3 In this context, the entity explanation prompt template first extracts entities from the input text, then explains and outputs the entities, which can help readers obtain a clearer and more accurate understanding of the extent of urban flooding. Figure 4 The system performs sentiment analysis on the input text and outputs corresponding sentiment tags to further help readers understand the extent of urban flooding.

[0080] Figure 5 A schematic diagram for expanding semantics in a large language model. Figure 5 This paper describes the diffusion of original water disaster text through a large language model, and the diffusion of water disaster images through the diffusion model, which enables readers to obtain a clearer, more intuitive and accurate picture of urban water disaster damage. Figure 5 The original text diffusion generates the sentence image as described in this invention, and the entity diffusion generates the entity image as described in this invention. In this embodiment, the diffusion model used is AltDiffusion-m18.

[0081] I. Acquisition of Multimodal Fusion Features

[0082] 1.1 Acquisition of Text Features

[0083] In one specific embodiment of the present invention, the original flood disaster text, flood disaster entity text, and hierarchical text are encoded by a text encoder to obtain the following expression of text features:

[0084] ,

[0085] ,

[0086] ,

[0087] ,

[0088] In the formula, Represents the original water disaster text Or water disaster entity text or hierarchical text , Indicates text encoder Text encoding operations, For text encoders Learnable parameters , and Text encoders Original water disaster text Flood disaster entity text Hierarchical text Encoded text features For batch size, The length of the text sequence. For feature dimensions.

[0089] 1.2 Acquisition of visual features

[0090] In one specific embodiment of the present invention, the original flood disaster image, sentence image, and entity image are encoded by a visual encoder to obtain the following expression of visual features:

[0091] ,

[0092] ,

[0093] ,

[0094] ,

[0095] In the formula, Represents the original water disaster image or physical image or sentence image , Indicates a visual encoder Visual encoding operations, For visual encoders Learnable parameters , , Visual encoders Original water disaster images Entity-to-image generation Sentence Images The encoded visual features. That is, the encoder. The extracted visual features are multi-level visual features.

[0096] 1.3 Obtaining Multimodal Fusion Features

[0097] In one specific embodiment of the present invention, in order to achieve deep fusion of visual features and text features, a hierarchical visual cue mechanism is constructed. First, the visual encoder... The extracted multi-level visual features are used to generate visual cues through linear transformation. Then visual cues According to feature dimensions Split to get a compatible text editor Key-value pairs in attention mechanisms Then key-value pairs Inject text editor Multimodal fusion characteristics are obtained through cross-attention.

[0098] Visual cues It is obtained by calculation using the following formula:

[0099] ,

[0100] In the formula, Flattening operation for visual features and For linear operations, This is the activation function.

[0101] Visual feature adaptation text encoder Key-value pairs in attention mechanisms It is obtained by calculation using the following formula:

[0102] ,

[0103] In the formula, This is a splitting operation, indicating that visual cues will be... According to feature dimensions Split into an adaptive text encoder Key-value pairs in attention mechanisms ,

[0104] Multimodal fusion features are calculated using the following formula:

[0105] ,

[0106] In the formula, For multimodal fusion features, Adapting text encoders to visual features Key-value pairs in the attention mechanism The mask for multimodal fusion features is calculated using the following formula:

[0107] ,

[0108] In the formula, An attention mask for visual features. Attention mask for text features, This is for mask concatenation operations.

[0109] II. Loss Function

[0110] In one specific embodiment of the present invention, to improve the recognition accuracy and generalization ability of the pre-trained flood disaster entity recognition model, a multi-granularity contrast and matching learning strategy is proposed, and a training strategy of constructing a multi-task loss weighted summation function is constructed. This loss function includes four dimensions of contrast loss:

[0111] Global contrast loss ( By narrowing the feature distance between the original flood disaster text and the original flood disaster image, alignment of the original flood disaster text and the original flood disaster image is achieved; alignment means that the image can accurately reflect the information of the text.

[0112] Entity-level contrast loss ( By narrowing the feature distance between the extracted water disaster entity text and the entity image generated based on the water disaster entity text, fine-grained alignment of water disaster entity text and entity generated image is achieved.

[0113] Sentence-level contrast loss ( By narrowing the feature distance between the original flood disaster text and the sentence image generated based on the original flood disaster text, fine-grained alignment of the original flood disaster text and sentence image is achieved, and the semantic consistency between modalities is enhanced by text-image matching loss.

[0114] Image-text matching loss ( It determines whether the image and text match, thereby suppressing irrelevant visual noise.

[0115] The total loss function is calculated using the following formula:

[0116] ,

[0117] In the formula, For the total loss function, For conditional random fields, the negative log-likelihood principal loss is used. For text-image matching loss, The global contrast loss is calculated between the original flood disaster text and the original flood disaster image. For fine-grained contrast loss between water disaster entity text and entity images, This is a fine-grained contrast loss between the original flood disaster text and the sentence image.

[0118] Negative log-likelihood principal loss in conditional random fields It is obtained by calculation using the following formula:

[0119] ,

[0120] In the formula, This represents the sequence number of the original flood disaster text or multimodal feature, ranging from 1 to B, where B is the total number of original flood disaster texts. For the first A sequence of real labels from original flood disaster text. For the first A multimodal fusion feature, For the learnable parameters of the conditional random field layer, For the first Attention mask for original water disaster text features,

[0121] Text-image matching loss It is obtained by calculation using the following formula:

[0122] ,

[0123] In the formula, The pooling result is for multimodal fusion features. Pooling features for sampled mismatched text-image pairs, Let cross-entropy be the loss function. To match the judgment header, To match tags, =1 indicates a match. =0 indicates no match.

[0124] Global contrast loss between original flood disaster text and original flood disaster image It is obtained by calculation using the following formula:

[0125] ,

[0126] In the formula, The projection result of text features after encoding the original water disaster text. The projection result of the visual features encoded from the original flood disaster image. To compare the temperature coefficient of loss, For feature projection,

[0127] Fine-grained contrast loss between flood disaster entity text and entity images It is obtained by calculation using the following formula:

[0128] ,

[0129] In the formula, The projection result of text features after encoding water disaster entity text. The projection result of the visual features encoded in the entity image.

[0130] Fine-grained contrast loss between original flood disaster text and sentence images It is obtained by calculation using the following formula:

[0131] ,

[0132] In the formula, The projection result of the visual features after the sentence image is encoded.

[0133] III. Obtaining the physical entity of urban flood disaster

[0134] Multimodal fusion features, which combine textual and visual features, are input into a pre-trained flood disaster entity recognition model. Specifically, these features are input into the conditional random field layer of the pre-trained flood disaster entity recognition model. ,pass The system decodes and predicts the multimodal fusion features to obtain the optimal label sequence and outputs the final water disaster entity, including time, location, water depth, etc., thus completing the multimodal entity recognition task.

[0135] The urban flood disaster entity is calculated using the following formula:

[0136] ,

[0137] In the formula, For predicted urban flood disaster entity label sequences, This is a sequence decoding operation.

[0138] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for urban water disaster entity recognition, characterized in that, include: Obtain the original text of the water disaster to be identified and its corresponding original image of the water disaster; The original text of the flood disaster to be identified is fused with its corresponding original image to obtain the multimodal fusion feature to be identified. This multimodal fusion feature is then input into a pre-trained flood disaster entity recognition model for sequence decoding to obtain urban flood disaster entities. The pre-trained flood disaster entity recognition model is obtained through the following steps: The system extracts water disaster entity text from the original water disaster text using a large language model, generates entity explanations and sentiment information based on the original water disaster text, and generates hierarchical text based on the original water disaster text, water disaster entity text, entity explanations and sentiment information. Sentence images and entity images are generated based on the original water disaster text and water disaster entity text, respectively, using a diffusion model. Text features are obtained by encoding the original flood disaster text, flood disaster entity text, and hierarchical text using a text encoder; visual features are obtained by encoding the original flood disaster image, sentence image, and entity image using a visual encoder. By fusing visual features with textual features, historical multimodal fusion features are obtained; Historical multimodal fusion features are input into the water disaster entity recognition model. The model is trained with the goal of minimizing the total loss function, resulting in a trained water disaster entity recognition model.

2. The urban water disaster entity recognition method of claim 1, wherein, The extraction of flood disaster entity text from the original flood disaster text using a large language model is performed by extracting it using entity prompt templates from the large language model. The generation of entity explanations and sentiment information based on the flood disaster entity text is performed by generating them using entity explanation prompt templates and sentiment prompt templates from the large language model, respectively.

3. The urban water disaster entity recognition method of claim 1, wherein, The original flood disaster text, flood disaster entity text, and hierarchical text are encoded using a text encoder, resulting in the following expression for text features: ， ， ， ， In the formula, Represents the original water disaster text Or water disaster entity text or hierarchical text , Indicates text encoder Text encoding operations, For text encoders Learnable parameters, , and Text encoders Original water disaster text Flood disaster entity text Hierarchical text Encoded text features For batch size, The length of the text sequence. For feature dimensions.

4. The urban flood disaster entity identification method according to claim 3, characterized in that, The original flood disaster image, sentence image, and entity image are encoded using a visual encoder, resulting in the following expression for the visual features: ， ， ， ， In the formula, Represents the original water disaster image or physical image or sentence image , Indicates a visual encoder Visual encoding operations, For visual encoders Learnable parameters, , , Visual encoders Original water disaster images Entity-to-image generation Sentence Images The encoded visual features.

5. The urban flood disaster entity identification method according to claim 4, characterized in that, Multimodal fusion features are calculated using the following formula: ， In the formula, For multimodal fusion features, Adapting text encoders to visual features Key-value pairs in the attention mechanism The mask for multimodal fusion features is calculated using the following formula: ， In the formula, An attention mask for visual features. Attention mask for text features, This is for mask concatenation operations.

6. The method for identifying urban flood disaster entities according to claim 5, characterized in that, Visual feature adaptation text encoder Key-value pairs in attention mechanisms It is obtained by calculation using the following formula: ， In the formula, This is a splitting operation, indicating that visual cues will be... According to feature dimensions Split into an adaptive text encoder Key-value pairs in attention mechanisms Among them, visual cues It is obtained by calculation using the following formula: ， In the formula, Flattening operation for visual features and For linear operations, This is the activation function.

7. The urban flood disaster entity identification method according to claim 6, characterized in that, The total loss function is calculated using the following formula: ， In the formula, For the total loss function, For conditional random fields, the negative log-likelihood principal loss is used. For text-image matching loss, The global contrast loss is calculated between the original flood disaster text and the original flood disaster image. For fine-grained contrast loss between water disaster entity text and entity images, This is a fine-grained contrast loss between the original flood disaster text and the sentence image.

8. The method for identifying urban flood disaster entities according to claim 7, characterized in that, Negative log-likelihood principal loss in conditional random fields It is obtained by calculation using the following formula: ， In the formula, This represents the sequence number of the original flood disaster text or multimodal feature, ranging from 1 to B, where B is the total number of original flood disaster texts. For the first A sequence of real labels from original flood disaster text. For the first A multimodal fusion feature, For the learnable parameters of the conditional random field layer, For the first Attention mask for original water disaster text features, Text-image matching loss It is obtained by calculation using the following formula: ， In the formula, The pooling result is for multimodal fusion features. Pooling features for sampled mismatched text-image pairs, Let cross-entropy be the loss function. To match the judgment header, To match tags, =1 indicates a match. =0 indicates no match.

9. The method for identifying urban flood disaster entities according to claim 8, characterized in that, Global contrast loss between original flood disaster text and original flood disaster image It is obtained by calculation using the following formula: ， In the formula, The projection result of text features after encoding the original water disaster text. The projection result of the visual features encoded from the original flood disaster image. To compare the temperature coefficient of loss, For feature projection, Fine-grained contrast loss between flood disaster entity text and entity images It is obtained by calculation using the following formula: ， In the formula, The projection result of text features after encoding water disaster entity text. The projection result of the visual features encoded in the entity image. Fine-grained contrast loss between original flood disaster text and sentence images It is obtained by calculation using the following formula: ， In the formula, The projection result of the visual features after the sentence image is encoded.

10. The method for identifying urban flood disaster entities according to claim 9, characterized in that, The physical structure of urban flood hazards is calculated using the following formula: ， In the formula, For predicted urban flood disaster entity label sequences, This is a sequence decoding operation.

Images