A method for constructing an urban waterlogging disaster state detection model and application thereof

Abstract

The application discloses a kind of urban waterlogging disaster state detection model construction method and application, belong to computer vision technical field;Firstly, the high-dimensional visual features of city image in training sample pair are extracted by image coding module, i.e.image coding features, the embedding features of the real text describing the waterlogging disaster state in the city image are extracted by embedding layer;Then, the image coding features and text embedding features are fused based on cross attention mechanism in image decoding module, the association between image and text is improved, the deep interaction between different modalities is promoted, so that the visual modal information and text modal information are effectively fused in feature space, the fine-grained information of semantic level is enriched, the understanding ability of complex flood scene of urban waterlogging disaster state detection model is improved, and accurate waterlogging disaster state detection can be carried out on complex urban scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and more particularly relates to a construction method and application of a city waterlogging disaster state detection model. BACKGROUND

[0002] In recent years, the global range of adverse weather phenomena is growing, and short-term heavy rainfall, continuous rainfall and other events occur. Due to a large number of buildings and concrete surfaces in urban areas, the complexity of the terrain is greatly increased, which makes the drainage capacity of the area insufficient, and frequent natural disasters such as floods and floods occur, causing significant property losses and life safety to humans. City waterlogging is relatively common in China. From the perspective of the region where it occurs, some low-lying coastal areas and inland cities with heavy rainfall are prone to city waterlogging, and the incidence of some specific locations is higher, such as overpasses, underground passages, railway bridges, and highway bridges. Therefore, when a disaster occurs, quickly and accurately finding the waterlogging area and determining its disaster severity is of great help and significance to effectively arrange and implement rescue and disaster relief work.

[0003] At present, city waterlogging disaster state detection mainly has the following ways: waterlogging disaster state detection based on water level meter measurement, waterlogging disaster state detection based on remote sensing image, waterlogging disaster state detection based on unmanned aerial vehicle cruising, and waterlogging disaster state detection based on monitoring camera. Compared with other ways, waterlogging disaster state detection based on monitoring camera shows significant advantages. The cost of monitoring camera is relatively low, the deployment process is simple, and the application range in the city is wide, which improves the coverage and real-time performance of waterlogging disaster state detection. In the existing waterlogging disaster state detection method based on monitoring camera, in order to more widely estimate the water level, some studies use reference objects to indirectly judge the severity of waterlogging. For example, Chaudhary et al. use semantic segmentation to identify and segment flood, people, cars, buses, bicycles and houses, divide the flood risk into 11 levels, and label the reference objects according to the level division standard to estimate the water level. Although these methods have shown excellent performance, these methods still have some defects in analyzing the deep semantics of complex scenes, and it is difficult to comprehensively understand the scene. For example, although the waterlogging recognition method can recognize water bodies, it cannot distinguish between waterlogging and natural water areas such as ponds, lakes and rivers in specific situations, and the algorithm lacks comprehensive understanding of the image background and context; for another example, the relationship between reference objects of different sizes (such as cars, SUVs, buses or trucks) and water levels is different, and the height difference between adults and children also affects the accuracy of waterlogging disaster state detection; in addition, in specific application scenarios, people may be engaged in water activities in non-waterlogging situations, such as recreational activities in ponds or lakes, although these scenes contain water bodies, they do not constitute a waterlogging risk. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a method for constructing and applying an urban flood disaster detection model, so as to solve the technical problem that the existing technology cannot accurately detect the flood disaster status in complex urban scenarios.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for constructing an urban flooding disaster status detection model, comprising:

[0006] Obtain a training sample set; the training sample set includes multiple training sample pairs, each of which includes a city image and corresponding real text describing the state of urban flooding.

[0007] Construct a multimodal model; the multimodal model includes: an image feature extraction module, an image encoding module, a tokenization layer, an embedding layer, an image decoding module, and a mapping module; the image encoding module includes a Transformer encoder; the image decoding module includes a Transformer decoder;

[0008] The training sample pairs are input into the multimodal model. The city images are sequentially processed through an image feature extraction module and an image encoding module to obtain image encoded features; the real text is sequentially processed through a tokenization layer and an embedding layer to obtain text embedding features {e1,e2,…,e m}; where e l Let l be the embedding feature of the l-th token in the real text, where l = 1, 2, ..., m; insert a preset start symbol feature e0 before the text embedding feature; at the (j+1)-th time step, combine the image encoding feature with the feature {e0, e1, e2, ..., e...} j The input is fused into the image decoding module to obtain the corresponding modality fusion features, and then the corresponding time step text output is obtained through the mapping module; after m+1 time steps, the reconstructed text describing the state of urban flooding is obtained; j = 0, 1, ..., m; m is the total number of tokens in the real text;

[0009] The multimodal model is trained by minimizing the difference loss between the reconstructed text and the corresponding real text.

[0010] After training, an urban flooding disaster status detection model is constructed, which includes a cascaded image feature extraction module, image encoding module, image decoding module, and mapping module.

[0011] More preferably, the above-mentioned multimodal model further includes: a pre-trained language model and a text decoding module; the text decoding module includes a Transformer decoder;

[0012] The text features are input into a pre-trained language model to obtain text features; at the j+1 time step, the text features and the features {e0, e1, e2, …, e j} are also input into a text decoding module for fusion to obtain corresponding text fusion features, and then the mapping module is used to obtain the character output at the corresponding time step; after m+1 time steps, the generated text describing the urban waterlogging disaster state is obtained.

[0013] When training the multi-modal model, the difference loss between the generated text and the corresponding real text is also minimized.

[0014] Further preferably, when training the multi-modal model, the similarity between the image encoding features of the city image and the text features of the corresponding real text is minimized, the similarity between the image encoding features of the city image and the text features of the non-corresponding real text is maximized, the similarity between the text features of the real text and the image encoding features of the corresponding city image is minimized, and the similarity between the text features of the real text and the image encoding features of the non-corresponding city image is maximized.

[0015] Further preferably, when the image encoding module is a plurality of cascaded Transformer encoders, the image encoding features used when training the multi-modal model are the image encoding features output by the last level Transformer encoder of the image encoding module, or the image encoding features output by the last level Transformer encoder and one or more other Transformer encoders of the image encoding module.

[0016] Further preferably, the image encoding module comprises a plurality of cascaded Transformer encoders; the image decoding module comprises a plurality of cascaded Transformer decoders;

[0017] The input of the rth level Transformer encoder is the image encoding features output by the (r-1)th level Transformer encoder; r=2, 3, …, R; R is the total number of Transformer encoders; the input of the first level Transformer encoder is the features extracted by the image feature extraction module from the city image;

[0018] The input of the pth stage Transformer decoder is the modality fusion feature output by the (p-1)th stage Transformer decoder and the image encoding feature output by the Rth stage Transformer encoder; p=2, 3, …, P; P is the total number of the Transformer decoder; the input of the first stage Transformer decoder is a preset start symbol feature and the image encoding feature output by the Rth stage Transformer encoder.

[0019] Further preferably, the real text describing the urban waterlogging disaster state comprises: relationship information between a reference object and a water level of the urban image; wherein the relationship information between the reference object and the water level is represented by a water level grade; the reference object comprises: a person and / or a vehicle;

[0020] For the reference object person, when the person is not in the water area, the water level grade is marked as the 0th grade; when the person is in the water area and the water level line is below the knee of the person, the water level grade is marked as the 1st grade; when the water level line is above the knee and below the waist of the person, the water level grade is marked as the 2nd grade; when the water level line is above the waist of the person, the water level grade is marked as the 3rd grade;

[0021] For the reference object vehicle, when the vehicle is not in the water area, the water level grade is marked as the 0th grade; when the vehicle is in the water area and the water level line is below the tire radius height of the vehicle, the water level grade is marked as the 1st grade; when the water level line is above the tire radius height and below the license plate height of the vehicle, the water level grade is marked as the 2nd grade; when the water level line is above the license plate height of the vehicle, the water level grade is marked as the 3rd grade;

[0022] The higher the water level grade is, the more serious the waterlogging disaster situation is.

[0023] In a second aspect, the present application provides a method for detecting the urban waterlogging disaster state, comprising:

[0024] The urban image to be detected is input into the urban waterlogging disaster state detection model constructed by the method for constructing the urban waterlogging disaster state detection model provided in the first aspect of the present application, and the urban image sequentially passes through the image feature extraction module and the image encoding module of the urban waterlogging disaster state detection model to obtain the image encoding feature; at the first time step, the preset start symbol feature and the image encoding feature are input into the image decoding module of the urban waterlogging disaster state detection model for fusion to obtain the corresponding modality fusion feature; starting from the second time step, the modality fusion feature at the last time step and the image encoding feature are input into the image decoding module for fusion to obtain the corresponding modality fusion feature; the modality fusion feature at each time step is obtained by the mapping module of the urban waterlogging disaster state detection model to obtain the text output at the corresponding time step; after a plurality of time steps, the reconstructed text describing the urban waterlogging disaster state is obtained.

[0025] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method provided in the first aspect or the second aspect.

[0026] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method provided in the first aspect or the second aspect.

[0027] In a fifth aspect, the present application further provides a computer program product, which comprises a computer program or instructions, and the computer program or instructions are executed by a processor to implement the steps of the method provided in the first aspect or the second aspect.

[0028] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0029] 1. The present application provides a construction method of an urban waterlogging disaster state detection model. First, the image coding module extracts the high-dimensional visual features of the urban image in the training sample pair, i.e., image coding features, and the embedding layer extracts the embedding features of the real text describing the waterlogging disaster state in the urban image. Then, the image decoding module fuses the image coding features and the text embedding features based on the cross-attention mechanism, improves the correlation between the image and the text, promotes the deep interaction between different modalities, effectively fuses the visual modality information and the text modality information in the feature space, enriches the fine-grained information at the semantic level, improves the understanding ability of the urban waterlogging disaster state detection model to complex flood scenes, and can accurately detect the waterlogging disaster state of complex urban scenes.

[0030] 2、Further, the construction method of the urban waterlogging disaster state detection model provided by the present application considers that the urban waterlogging disaster state detection scene has higher complexity compared to other scenes, the urban image contains rich water level reference information, and more accurate semantic information is needed, therefore, the designed multi-modal model also includes a pre-trained language model and a text decoding module; the pre-trained language model helps to extract more abstract flood level features, so as to provide more rich semantic information for the text decoding module; the text decoding module reconstructs the real text of the urban image based on the high-dimensional text features of the real text extracted by the pre-trained language model, obtains the corresponding generated text, and strengthens the understanding and generation ability of the multi-modal model for language expression at the text level. By further increasing the text language description as auxiliary supervision, the text decoder can help the multi-modal model to more deeply understand the structure and semantics of language, more effectively fuse the text features and image features, assist in supervising the generation of more accurate flood risk description of the flood image, thereby narrowing the feature distance between the image and the text, further strengthening the internal consistency and semantic coherence of the multi-modal model when processing text information, and thus improving the accuracy and naturalness of the reconstructed text generated by the urban waterlogging disaster state detection model.

[0031] 3、Further, the construction method of the urban waterlogging disaster state detection model provided by the present application, when training the multi-modal model, also minimizes the similarity between the image encoding features of the urban image and the text features of the corresponding real text, maximizes the similarity between the image encoding features of the urban image and the text features of the non-corresponding real text, minimizes the similarity between the text features of the real text and the image encoding features of the corresponding urban image, and maximizes the similarity between the text features of the real text and the image encoding features of the non-corresponding urban image; the above process enhances the alignment ability of the multi-modal model to the image and text features in the shared space, and through contrast learning, the feature representations of the corresponding image and text are made closer, while the feature representations of the non-corresponding image and text are made farther, thereby ensuring the high consistency of the generated text and the image content, promoting the deep semantic alignment and complementary learning between the image and the text, and further improving the understanding and generation ability of the urban waterlogging disaster state detection model.

[0032] 4、Further, in the method for constructing a city waterlogging disaster state detection model provided by the application, when the image encoding module is a plurality of cascaded Transformer encoders, the image encoding features used when training the multi-modal model can be the image encoding features output by the last level Transformer encoder and one or more other Transformer encoders of the image encoding module. This multi-level feature contrast learning strategy helps the model capture the mutual relationship between images and texts at different abstraction levels, thereby achieving deeper semantic understanding and representation.

[0033] 5、Further, in the method for constructing a city waterlogging disaster state detection model provided by the application, the real text describing the city waterlogging disaster state includes relationship information between the reference object and the water level of the city image, and the relationship information between the reference object and the water level is represented by a water level grade. According to different reference object types, the water level grade is divided in combination with the actual scene, and then the waterlogging situation is divided into multiple grades, thereby realizing more fine-grained detection of the city waterlogging situation. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A flowchart of a method for constructing a city waterlogging disaster state detection model provided by the application;

[0035] Figure 2 A schematic diagram of the relationship information between the reference object and the water level represented by the water level grade provided by the embodiment of the application;

[0036] Figure 3 A schematic diagram of the real text in the training sample pair provided by the embodiment of the application;

[0037] Figure 4 A schematic diagram of the overall structure of the multi-modal model provided by the embodiment of the application;

[0038] Figure 5 A schematic diagram of the detailed structure of the multi-modal model provided by the embodiment of the application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0040] In order to achieve the above-mentioned purpose, in a first aspect, as shown in Figure 1 The application provides a method for constructing a city waterlogging disaster state detection model, comprising:

[0041] Obtain a training sample set; the training sample set includes multiple training sample pairs, each of which includes a city image and corresponding real text describing the state of urban flooding.

[0042] Construct a multimodal model; the multimodal model includes: an image feature extraction module, an image encoding module, a tokenization layer, an embedding layer, an image decoding module, and a mapping module; the image encoding module includes a Transformer encoder; the image decoding module includes a Transformer decoder;

[0043] The training sample pairs are input into the multimodal model. The city images are sequentially processed through an image feature extraction module and an image encoding module to obtain image encoded features; the real text is sequentially processed through a tokenization layer and an embedding layer to obtain text embedding features {e1,e2,…,e m}; where e l Let l be the embedding feature of the l-th token in the real text, where l = 1, 2, ..., m; insert a preset start symbol feature e0 before the text embedding feature; at the (j+1)-th time step, combine the image encoding feature with the feature {e0, e1, e2, ..., e...} j The input is fed into the image decoding module and fused using a cross-attention mechanism to obtain the corresponding modality fusion features. Then, the mapping module outputs the text at the corresponding time step. After m+1 time steps, the reconstructed text describing the urban flooding disaster is obtained; j = 0, 1, ..., m; m is the total number of tokens in the real text. It should be noted that the preset start symbol feature e0 is a special symbol indicating the start, such as... <bos>.

[0044] The multi-modal model is trained by minimizing a difference loss between the reconstructed text and the corresponding real text.

[0045] After the training is completed, a city waterlogging disaster state detection model including a cascaded image feature extraction module, an image encoding module, an image decoding module, and a mapping module is constructed.

[0046] It should be noted that the real text describing the city waterlogging disaster state includes relationship information between a reference object and a water level of the city image; wherein the relationship information between the reference object and the water level is represented by a water level grade; the reference object includes a person, a vehicle, etc.

[0047] As shown in Figure 2 For the reference object person, when the person is not in the water area, the water level grade is marked as level 0; when the person is in the water area and the water level line is below the person's knees, the water level grade is marked as level 1; when the water level line is above the person's knees and below the waist, the water level grade is marked as level 2; and when the water level line is above the person's waist, the water level grade is marked as level 3.

[0048] For the reference object vehicle, when the vehicle is not in the water area, the water level grade is marked as level 0; when the vehicle is in the water area and the water level line is below the tire radius height, the water level grade is marked as level 1; when the water level line is above the tire radius height and below the license plate height, the water level grade is marked as level 2; and when the water level line is above the license plate height, the water level grade is marked as level 3.

[0049] The higher the water level grade, the more serious the waterlogging disaster.

[0050] Preferably, in an optional embodiment, the real text describing the city waterlogging disaster state further includes a global scene description and an image global flood risk description of the city image.

[0051] It should be noted that the image encoding module can be a Transformer encoder or multiple cascaded Transformer encoders; and the image decoding module can be a Transformer decoder or multiple cascaded Transformer decoders.

[0052] When the image encoding module is multiple cascaded Transformer encoders, the input of the rth Transformer encoder is the image encoding feature output by the (r-1)th Transformer encoder; r=2, 3, …, R; R is the total number of Transformer encoders; and the input of the first Transformer encoder is the feature extracted by the image feature extraction module from the city image.

[0053] The input of the pth Transformer decoder is the modality fusion feature output by the (p-1)th Transformer decoder and the image encoding feature output by the Rth Transformer encoder; p = 2, 3, …, P; P is the total number of the Transformer decoders in the image decoding module; wherein the input of the first Transformer decoder is the preset start symbol feature and the image encoding feature output by the Rth Transformer encoder.

[0054] In an optional implementation, the above multi-modal model further comprises: a pre-trained language model and a text decoding module; the text decoding module comprises a Transformer decoder;

[0055] The above text embedding feature is input into the pre-trained language model to obtain a text feature; at the (j+1)th time step, the text feature is also input into the text decoding module based on the cross-attention mechanism together with the above features {e0, e1, e2, …, e j} to obtain corresponding text fusion features, and then the text output of the corresponding time step is obtained through a mapping module; after m+1 time steps, the generated text describing the urban waterlogging disaster state is obtained.

[0056] When training the multi-modal model, the difference loss between the generated text and the corresponding real text is also minimized.

[0057] It should be noted that the text decoding module can be a Transformer decoder or a plurality of cascaded Transformer decoders.

[0058] The input of the sth Transformer decoder is the text fusion feature output by the (s-1)th Transformer decoder and the text feature output by the pre-trained language model; s = 2, 3, …, S; S is the total number of the Transformer decoders in the text decoding module; wherein the input of the first Transformer decoder is the preset start symbol feature and the text feature output by the pre-trained language model.

[0059] In an optional implementation, when the multi-modal model is trained, a contrastive loss for aligning the image encoding features of the city image and the text features of the corresponding real text is also minimized, specifically comprising: the similarity between the image encoding features of the city image and the text features of the corresponding real text is also minimized, the similarity between the image encoding features of the city image and the text features of the non-corresponding real text is maximized, the similarity between the text features of the real text and the image encoding features of the corresponding city image is minimized, and the similarity between the text features of the real text and the image encoding features of the non-corresponding city image is maximized.

[0060] It should be noted that when the image encoding module is a plurality of cascaded Transformer encoders, the image encoding features described above can be only the image encoding features output by the last level Transformer encoder, and can also be the image encoding features output by the last level Transformer encoder and one or more other Transformer encoders.

[0061] It should be noted that the image feature extraction module can be CNN, ResNet, MobileNet, VisionTransformer, etc., which is not limited here. The mapping module can be MLP, KAN, etc., which is not limited here. The pre-trained language model can be a BERT model, a RoBERTa model, a DeBERTa model, etc., which is not limited here.

[0062] In order to further illustrate the construction method of the urban waterlogging disaster state detection model provided by the present application, a specific embodiment will be described in detail below:

[0063] The construction method of the urban waterlogging disaster state detection model provided in this embodiment comprises:

[0064] Step 1: Obtain the urban image waterlogging description dataset, the images in the dataset of the present application are mostly obtained from the Internet, search engines such as Baidu, Google and Bing are searched using Python crawlers, and the keywords used mainly include urban flood, waterlogging and urban flood. The data contains 3220 urban images, each image is equipped with detailed JSON format metadata to accurately reflect the flood scene information and risk level in the urban image in the descriptive text as a true text describing the disaster state of urban waterlogging. Among them, 2576 images are used for training to build a training sample set, and 644 images are used for testing to build a test sample set. The training sample set uses reference objects to evaluate flood risk, the annotation of each image not only includes the overall scene and key details, but also refines the risk level description, and a total of four levels are selected for cars and people as the standard for evaluating the disaster state of urban waterlogging. Specifically, for the reference object person, when the person is not in the water area, the water level is marked as level 0; when the person is in the water area, and the water level is below the person's knees, the water level is marked as level 1; when the water level is above the person's knees and below the waist, the water level is marked as level 2; when the water level is above the person's waist, the water level is marked as level 3. For the reference object car, when the car is not in the water area, the water level is marked as level 0; when the car is in the water area, and the water level is below the tire radius height, the water level is marked as level 1; when the water level is above the tire radius height and below the license plate height, the water level is marked as level 2; when the water level is above the license plate height, the water level is marked as level 3. The higher the water level, the more serious the waterlogging disaster; in this embodiment, 0-3 represent no flood, the probability of flood is less than or equal to 25%, the probability of flood is greater than 25% and less than or equal to 50%, and the probability of flood is greater than 50%, respectively.

[0065] An example of annotation is shown as Figure 3 As shown, it specifically includes the following six main annotation parts:

[0066] 1) image_content_caption: describes the scene information in the image, emphasizing the objects and elements in the scene.

[0067] 2) image_risk_caption: based on the risk level system, the overall risk of the urban waterlogging scene depicted in the image is evaluated.

[0068] 3) car_risk_caption: evaluates the risk level of vehicles in the urban image.

[0069] 4) car_position_caption: According to the spatial position relationship between the vehicle and the flood, the key position context in the scene is provided using words such as "tire" and "hood".

[0070] 5) person_risk_caption: Assess the risk level of pedestrians.

[0071] 6) person_position_caption: Describes the relative position relationship between the individual and the flood, and the position description of the water level and the human body is achieved through the key parts of the human body "ankle", "knee", "waist", which depicts their proximity to the flood and the possible interactive dynamics.

[0072] Step 2: Construct a multi-modal model;

[0073] Through deep feature extraction and serialization processing, accurate semantic analysis of complex scenes is achieved. The overall structure diagram of the multi-modal model constructed in this embodiment is shown in Figure 4 .

[0074] City images and corresponding text information have different characteristics and representations. Images are pixel-based visual information, while text is symbol-based semantic information. Fusing these two types of information requires addressing data heterogeneity and complexity. The present invention proposes a multi-modal collaborative attention mechanism based on Transformer, which encodes and decodes images and text through self-attention mechanisms and cross-attention mechanisms to learn the association between images and text.

[0075] First, the multi-modal model uses a convolutional neural network to extract features from the input city image. The convolutional neural network used in this embodiment is ResNet-101, which is a network with a depth of 101 layers. Its deep structure allows the network to capture complex and subtle features in the image at different levels, including not only basic visual elements such as color, shape, and texture, but also higher-order abstract features, which help to fully analyze the content and context of the flood image. In addition, the residual connection design of ResNet helps to solve the performance degradation problem that may occur in the model, ensuring that the model can effectively learn even when the model is very deep.

[0076] In addition, in order to enhance the semantic representation ability of visual features, text features are also introduced as auxiliary information in this embodiment. The corresponding real text is converted into tokens and position encodings are added through the tokenization layer and embedding layer to obtain the corresponding text embedding features, ensuring the integrity of the word sequence information. The text contains specific descriptions of the flood scene, including the relative spatial relationship between the reference objects, pedestrians and vehicles, and the flood water level, as well as the relevant description of the flood risk level. Further, the text embedding features are encoded using the BERT model for context-aware feature encoding to obtain the corresponding text features. As a deep learning architecture based on Transformer, the BERT model can capture complex context relationships in text data through its bidirectional encoding mechanism, which enables the model to not only recognize key words (such as "water depth", "submerged", etc.) in the flood image description, but also understand the meaning of these words in a specific context, thereby generating more accurate and detailed image descriptions. In addition to the bidirectional encoding mechanism, the deep feature extraction capability of BERT enables the model to identify and understand fine-grained information in the image description. For example, the model can distinguish between different degrees of flooding, such as light waterlogging or severe flooding.

[0077] Subsequently, the features of these images and texts are passed into a Transformer-based multi-modal co-attention mechanism module, which consists of a series of Transformer encoders and decoders; Transformers can effectively learn the complex correlations and dependencies between data by considering the relationships between all elements in the input data when processing the input data using self-attention mechanisms, which can capture long-range dependencies in the input data, which is particularly important for understanding complex, context-dependent information in texts and images. In this embodiment, each encoder layer further refines and enhances the features, strengthening the model's global understanding of the scene. The decoder part focuses on converting the encoded mixed features back into detailed text descriptions closely related to the original urban image content, achieving the reconstruction of the reconstructed text describing the image urban waterlogging disaster state. This process can be regarded as a process of "reconstructing" or "restoring" text features from visual features. Multi-head attention mechanisms are embedded in both the encoder and the decoder. This mechanism assigns input features to different subspaces and calculates attention scores independently within each subspace, giving the model the ability to learn on multiple feature subsets. In the context of flood risk description, the multi-head attention mechanism not only focuses on key objects in the scene, but also combines the visual context of the image with the corresponding text information, enabling the model to capture not only single features but also the complex interaction relationships of these objects in a specific environment, demonstrating its insight into the deep semantics of the scene, and thus generating descriptions with rich context awareness. For example, in a flood scene, there are many visual elements involved, including water level, affected vehicles, pedestrians, etc. The use of multi-head attention mechanisms allows parallel attention to different aspects of these visual elements. Specifically, one attention head of the model may focus on changes in water level, while another may concentrate on locating disaster personnel. Through this multi-dimensional observation, the model achieves a comprehensive capture and expression of the flood risk panorama.

[0078] Finally, the features generated by the decoder are further processed through a multi-layer perceptron (MLP), which outputs risk assessment levels and descriptive sentences according to the specific characteristics of the scene, providing detailed semantic descriptions for each element detected in the scene, such as "person risk level is 3", "water level is below half the tire", "a group of people walking through a flooded area", etc.

[0079] Specifically, the detailed structure diagram of the multi-modal model is as follows Figure 5 As shown, the image and text are encoded and decoded by the self-attention mechanism and cross-attention mechanism to learn the association between the image and the text. The city image obtains a high-dimensional feature vector ax through a feature extraction network, and the corresponding real text is tokenized to obtain an enhanced text embedding feature e i , which is then deeply encoded using a BERT model to obtain rich text features t i . Among them, the image feature vector obtains a deep visual representation v i through an image encoder, which is referred to as image encoding features. In the image encoder, the attention mechanism is split into multiple "heads" using the multi-head attention mechanism, each head independently learns different parts of the input data, so that the model captures information in different representation subspaces, thereby obtaining richer expression capabilities. In order to further enhance the semantic processing ability of the model and make semantic information assist image generation to generate more accurate descriptions, two Transformer decoders are designed in this embodiment, one is an image decoder and the other is a text decoder. The image decoder fuses the text embedding feature e i and the image encoding feature v i through cross-attention mechanism to promote deep interaction between different modalities, so as to mine complex relationships and patterns between modalities, thereby enriching and enhancing the expression ability of the features, realizing the reconstruction process from image to description, and ensuring that the generated description can accurately reflect the image content. The text decoder is mainly responsible for the reconstruction process of the description content, and its core role is to strengthen the model's understanding and generation ability of language expression on the text level. By reconstructing the original description information, the text decoder helps the model to more deeply understand the structure and semantics of language, strengthens the internal consistency and semantic coherence of the model when processing text information, thereby improving the accuracy and naturalness of the generated text. The combination of the two decoders makes the model more accurate when understanding and generating descriptions. The image decoder fuses visual and language information to ensure that the generated description is highly related to the image content; the text decoder enhances language processing ability, improving the accuracy and fluency of the model in text generation. The design of the double decoder reflects the advantages of multi-modal fusion, which not only improves the performance of the model on the language level, but also achieves significant progress in the fusion of vision and language.

[0080] In addition, the embodiment adopts the strategy of image-text multiple loss function to jointly train the multi-modal model to enhance its performance in processing multi-modal data. The loss function includes three parts: image-text generation loss, image-text text contrast loss, and text autoregressive generation loss.

[0081] Image-text generation loss measures the difference between the reconstructed text and the corresponding real text, ensuring that the model can generate text that is semantically close to the real description, thereby improving the accuracy of converting visual information into linguistic description. It can be measured using cross-entropy loss, KL divergence loss, mean squared error loss, etc. This embodiment uses cross-entropy loss for calculation. Image-text contrastive loss aims to enhance the model's ability to align image-encoded features and corresponding text features in the shared space. Through contrastive learning, it makes the feature representations of corresponding images and texts closer, while widening the distance between the feature representations of non-corresponding images and texts, thus ensuring a high degree of consistency between the generated text and image content. It can be measured using contrastive loss functions (such as normalized temperature scale cross-entropy contrastive loss), cosine similarity loss, Focal Loss, etc. This embodiment uses normalized temperature scale cross-entropy contrastive loss function for optimization. The text autoregressive generation loss is used to measure the difference between the generated text and the corresponding real text in order to evaluate the ability of the language model to generate text. It can be measured by loss functions such as cross-entropy loss function, KL divergence loss function, and mean squared error loss function. This implementation uses the cross-entropy loss function for calculation.

[0082] By using multiple loss functions, the model can not only generate corresponding text descriptions from images more accurately, but also achieve effective semantic alignment between multimodal data, ultimately improving the overall performance of the model in multimodal learning tasks.

[0083] The total loss function expression for the multimodal model is:

[0084] l total =l ce_ITG +l ce_ITG +βl NT_ITG

[0085] Among them, l ce_ITG Represents the image-to-text generation loss; l ce_ITG Generate loss for text autoregression; NT_ITG For the image-text contrastive loss, only the image encoding features output by the image encoding module and the text features can be selected for contrastive learning; when the image encoding module includes multiple cascaded Transformer encoders, the contrastive loss can also evaluate the consistency between the image encoding features output by different layers of the Transformer encoder in the image encoding module and the generated description text. Specifically, the image encoding features output by different layers of the Transformer decoder in the image encoding module (such as the image encoding features output by the last layer, the third last layer, and the fifth last layer of the Transformer decoder) can be selected respectively for contrastive learning with the text features. This multi-level feature contrastive learning strategy helps the model to capture the mutual relationship between images and texts at different levels of abstraction, thereby achieving deeper semantic understanding and representation in the multi-modal learning task. By adjusting the weight β of the information loss (set to 0.2 in this embodiment), the contribution of each part of the loss to the total loss during training can be controlled. This weighting strategy enables the model not only to generate accurate descriptions, but also to better understand and express the details and context information of the image.

[0086] wherein the contrastive loss l NT_ITG The normalized temperature cross-entropy contrastive loss function is measured, and its calculation formula is:

[0087]

[0088]

[0089] It should be noted that the image and the text in the same training sample pair in the training sample set correspond to each other, and the images and the texts in different training sample pairs do not correspond to each other; N is the total number of training sample pairs in the training sample set; λ is a weight parameter; z vi , z ui , z vk , z uk is the feature obtained by projecting the image encoding feature of the city image I i , the text feature of the real text T i in the i-th training sample pair (I i , T i ), and the image encoding feature of the city image I k , the text feature of the real text T k in the k-th training sample pair (I k , T k ) to a common low-dimensional feature space; sim(z vi , z ui ) represents the features zx i and z ui The similarity between them is measured by cosine similarity in this embodiment; τ is a temperature parameter used to adjust the sensitivity of the contrast loss.

[0090] l image_text,i Image-centric pairing focuses on the association of images to text, helping the multi-modal model to understand image content and generate corresponding text descriptions, which enhances the ability of the multi-modal model to convert visual information into language descriptions. text_image,i Text-centric pairing focuses on the association of text to images, enabling the multi-modal model to identify relevant visual content based on text descriptions, which enhances the understanding of details in text descriptions and matches specific visual features. This bidirectional processing mechanism helps to ensure that the model can not only understand text from images but also understand images from text, achieving more comprehensive feature learning in multi-modal flood scene learning tasks.

[0091] Multiple loss functions help improve the ability of the multi-modal model to process and interpret image content, ensuring that the reconstructed text not only matches the true text description but also closely relates to the image content.

[0092] This embodiment uses the ADAM optimizer to train the multi-modal model, with a batch size of 16, an initial learning rate of 1e -4 -4, and a maximum number of iterations of the multi-modal model set to 100. The learning rate decreases every 20 iterations during training. When this value is reached or not reached, the model has converged, and the multi-modal model terminates iteration.

[0093] Step 3: After the multi-modal model is trained, a city flood disaster state detection model is constructed, including a cascaded image feature extraction module, an image encoding module, an image decoding module, and a mapping module.

[0094] In a second aspect, the present application provides a city flood disaster state detection method, comprising:

[0095] The city image to be detected is input into the city waterlogging disaster state detection model constructed by the method for constructing a city waterlogging disaster state detection model according to the first aspect of the present application, and sequentially passes through the image feature extraction module and the image coding module of the city waterlogging disaster state detection model to obtain image coding features; at a first time step, a preset start symbol feature and the image coding features are input into the image decoding module of the city waterlogging disaster state detection model based on the cross attention mechanism to obtain corresponding modal fusion features; starting from a second time step, the modal fusion features at the previous time step and the image coding features are input into the image decoding module of the city waterlogging disaster state detection model based on the cross attention mechanism to obtain corresponding modal fusion features; the modal fusion features at each time step are respectively input into the mapping module of the city waterlogging disaster state detection model to obtain text output at the corresponding time step; after a plurality of time steps, a reconstructed text describing the city waterlogging disaster state in the city image to be detected is obtained.

[0096] The related technical solutions are the same as the method for constructing a city waterlogging disaster state detection model according to the first aspect of the present application, which will not be repeated here.

[0097] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the method according to the first aspect or the second aspect of the present application.

[0098] The related technical solutions are the same as the method for constructing a city waterlogging disaster state detection model according to the first aspect of the present application and the method for detecting a city waterlogging disaster state according to the second aspect of the present application, which will not be repeated here.

[0099] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to execute the method according to the first aspect or the second aspect of the present application.

[0100] The related technical solutions are the same as the method for constructing a city waterlogging disaster state detection model according to the first aspect of the present application and the method for detecting a city waterlogging disaster state according to the second aspect of the present application, which will not be repeated here.

[0101] In a fifth aspect, the present application further provides a computer program product, comprising a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps of the method according to the first aspect or the second aspect of the present application.

[0102] The related technical solutions are the same as the method for constructing a city waterlogging disaster state detection model according to the first aspect of the present application and the method for detecting a city waterlogging disaster state according to the second aspect of the present application, which will not be repeated here.

[0103] It is to be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reading and understanding the above description. The scope of the application should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.< / bos>

Claims

1. A method for constructing an urban waterlogging disaster state detection model, characterized in that, The method comprises the following steps: obtaining a training sample set; the training sample set comprises a plurality of training sample pairs, each training sample pair comprising a city image and corresponding real text describing the disaster state of urban waterlogging; constructing a multi-modal model; the multi-modal model comprises an image feature extraction module, an image encoding module, a tokenization layer, an embedding layer, an image decoding module, and a mapping module; the image encoding module comprises a Transformer encoder; the image decoding module comprises a Transformer decoder; The training sample pair is input into the multi-modal model, wherein the urban image sequentially passes through the image feature extraction module and the image encoding module to obtain image encoding features; and the real text sequentially passes through the tokenization layer and the embedding layer to obtain text embedding features ; wherein, is an embedding feature of the i-th token of the real text, l ; a preset start symbol feature is inserted before the text embedding feature ; at the i+1 time step, the image encoding features and the features are input into the image decoding module for fusion to obtain corresponding modal fusion features, and then the mapping module is used to obtain the text output at the corresponding time step; after i+1 time steps, the reconstructed text describing the urban waterlogging disaster state is obtained; j m ; and ; m is the total number of tokens of the real text.​​ training the multi-modal model by minimizing the difference loss between the reconstructed text and the corresponding real text; after training, constructing a city waterlogging disaster state detection model comprising the image feature extraction module, the image encoding module, the image decoding module, and the mapping module in cascade; the multi-modal model further comprises a pre-trained language model and a text decoding module; the text decoding module comprises a Transformer decoder; The text embedding features are input into the pre-trained language model to obtain text features; in the... j At +1 time step, the text features are also compared with the features The text is input into the text decoding module for fusion to obtain the corresponding text fusion features, and then the text output at the corresponding time step is obtained through the mapping module; after... m Add 1 time step to obtain the generated text describing the urban flooding disaster status; when training the multi-modal model, the difference loss between the generated text and the corresponding real text is also minimized; when training the multi-modal model, the similarity between the image encoding features of the city image and the text features of the corresponding real text is minimized, the similarity between the image encoding features of the city image and the text features of the non-corresponding real text is maximized, the similarity between the text features of the real text and the image encoding features of the corresponding city image is minimized, and the similarity between the text features of the real text and the image encoding features of the non-corresponding city image is maximized.

2. The method according to claim 1, wherein, When the image encoding module is a plurality of cascaded Transformer encoders, the image encoding features used for training the multi-modal model are the image encoding features output by the last level Transformer encoder of the image encoding module, or the image encoding features output by the last level Transformer encoder and one or more other Transformer encoders of the image encoding module.

3. The method according to any one of claims 1-2, wherein, The image encoding module comprises a plurality of cascaded Transformer encoders; the image decoding module comprises a plurality of cascaded Transformer decoders; No. r The input of the first-level Transformer encoder is the second-level encoder. r Image encoding features output by a -1 level Transformer encoder; R ; R The total number of Transformer encoders; the first One The input to the Transformer encoder is the features extracted from the city image after passing through the image feature extraction module; The input of the first-stage Transformer decoder is the first-stage image encoding feature output by the first-stage Transformer encoder. p The input of the first-stage Transformer decoder is the first-stage image encoding feature output by the first-stage Transformer encoder. p The input of the first-stage Transformer decoder is the first-stage image encoding feature output by the first-stage Transformer encoder. R The input of the first-stage Transformer decoder is the first-stage image encoding feature output by the first-stage Transformer encoder. P ; P The total number of the Transformer decoders is N; the input of the first-stage Transformer decoder is the preset start symbol feature and the image encoding feature output by the first-stage Transformer encoder. One The total number of the Transformer decoders is N; the input of the first-stage Transformer decoder is the preset start symbol feature and the image encoding feature output by the first-stage Transformer encoder. R The total number of the Transformer decoders is N; the input of the first-stage Transformer decoder is the preset start symbol feature and the image encoding feature output by the first-stage Transformer encoder.

4. The method according to any one of claims 1-2, wherein, The real text describing the disaster state of urban waterlogging comprises relationship information between a reference object and a water level of the city image; wherein the relationship information between the reference object and the water level is represented by a water level grade; the reference object comprises a person and / or a vehicle; For the reference object person, when the person is not in the water area, the water level grade is marked as level 0; when the person is in the water area and the water level line is below the person's knees, the water level grade is marked as level 1; when the water level line is above the person's knees and below the person's waist, the water level grade is marked as level 2; when the water level line is above the person's waist, the water level grade is marked as level 3; For the reference vehicle, when the vehicle is not in the water area, the water level is marked as level 0; when the vehicle is in the water area and the water level line is below the tire radius height, the water level is marked as level 1; when the water level line is above the tire radius height and below the license plate height, the water level is marked as level 2; when the water level line is above the license plate height, the water level is marked as level 3. The higher the water level, the more serious the waterlogging disaster.

5. A method for detecting a state of urban waterlogging disaster, characterized by, The method comprises the following steps: The image to be detected is input into the urban waterlogging disaster state detection model constructed by the method for constructing an urban waterlogging disaster state detection model according to any one of claims 1-4, and the image is sequentially subjected to image feature extraction and image coding by the image feature extraction module and the image coding module of the urban waterlogging disaster state detection model to obtain image coding features; at the first time step, a preset start symbol feature and the image coding features are input into the image decoding module of the urban waterlogging disaster state detection model for fusion to obtain corresponding modal fusion features; From the second time step, the modal fusion features at the previous time step and the image coding features are input into the image decoding module for fusion to obtain corresponding modal fusion features; the modal fusion features at each time step are respectively subjected to mapping by the mapping module of the urban waterlogging disaster state detection model to obtain text output at the corresponding time step; after a plurality of time steps, reconstructed text describing the urban waterlogging disaster state is obtained.

6. An electronic device, comprising: The method comprises the following steps: A memory and a processor, the memory stores a computer program, and the processor executes the computer program to execute the method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is executed by a processor to execute the method according to any one of claims 1-5.

8. A computer program product, characterised in that, A computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the steps of the method according to any one of claims 1-5.

Images