Flood event multi-modal retrieval method and device based on semantic enhancer

Abstract

The invention provides a flood event multi-mode retrieval method and device based on a semantic intensifier, and relates to the technical field of data retrieval, and the method comprises the steps: carrying out multi-stage semantic enhancement on keywords through a global flood event knowledge base and internet news information included in the semantic intensifier, then quality priority retrieval and preliminary screening are carried out to obtain a to-be-matched multi-source remote sensing image, and corresponding optical and radar image features and context semantic features of a semantic enhancement text are extracted from an image mode and a text mode respectively; and then fusing the features of the image modality and the text modality to obtain a multi-modality fusion feature, mapping to obtain the similarity between the semantic enhancement text and the to-be-matched multi-source remote sensing image, and finally determining a retrieval result of the keyword. According to the method and the device, the problems of low retrieval matching degree and low retrieval efficiency caused by lack of semantics and difficulty in acquisition of event-related space-time ranges in flood event remote sensing data retrieval are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data retrieval, and in particular to a multimodal retrieval method and device for flood events based on a semantic enhancer. Background Art

[0002] With the continuous development of earth observation technology, the types of satellite sensors continue to increase, and the amount of remote sensing data has shown exponential growth, with multi-source, multi-scale, multi-phase, high-dimensional and other characteristics, marking the arrival of the era of "Earth Big Data". In addition, as governments and research institutions gradually open up data resources, PB-level remote sensing data can be obtained free of charge through cloud platforms such as Amazon Cloud AWS and Google Earth Engine (GEE), providing rich data support for large-scale resource and environmental research on a global scale.

[0003] However, the existing remote sensing data retrieval technology lags far behind the data growth rate. Especially in complex scenarios such as flood events, it is still difficult to quickly filter out the data that meets the needs from massive, multi-source remote sensing data. Although high-resolution remote sensing images contain rich potential semantic information, traditional retrieval methods mainly rely on shallow features such as spatiotemporal range, sensor parameters or label categories, and lack deep semantic support, resulting in unsatisfactory retrieval matching and efficiency when dealing with complex events (such as floods).

[0004] In addition, in the process of remote sensing data retrieval of flood events, the problem of insufficient semantic information of events is often faced. Flood events involve key details such as specific locations, time ranges, causes, and affected areas, but this information is difficult to directly present or clearly express in traditional remote sensing data. At the same time, retrieval usually relies on the specific spatiotemporal range and keywords that describe the event. This information is often difficult to accurately obtain in complex remote sensing scenes, making it difficult for the retrieval system to accurately match relevant data. The problems of insufficient information and missing keywords directly affect the matching degree of the retrieval results, resulting in a low system recall rate and reduced retrieval efficiency, which results in a large amount of relevant data not being effectively retrieved. Summary of the invention

[0005] The present invention provides a multimodal retrieval method and device for flood events based on a semantic enhancer, which is used to solve the problems in the prior art of flood event-related remote sensing data retrieval, such as insufficient event semantic information and difficulty in obtaining keywords such as event-related spatiotemporal range, resulting in low data retrieval matching degree and low retrieval efficiency.

[0006] The present invention provides a flood event multimodal retrieval method based on a semantic enhancer, the method comprising the following steps: For the keywords input by the user, multi-level semantic enhancement is performed on the keywords through the global flood event knowledge base and Internet news information included in the semantic enhancer to obtain semantically enhanced text; The satellite remote sensing images of the flood event are screened by quality priority to obtain candidate multi-source remote sensing images, and the candidate multi-source remote sensing images are preliminarily retrieved based on the semantically enhanced text to obtain multi-source remote sensing images to be matched, wherein the multi-source remote sensing images to be matched include optical images and radar images; Extracting optical image features and radar image features of the multi-source remote sensing image to be matched based on the ResNet model, and extracting contextual semantic features of the semantically enhanced text based on the BERT model; The radar image feature and the optical image feature are spliced ​​to obtain a multi-source remote sensing fusion feature, and the multi-source remote sensing fusion feature and the context semantic feature are multimodally fused based on the BERT model to obtain a multimodal fusion feature; The multimodal fusion features are mapped to obtain the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched, and the multi-source remote sensing image to be matched with the highest similarity and the corresponding semantically enhanced text are selected as the search results of the keywords.

[0007] In some embodiments, the semantic enhancer includes a global flood event knowledge base and Internet news information, and performs multi-level semantic enhancement on the keywords to obtain semantically enhanced text, including: In the first stage, the overall similarity between the keyword and the flood event records in the global flood event knowledge base included in the semantic enhancer is calculated, and the overall similarity is obtained by weighted calculation of edit distance, text cosine similarity, fuzzy matching similarity and sequence matching similarity; In the second stage, when the overall similarity is higher than a set threshold, the flood event record with the highest overall similarity is retrieved from the global flood event knowledge base as the semantically enhanced text; In the third stage, when the overall similarity is lower than or equal to the set threshold, the keywords are semantically enhanced by the Internet news information included in the semantic enhancer to obtain a semantically enhanced text.

[0008] In some embodiments, the step of screening the satellite remote sensing images of the flood event by quality priority to obtain candidate multi-source remote sensing images includes: According to the location and duration of the flood event, corresponding search parameters are set to retrieve satellite remote sensing images; Among the satellite remote sensing images, optical images with cloud cover below a threshold, radar images in an online state, and radar images in an offline state and activated are obtained as candidate multi-source remote sensing images.

[0009] In some embodiments, the preliminary retrieval of the candidate multi-source remote sensing images based on the semantically enhanced text to obtain the multi-source remote sensing images to be matched includes: Extracting spatiotemporal range information from the semantically enhanced text, and converting the spatiotemporal range information into longitude and latitude coordinates; The latitude and longitude coordinates are used as pre-screening conditions, and the candidate multi-source remote sensing images are screened according to the pre-screening conditions to obtain the multi-source remote sensing images to be matched.

[0010] In some embodiments, the multi-modal fusion of the multi-source remote sensing fusion features and the contextual semantic features based on the BERT model to obtain the multi-modal fusion features includes: Calling the BERT model to embed the multi-source remote sensing fusion features and the contextual semantic features to obtain corresponding embedded multimodal features; The embedded multimodal features are subjected to multimodal feature fusion through the BERT model to obtain multimodal fusion features.

[0011] In some embodiments, the mapping process of the multimodal fusion features to obtain the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched includes: Selecting a prediction label of a flood event from the global flood event knowledge base; The multimodal fusion features are mapped to obtain a matching similarity for the predicted label, and the matching similarity is used as a label matching similarity between the semantically enhanced text and the multi-source remote sensing image to be matched.

[0012] The present invention also provides a flood event multimodal retrieval device based on a semantic enhancer, the device comprising the following modules: A semantic enhancement module, for performing multi-level semantic enhancement on keywords input by a user through a global flood event knowledge base and Internet news information included in a semantic enhancer, to obtain semantically enhanced text; A feature extraction module is used to perform quality-first screening on the satellite remote sensing images of the flood event to obtain candidate multi-source remote sensing images, and to perform preliminary retrieval on the candidate multi-source remote sensing images based on the semantically enhanced text to obtain multi-source remote sensing images to be matched, wherein the multi-source remote sensing images to be matched include optical images and radar images; Extracting optical image features and radar image features of the multi-source remote sensing image to be matched based on the ResNet model, and extracting contextual semantic features of the semantically enhanced text based on the BERT model; A feature fusion module, used for splicing the radar image feature with the optical image feature to obtain a multi-source remote sensing fusion feature, and performing multi-modal fusion on the multi-source remote sensing fusion feature and the contextual semantic feature based on a BERT model to obtain a multi-modal fusion feature; The multimodal retrieval module is used to map the multimodal fusion features to obtain the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched, and select the multi-source remote sensing image to be matched with the highest similarity and the corresponding semantically enhanced text as the retrieval result of the keyword.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a multimodal retrieval method for flood events based on a semantic enhancer as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multimodal retrieval methods for flood events based on a semantic enhancer as described above.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned multimodal retrieval methods for flood events based on a semantic enhancer.

[0016] The multimodal retrieval method and device for flood events based on semantic enhancers provided by the present invention firstly perform multi-level flood event related semantic enhancement on the search keywords based on the semantic enhancers of the global flood event knowledge base and Internet news information. Secondly, the search text after semantic enhancement is used to perform multi-source remote sensing image quality priority screening and preliminary search. Thirdly, based on the ResNet model and the BERT model, a multimodal feature extraction method combining remote sensing image features and flood event text semantics is constructed, which can obtain contextual semantic features related to flood events while obtaining deep-level image visual semantic features. Subsequently, a multimodal feature fusion method for multi-source remote sensing images and text information is constructed based on the BERT model, and feature fusion is performed on text modal features and image modal features. Finally, a multimodal remote sensing image retrieval combined with text semantics related to flood events is performed in combination with multimodal feature similarity matching and multimodal feature mapping methods. The method can solve the problems of low data retrieval matching degree and low retrieval efficiency caused by the lack of event semantic information and the difficulty in obtaining keywords such as event-related spatiotemporal range in the retrieval of remote sensing data related to flood events. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the multimodal retrieval method for flood events based on semantic enhancer provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the implementation principle of the multimodal retrieval method for flood events based on the semantic enhancer provided by the present invention.

[0020] Figure 3 It is a processing schematic diagram of the semantic enhancer provided by the present invention.

[0021] Figure 4 It is a schematic diagram of the framework of the multimodal retrieval provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of a flood event multimodal retrieval device based on a semantic enhancer provided by the present invention.

[0023] Figure 6 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The multimodal retrieval method for flood events based on a semantic enhancer provided by the present invention is described below in conjunction with the accompanying drawings.

[0026] Figure 1 is a flow chart of a multimodal flood event retrieval method based on a semantic enhancer provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 104.

[0027] Step 101: For keywords input by a user, multi-level semantic enhancement is performed on the keywords through a global flood event knowledge base and Internet news information included in a semantic enhancer to obtain semantically enhanced text.

[0028] First, we need to obtain search keywords related to flood events. Here, we can refer to Figure 2 , Figure 2 is a schematic diagram of the implementation principle of the flood event multimodal retrieval method based on semantic enhancer provided by the present invention, such as Figure 2As shown in the figure, when a user needs to query news reports related to flood events, he can directly enter the search keyword key (hereinafter referred to as keyword) in the terminal's search engine or search engine. After obtaining the keyword key related to the flood event, the keyword key entered by the user is multi-level semantically enhanced through the global flood event knowledge base and Internet news information included in the semantic enhancer to obtain a semantically enhanced text.

[0029] Specifically, the keyword key input by the user will be input into the semantic enhancer, which is built based on the global flood event knowledge base. After the semantic enhancer accepts the input keyword key, it first performs a first-level search on the global flood event knowledge base through the keyword key. Similarity calculation is performed during the retrieval process, specifically calculating the overall similarity between the keyword and the flood event records in the global flood event knowledge base included in the semantic enhancer. The overall similarity is obtained by weighted calculation of the edit distance, text cosine similarity, fuzzy matching similarity and sequence matching similarity.

[0030] Here, four similarity calculations are performed on keywords and flood event records in the global flood event knowledge base: edit distance, text cosine similarity, fuzzy string matching, and sequence matching. Figure 3 Let's introduce the semantic enhancement process of the semantic enhancer in detail. Figure 3 is a processing schematic diagram of the semantic enhancer provided by the present invention, such as Figure 3 As shown in the figure, after the user searches and inputs the keyword key, the semantic enhancer is entered. First, the similarity between the keyword key and the global flood event knowledge base is determined. Here, the global flood event knowledge base is pre-constructed, and is specifically composed of multiple flood event record queries. The flood event record query is retrieved from the Internet flood event news using the location and time of the flood event as search parameters. These flood event record queries describe rich semantic information related to the flood event. Therefore, the similarity calculation process is also the process of semantic information matching. The overall similarity between the keyword key and the global flood event knowledge base is calculated, that is, the semantic similarity between the keyword key and each flood event record query in the library is calculated. When performing the similarity calculation, four similarity calculations are performed: edit distance, text cosine similarity, fuzzy string matching, and sequence matching.

[0031] For each flood event record query in the global flood event knowledge base, determine the similarity between the keyword key and the flood event record query, where the similarity includes edit distance, text cosine similarity, fuzzy string matching, and sequence matching. When calculating the similarity, simultaneously calculate the edit distance LD (key, query), text cosine similarity COS (key, query), fuzzy string matching Fuzzy (key, query), and sequence matching Sequence (key, query) between the search keyword key and the flood event record query. Then obtain the preset similarity measurement weight to assign weights to the calculation results of each semantic similarity, expressed as , , , Next, the edit distance, text cosine similarity, fuzzy string matching and sequence matching are weighted and accumulated to obtain the overall similarity Similar(key, query) between the keyword and the flood event records in the global flood event knowledge base. The calculation method is expressed as the following formula (1): (1) In the above formula (1), is the edit distance The measurement weight of is the text cosine similarity The measurement weight of Fuzzy string matching The measurement weight of Sequence matching The measurement weight of , i represents the sequence number of the keyword key, represents the i-th keyword, j represents the sequence number of the flood event record query, represents the j-th flood event record.

[0032] In the actual weighted accumulation operation, the assigned metric weight can be calculated based on the result of the operation. , , , Adjustments are made and weighted accumulation operations are performed again. After multiple layers of weight adjustments, the optimal matching result is finally obtained, that is, the optimal overall similarity Similar(key, query).

[0033] The first-level search of keywords is realized based on the overall similarity. Next, the second-level search can be entered to select flood event records with high overall similarity (greater than the set threshold) from the global flood event knowledge base as semantically enhanced text. If the calculated overall similarity (less than or equal to the set threshold) is low, it means that no flood event records similar to the keyword key entered by the user can be matched from the global flood event knowledge base. Therefore, the third-level search is carried out, and the keyword key is transmitted to the Internet. The keyword key is semantically enhanced through the Internet news information included in the semantic enhancer to obtain the semantically enhanced text of the keyword. The Internet news search is used as the third-level search, and the search event sentence (news report) can be retrieved as the semantically enhanced text.

[0034] like Figure 3 As shown in the figure, during the second-level retrieval, when the overall similarity Similar(key, query) between the search keyword and the flood event records in the global flood event knowledge base is greater than the threshold T, it means that there are flood event records in the global flood event knowledge base that are semantically similar to the search keyword. Then, the flood event record query with the highest overall similarity is retrieved from the global flood event knowledge base as the semantically enhanced text of the keyword key, that is, the flood event record with the highest similarity is selected from the global flood event knowledge base as the semantically enhanced text.

[0035] When the overall similarity Similar(key, query) between the keyword and the global flood event knowledge base is less than or equal to the threshold T, it means that there is no flood event record with semantic similarity to the keyword in the global flood event knowledge base at this time, and the third-level search is performed. Internet news retrieval is performed for keywords whose overall similarity Similar(key, query) is less than or equal to the threshold T. Here, when the overall similarity Similar(key, query) is less than or equal to the set threshold, the keyword is transmitted to the Internet, and the keyword is semantically enhanced through the Internet news information included in the semantic enhancer, and the corresponding news record information (i.e., flood event record) is retrieved as the semantically enhanced text of the keyword key.

[0036] The embodiment of the present invention uses a semantic enhancer to perform multi-level retrieval using a global flood event knowledge base and Internet news information, thereby achieving multi-level semantic enhancement of user search keywords, and ultimately obtaining semantically enhanced text that semantically matches the search keywords, enriching the semantic information in the search keywords, and achieving a closer match between user needs and search queries.

[0037] Step 102: Perform quality-first screening on the satellite remote sensing images of the flood event to obtain candidate multi-source remote sensing images, and perform preliminary retrieval on the candidate multi-source remote sensing images based on the semantically enhanced text to obtain the multi-source remote sensing images to be matched.

[0038] Continue to see Figure 2 After determining the semantically enhanced text, the multi-source remote sensing images to be matched are screened based on the semantically enhanced text. Here, the satellite remote sensing images of the flood event are first screened by quality priority to obtain the candidate multi-source remote sensing images.

[0039] Specifically, first, the corresponding satellite remote sensing images can be retrieved from the cloud platform according to the search parameters such as the time, location, impact range and specific details of the flood event. Considering that the quality of these satellite remote sensing images varies, there may be factors such as cloudy and rainy images in the images, which cannot accurately describe the corresponding flood image characteristics. Therefore, it is necessary to set corresponding quality screening conditions, and prioritize the quality of the satellite remote sensing images of the flood event to obtain candidate multi-source remote sensing images, and eliminate images with poor quality. Among them, the candidate multi-source remote sensing images include optical images and radar images.

[0040] Next, the acquired radar images and optical images are preliminarily screened according to the semantically enhanced text. First, the time, location, and impact range of the flood event are determined from the semantically enhanced text as screening conditions for preliminary retrieval. In the preliminary retrieval process, multi-source remote sensing images that meet the screening conditions are retrieved from the candidate multi-source remote sensing images as the multi-source remote sensing images to be matched. Of course, the multi-source remote sensing images to be matched include optical images and radar images for subsequent retrieval.

[0041] In some embodiments, in the process of obtaining candidate multi-source remote sensing images by quality-prioritized screening of satellite remote sensing images of flood events, it is first necessary to obtain satellite remote sensing images related to flood events, but these satellite remote sensing images are cross-cloud platforms and belong to cross-platform remote sensing images. There are many types of remote sensing image data, which are divided into Landsat series satellite images (optical images) and Sentinel-1 satellite images (radar images), and there are also quality differences (such as cloudy or less cloudy). In addition, due to the different states of data provision, radar images may be offline or online in the cloud platform. To download and obtain these remote sensing image data on the cloud platform, it is necessary to use a multi-threaded method to call relevant interfaces for parallel downloading.

[0042] The embodiment of the present invention sets retrieval parameters according to the event location and duration of the flood event to retrieve satellite remote sensing images. Among the satellite remote sensing images, optical images with cloud cover below a threshold, radar images in an online state, and radar images in an offline state and activated are obtained as candidate multi-source remote sensing images.

[0043] Specifically, for satellite remote sensing images across cloud platforms, remote sensing event data acquisition tools are used for parallel acquisition. Regardless of whether it is Landsat series satellite images or Sentinel-1 satellite images, the search parameters are first set according to the location and duration of the flood event, and the remote sensing event data acquisition tool is used for retrieval to obtain satellite remote sensing images related to the flood event.

[0044] The location expansion of the central coordinates of the flood event location is used to describe the central location of the flood event and the range of diffusion from the central location, representing the spatial information of the flood event. The duration is used to represent the duration of the flood event process and represents the temporal information of the flood event. Therefore, using the spatial information and temporal information of the flood event as retrieval parameters can effectively and accurately screen the satellite remote sensing images of the flood event for easy downloading and acquisition.

[0045] Considering that the quality of these satellite remote sensing images varies, and there may be factors such as cloudy and rainy weather in the images, it is impossible to accurately describe the corresponding flood image characteristics. Therefore, it is necessary to set corresponding quality screening conditions. Here, the cloud cover threshold and online status are set to screen the candidate multi-source remote sensing images.

[0046] According to the search parameters, the satellite remote sensing images retrieved are divided into optical images and radar images. When the satellite remote sensing images are optical images, the satellite remote sensing images are screened according to the cloud threshold. Download optical images with cloud cover not exceeding the preset cloud cover threshold as candidate multi-source remote sensing images. Here, for the optical images of the Landsat series satellite images, that is, when the satellite remote sensing images are optical images, the extension and duration of the center point coordinates of the flood event are used as the spatial and temporal parameters of the search, and the Pystac tool that supports spatiotemporal asset catalog processing and data management is used as a remote sensing event data acquisition tool to realize the retrieval of Landsat satellite image data in the "landsat-c2-l2" dataset on the Microsoft Planetary Computing Platform to obtain satellite remote sensing images. The cloud cover threshold of 20% is preset as a pre-screening condition, and optical images with cloud cover not exceeding 20% ​​of the cloud cover threshold are selected and downloaded as candidate multi-source remote sensing images.

[0047] The download method can be to set chunk_size to 512 for a single cross-cloud platform remote sensing image, download it in blocks, so as to achieve breakpoint resumption and reduce IO resource consumption, and obtain multiple cross-cloud platform remote sensing images through multi-threaded concurrent access to complete efficient remote sensing image data download.

[0048] When the satellite remote sensing image is a radar image, it is necessary to determine the online status of the radar image and obtain the radar image based on the online status. Here, for the radar image of the Sentinel-1 satellite image, the position extension and duration of the center point coordinates of the flood event are also used as the spatial and temporal parameters for retrieval. The Sentinelsat API that supports the access and download of the European Space Agency (ESA) Sentinel satellite data is used as a remote sensing event data acquisition tool to implement the Sentinel-1 data retrieval provided by the Copernicus Open Access Center to obtain the satellite remote sensing image. Next, it is necessary to determine the online status of the satellite remote sensing image. Here, by parsing the Sentinel-1 satellite remote sensing data records in the retrieval results, the Online parameters in the records are used to determine whether the radar image is online.

[0049] If the online status is online, the online radar images are downloaded and obtained in parallel. Here, a data download list is constructed for the online radar images, and the online satellite remote sensing images are downloaded in parallel through the Sentinelsat API in a multi-threaded manner as candidate multi-source remote sensing images.

[0050] If the online status is offline, the radar images in the satellite remote sensing images that are offline are pre-activated, and a preset time interval (such as 40 minutes) is used to determine the activation status by asynchronous polling, that is, to determine whether it is activated. If activated, the activated radar images are added to the data download list, and the radar images with the activated activation status are downloaded in parallel as candidate multi-source remote sensing images. If not activated, they will be activated again after the time interval and wait for the next polling.

[0051] In the embodiment of the present invention, after obtaining the semantically enhanced text, the corresponding search parameters are set to retrieve the satellite remote sensing images, and the satellite remote sensing images are preliminarily screened by cloud cover and online status, which can improve the data quality of the remote sensing images and facilitate subsequent search operations based on the semantically enhanced text.

[0052] In some embodiments, after obtaining candidate multi-source remote sensing images through quality-first screening, a preliminary search is then performed on the candidate multi-source remote sensing images based on semantically enhanced text to obtain multi-source remote sensing images to be matched.

[0053] First, the spatiotemporal range information of the flood event (i.e., the diffusion range and duration of the central coordinate position of the flood event location) is extracted from the semantically enhanced text, and the spatiotemporal range information is converted into longitude and latitude coordinates as a pre-screening condition.

[0054] Here, the spatial and temporal information describing the flood event is converted into longitude and latitude coordinates using the interface provided by the map tool or map software as pre-screening conditions. Then, the acquired radar images and optical images are screened according to the pre-screening conditions to obtain the multi-source remote sensing images to be matched, which are then used for multi-modal retrieval in combination with semantically enhanced text.

[0055] like Figure 2 As shown, based on the retrieval event statement (i.e., semantically enhanced text), the satellite remote sensing images acquired in parallel are screened, and the retrieval scope is narrowed to obtain the image data to be screened (i.e., the multi-source remote sensing images to be matched), and the multi-source remote sensing images to be matched also include optical images and radar images.

[0056] In the embodiment of the present invention, after acquiring the candidate multi-source remote sensing images, the data is first preliminarily searched through the spatial information and temporal information of the flood event, thereby narrowing the search scope of the semantically enhanced text, reducing the difficulty of the search, and benefiting the accuracy of the subsequent search results.

[0057] In step 103, optical image features and radar image features of the multi-source remote sensing images to be matched are extracted based on the ResNet model, and contextual semantic features of the semantically enhanced text are extracted based on the BERT model. The multi-source remote sensing images to be matched include optical images and radar images.

[0058] After obtaining the multi-source remote sensing image to be matched through preliminary search in step 102, multimodal search is then performed in combination with the semantically enhanced text. However, before the multimodal search, the features of the multi-source remote sensing image to be matched and the semantically enhanced text need to be extracted separately.

[0059] Here, different models are called to extract corresponding features. The optical image features and radar image features of the multi-source remote sensing images to be matched are extracted based on the ResNet model, and the contextual semantic features of the semantically enhanced text are extracted based on the BERT model.

[0060] In some embodiments, for the optical image and radar image in the multi-source remote sensing image to be matched, the ResNet model is called to extract the corresponding optical image features and radar image features respectively. Figure 2 As shown in the figure, the multi-source remote sensing image to be matched is taken as the image modality, and the retrieval event sentence is taken as the text modality, which are input into the encoder and decoder of the multimodal retrieval model for feature extraction, and then the feature fusion is performed through the BERT encoder to obtain the fused feature.

[0061] Combine the following Figure 4 To explain the processing of the multimodal retrieval model in detail, Figure 4 This is a schematic diagram of the framework of the multimodal retrieval provided by the present invention. Figure 4As shown in the figure, in the multimodal retrieval model, the semantically enhanced text is used as the text data of the text modality, and the multi-source remote sensing image to be matched is used as the image data of the image modality. Here, the feature extraction of the multi-source remote sensing image is performed through the residual neural network (Residual Network, ResNet) model to obtain the multi-source remote sensing fusion feature. In the actual application environment, the multi-source remote sensing images to be matched are divided into optical images and radar images. For the optical image in the multi-source remote sensing image to be matched, the feature encoding of the optical image is performed through the ResNet model to obtain the optical image feature. For the radar image in the multi-source remote sensing image to be matched, the feature encoding of the radar image is performed through the ResNet model to obtain the radar image feature.

[0062] At the same time, for the text data of semantically enhanced text, the Bidirectional Encoder Representations from Transformers (BERT) model based on transformer is used to extract the contextual semantic features of semantically enhanced text as text modality features. This achieves the feature acquisition of multi-source remote sensing images and text modalities under the same model framework, and obtains multi-modal features based on images and text.

[0063] The embodiment of the present invention independently extracts the features of the image modality and the text modality through the ResNet model and the BERT model, thereby realizing the accurate extraction of the semantic information and the visual information of the flood time, facilitating the subsequent further matching of the semantic information and the visual information, thereby effectively improving the accuracy of the image-text matching and ensuring the effectiveness of the retrieval of the flood event keywords.

[0064] Step 104: splice the radar image features and the optical image features to obtain multi-source remote sensing fusion features, and perform multi-modal fusion of the multi-source remote sensing fusion features and the contextual semantic features based on the BERT model to obtain multi-modal fusion features.

[0065] Through step 103, after extracting multimodal features (i.e., optical image features, radar image features, and contextual semantic features) under the same model framework, the radar image features and the optical image features are first spliced ​​to obtain multi-source remote sensing fusion features, and then the multi-source remote sensing fusion features and the contextual semantic features are multimodally fused based on the BERT model to obtain multimodal fusion features, thereby realizing the fusion of multimodal features.

[0066] like Figure 4 As shown in the figure, firstly, the features of radar and optical images are fused by vector concatenation to obtain multi-source remote sensing fusion features. Then, based on the BERT model, the multi-source remote sensing fusion features and contextual semantic features are multimodally fused to obtain multimodal fusion features and output them. In this way, the fusion of multimodal features is achieved.

[0067] In some embodiments, the BERT model is used to implement multimodal fusion of multi-source remote sensing fusion features and contextual semantic features to obtain multimodal fusion features. Specifically, the BERT model is used to embed multi-source remote sensing fusion features and contextual semantic features, and then the embedded features are merged to obtain corresponding embedded multimodal features.

[0068] Next, the embedded multimodal features are fused through the BERT model to obtain multimodal fusion features. Here, the embedded multimodal features are input into the BERT model, and the embedded multimodal features are encoded through the encoder of the BERT model to obtain multimodal fusion features, thereby realizing the multimodal feature fusion of images and texts.

[0069] In an embodiment of the present invention, the embedded text modality and image modality features are encoded through a BERT encoder and mapped to the same feature space, which fully mines the potential semantic association between the image modality and the text modality, thereby realizing multimodal feature fusion to obtain consistent semantic features, which not only improves the retrieval accuracy, but also provides rich semantic support for flood event retrieval.

[0070] Step 105 , mapping the multimodal fusion features to obtain the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched, and selecting the multi-source remote sensing image to be matched with the highest similarity and the corresponding semantically enhanced text as the search result of the keyword.

[0071] like Figure 4 As shown in the figure, the BERT model outputs multimodal fusion features, and then the multimodal fusion features are mapped to obtain the label matching similarity between the semantically enhanced text and the multi-source remote sensing image to be matched.

[0072] Specifically, to achieve accurate retrieval matching, it is necessary to calculate the matching similarity. Since the multimodal fusion features have been calculated through step 104, the multimodal fusion features can be classified through a mapping module (such as a fully connected layer or a multi-layer perceptron) and mapped to the probability of a certain flood event category label. The flood event category can be a type of flood, such as various types of floods caused by waterlogging, dam breaches, or other natural or human factors in a certain area. The higher the probability corresponding to the flood event category label, the higher the label matching similarity between the semantically enhanced text and the multi-source remote sensing image to be matched, and the higher the matching similarity between the two, the more accurate the retrieval effect.

[0073] Therefore, the multi-source remote sensing images to be matched with the highest similarity and the corresponding semantically enhanced text are selected as the search results for the keyword key. Here, under the corresponding flood event category label, the multi-source remote sensing images to be matched are sorted according to the size of the label matching similarity, and the k multi-source remote sensing images to be matched with the largest label matching similarity (top-k) can be screened out as the return value of the multimodal retrieval model. This return value is the semantic retrieval result of the flood event for the keyword key, that is, the semantic retrieval result of the flood event with the k multi-source remote sensing images to be matched with the highest label matching similarity and the semantically enhanced text as the keyword key.

[0074] like Figure 2 As shown, after the multimodal fusion features are obtained by fusion of text modal features and image modal features, the fusion features are mapped and processed to output the semantic retrieval results of flood events corresponding to the keyword key. In an embodiment of the present invention, the semantic retrieval results of flood events include: news reports (i.e., retrieval event statements), optical images, and radar images as multimodal retrieval results. Among them, the news reports (i.e., retrieval event statements) can be the semantically enhanced text corresponding to the keyword key, and the optical images and radar images are multi-source remote sensing images retrieved from the multi-source remote sensing images to be matched and that best match the semantically enhanced text.

[0075] In the embodiment of the present invention, first, based on the semantic enhancer of the global flood event knowledge base and Internet news information, the search keywords are semantically enhanced for flood events. Secondly, the search text after semantic enhancement is used to perform multi-source remote sensing image quality priority screening and preliminary retrieval. Thirdly, based on the ResNet model and the BERT model, a multimodal feature extraction method combining remote sensing image features and flood event text semantics is constructed, which can obtain contextual semantic features related to flood events while obtaining deep-level image visual semantic features. Subsequently, a multimodal feature fusion method of multi-source remote sensing images and text information is constructed based on the BERT model, and feature fusion of text modal features and image modal features is performed. Finally, combined with multimodal feature similarity matching and multimodal feature mapping methods, multimodal remote sensing image retrieval combined with text semantics related to flood events is performed. This method can solve the problems of low data retrieval matching and low retrieval efficiency caused by the lack of event semantic information and the difficulty in obtaining keywords such as event-related spatiotemporal range in the retrieval of remote sensing data related to flood events.

[0076] In some embodiments, determining the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched can be achieved by mapping to a prediction label in the global flood event knowledge base, which is described in detail below.

[0077] First, the prediction label of the flood event is selected from the global flood event knowledge base. Here, a prediction label can be set according to the label matched by the global flood event knowledge base in the semantic enhancer or the common flood event label. For example, the type of flood event caused by waterlogging, dam breach or other natural or man-made factors in a certain area, the corresponding flood event type is used as the prediction label.

[0078] The multimodal fusion features are then mapped to obtain the matching similarity for the predicted labels. Here, the mapping process can map and calculate the multimodal fusion features through a linear layer (such as a fully connected layer) or a multi-layer perceptron, and output the matching similarity for the predicted labels as the label matching similarity between the semantically enhanced text and the multi-source remote sensing image to be matched.

[0079] Of course, there is more than one type of flood event, and more than one prediction label is selected. During the mapping calculation, the matching similarity of the corresponding prediction label will be output for each prediction label.

[0080] The embodiment of the present invention uses the prediction label of the flood event as a basis, determines the search results of keywords according to the multimodal fusion features, realizes the search decision of the flood event by using the multimodal joint judgment results, associates the semantic description information of the complex remote sensing event with the search conditions, enhances the semantic information density of the search statement, and realizes fast and accurate semantic retrieval of massive remote sensing data.

[0081] The multimodal retrieval device for flood events based on a semantic enhancer provided by the present invention is described below. The multimodal retrieval device for flood events based on a semantic enhancer described below and the multimodal retrieval method for flood events based on a semantic enhancer described above can be referenced to each other.

[0082] See also Figure 5 , Figure 5 is a structural schematic diagram of a flood event multimodal retrieval device based on a semantic enhancer provided by the present invention, such as Figure 5As shown, the multimodal retrieval device for flood events based on semantic enhancer includes: semantic enhancement module 501, feature extraction module 502, feature fusion module 503, and multimodal retrieval module 504. Among them, the semantic enhancement module 501 is used to perform multi-level semantic enhancement on the keywords input by the user through the global flood event knowledge base and Internet news information included in the semantic enhancer to obtain semantically enhanced text; the feature extraction module 502 is used to perform quality-prioritized screening on the satellite remote sensing images of the flood events to obtain candidate multi-source remote sensing images, and perform preliminary retrieval on the candidate multi-source remote sensing images based on the semantically enhanced text to obtain multi-source remote sensing images to be matched, and the multi-source remote sensing images to be matched include optical images and radar images; the optical image features and radar image features of the multi-source remote sensing images to be matched are extracted based on the ResNet model. Features, extracting contextual semantic features of the semantically enhanced text based on the BERT model; a feature fusion module 503, used to splice the radar image features with the optical image features to obtain multi-source remote sensing fusion features, and multi-modally fuse the multi-source remote sensing fusion features and the contextual semantic features based on the BERT model to obtain multi-modal fusion features; a multi-modal retrieval module 504, used to map the multi-modal fusion features to obtain the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched, and select the multi-source remote sensing image to be matched with the highest similarity and the corresponding semantically enhanced text as the retrieval result of the keyword.

[0083] It should be noted that the beneficial effects of the multimodal retrieval device for flood events based on the semantic enhancer here and the multimodal retrieval method for flood events based on the semantic enhancer mentioned above can correspond to each other, so the beneficial effects of the multimodal retrieval device for flood events based on the semantic enhancer will not be repeated here.

[0084] Figure 6 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute a multimodal retrieval method for flood events based on a semantic enhancer, the method comprising: for keywords input by a user, multi-level semantic enhancement is performed on the keywords through the global flood event knowledge base and Internet news information included in the semantic enhancer to obtain semantically enhanced text; satellite remote sensing images of flood events are screened with priority on quality to obtain candidate multi-source remote sensing images, and the candidate multi-source remote sensing images are preliminarily retrieved based on the semantically enhanced text to obtain multi-source remote sensing images to be matched, the multi-source remote sensing images to be matched include optical images and radar images; based on the ResNet model The optical image features and radar image features of the multi-source remote sensing image to be matched are extracted, and the contextual semantic features of the semantically enhanced text are extracted based on the BERT model; the radar image features are spliced ​​with the optical image features to obtain multi-source remote sensing fusion features, and the multi-source remote sensing fusion features and the contextual semantic features are multimodally fused based on the BERT model to obtain multimodal fusion features; the multimodal fusion features are mapped to obtain the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched, and the multi-source remote sensing image to be matched with the highest similarity and the corresponding semantically enhanced text are selected as the search results of the keywords.

[0085] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0086] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal retrieval method for flood events based on a semantic enhancer provided by the above methods. The method includes: for keywords input by a user, multi-level semantic enhancement is performed on the keywords through a global flood event knowledge base and Internet news information included in the semantic enhancer to obtain semantically enhanced text; satellite remote sensing images of flood events are screened with quality priority to obtain candidate multi-source remote sensing images, and the candidate multi-source remote sensing images are preliminarily retrieved based on the semantically enhanced text to obtain multi-source remote sensing images to be matched. The multi-source remote sensing images to be matched include optical images and radar images; the optical image features and radar image features of the multi-source remote sensing images to be matched are extracted based on the ResNet model, and the contextual semantic features of the semantically enhanced text are extracted based on the BERT model; the radar image features and the optical image features are spliced ​​to obtain multi-source remote sensing fusion features, and the multi-source remote sensing fusion features and the contextual semantic features are multimodally fused based on the BERT model to obtain multimodal fusion features; the multimodal fusion features are mapped to obtain the similarity between the semantically enhanced text and the multi-source remote sensing images to be matched, and the multi-source remote sensing images to be matched with the highest similarity and the corresponding semantically enhanced text are selected as the search results of the keywords.

[0087] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal retrieval method for flood events based on a semantic enhancer provided by the above-mentioned methods, the method comprising: for keywords input by a user, performing multi-level semantic enhancement on the keywords through a global flood event knowledge base and Internet news information included in the semantic enhancer to obtain semantically enhanced text; performing quality-prioritized screening on satellite remote sensing images of flood events to obtain candidate multi-source remote sensing images, and performing a preliminary search on the candidate multi-source remote sensing images based on the semantically enhanced text to obtain multi-source remote sensing images to be matched, the multi-source remote sensing images to be matched comprising optical images and radar images; extracting optical image features and radar image features of the multi-source remote sensing image to be matched based on the ResNet model, and extracting contextual semantic features of the semantically enhanced text based on the BERT model; splicing the radar image features with the optical image features to obtain multi-source remote sensing fusion features, and performing multimodal fusion on the multi-source remote sensing fusion features and the contextual semantic features based on the BERT model to obtain multimodal fusion features; mapping the multimodal fusion features to obtain the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched, and selecting the multi-source remote sensing image to be matched with the highest similarity and the corresponding semantically enhanced text as the retrieval results of the keywords.

[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal retrieval method for flood events based on semantic enhancer, characterized in that: The method comprises: For the keywords input by the user, multi-level semantic enhancement is performed on the keywords through the global flood event knowledge base and Internet news information included in the semantic enhancer to obtain semantically enhanced text; The satellite remote sensing images of the flood event are screened by quality priority to obtain candidate multi-source remote sensing images, and the candidate multi-source remote sensing images are preliminarily retrieved based on the semantically enhanced text to obtain multi-source remote sensing images to be matched, wherein the multi-source remote sensing images to be matched include optical images and radar images; Extracting optical image features and radar image features of the multi-source remote sensing image to be matched based on the ResNet model, and extracting contextual semantic features of the semantically enhanced text based on the BERT model; The radar image feature and the optical image feature are spliced ​​to obtain a multi-source remote sensing fusion feature, and the multi-source remote sensing fusion feature and the context semantic feature are multimodally fused based on the BERT model to obtain a multimodal fusion feature; The multimodal fusion features are mapped to obtain the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched, and the multi-source remote sensing image to be matched with the highest similarity and the corresponding semantically enhanced text are selected as the search results of the keywords.

2. The multimodal retrieval method for flood events based on semantic enhancer according to claim 1 is characterized in that: The semantic enhancer includes a global flood event knowledge base and Internet news information, and performs multi-level semantic enhancement on the keywords to obtain semantically enhanced text, including: In the first stage, the overall similarity between the keyword and the flood event records in the global flood event knowledge base included in the semantic enhancer is calculated, and the overall similarity is obtained by weighted calculation of edit distance, text cosine similarity, fuzzy matching similarity and sequence matching similarity; In the second stage, when the overall similarity is higher than a set threshold, the flood event record with the highest overall similarity is retrieved from the global flood event knowledge base as the semantically enhanced text; In the third stage, when the overall similarity is lower than or equal to the set threshold, the keywords are semantically enhanced by the Internet news information included in the semantic enhancer to obtain a semantically enhanced text.

3. The multimodal retrieval method for flood events based on semantic enhancer according to claim 1 is characterized in that: The satellite remote sensing images of flood events are screened by quality priority to obtain candidate multi-source remote sensing images, including: According to the location and duration of the flood event, corresponding search parameters are set to retrieve satellite remote sensing images; Among the satellite remote sensing images, optical images with cloud cover below a threshold, radar images in an online state, and radar images in an offline state and activated are obtained as candidate multi-source remote sensing images.

4. The multimodal retrieval method for flood events based on semantic enhancer according to claim 1 is characterized in that: The preliminary retrieval of the candidate multi-source remote sensing images based on the semantically enhanced text to obtain the multi-source remote sensing images to be matched includes: Extracting spatiotemporal range information from the semantically enhanced text, and converting the spatiotemporal range information into longitude and latitude coordinates; The latitude and longitude coordinates are used as pre-screening conditions, and the candidate multi-source remote sensing images are screened according to the pre-screening conditions to obtain the multi-source remote sensing images to be matched.

5. The multimodal retrieval method for flood events based on semantic enhancer according to claim 1 is characterized in that: The multi-modal fusion of the multi-source remote sensing fusion features and the contextual semantic features based on the BERT model to obtain the multi-modal fusion features includes: Calling the BERT model to embed the multi-source remote sensing fusion features and the contextual semantic features to obtain corresponding embedded multimodal features; The embedded multimodal features are subjected to multimodal feature fusion through the BERT model to obtain multimodal fusion features.

6. The multimodal retrieval method for flood events based on semantic enhancer according to claim 1 is characterized in that: The mapping process is performed on the multimodal fusion features to obtain the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched, including: Selecting a prediction label of a flood event from the global flood event knowledge base; The multimodal fusion features are mapped to obtain a matching similarity for the predicted label, and the matching similarity is used as a label matching similarity between the semantically enhanced text and the multi-source remote sensing image to be matched.

7. A multimodal retrieval device for flood events based on semantic enhancer, characterized in that: The device comprises: A semantic enhancement module, for performing multi-level semantic enhancement on keywords input by a user through a global flood event knowledge base and Internet news information included in a semantic enhancer, to obtain semantically enhanced text; A feature extraction module is used to perform quality-first screening on the satellite remote sensing images of the flood event to obtain candidate multi-source remote sensing images, and to perform preliminary retrieval on the candidate multi-source remote sensing images based on the semantically enhanced text to obtain multi-source remote sensing images to be matched, wherein the multi-source remote sensing images to be matched include optical images and radar images; Extracting optical image features and radar image features of the multi-source remote sensing image to be matched based on the ResNet model, and extracting contextual semantic features of the semantically enhanced text based on the BERT model; A feature fusion module, used for splicing the radar image feature with the optical image feature to obtain a multi-source remote sensing fusion feature, and performing multi-modal fusion on the multi-source remote sensing fusion feature and the contextual semantic feature based on a BERT model to obtain a multi-modal fusion feature; The multimodal retrieval module is used to map the multimodal fusion features to obtain the similarity between the semantically enhanced text and the multi-source remote sensing image to be matched, and select the multi-source remote sensing image to be matched with the highest similarity and the corresponding semantically enhanced text as the retrieval result of the keyword.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal retrieval method for flood events based on a semantic enhancer as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal retrieval method for flood events based on a semantic enhancer as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal retrieval method for flood events based on a semantic enhancer as described in any one of claims 1 to 6 is implemented.

Images