Disaster scenario-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method

Abstract

The application relates to the technical field of image-text retrieval, in particular to a disaster-scene-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method, which comprises the following steps: respectively preprocessing acquired paired image sample data and text sample data, and constructing a training set; constructing a coarse-grained cross-modal alignment module, wherein the coarse-grained cross-modal alignment module is composed of an image encoder containing a domain adapter and a text encoder containing a domain adapter; constructing a fine-grained difficult negative sample mining module, wherein the fine-grained difficult negative sample mining module is composed of a momentum image encoder, a momentum text encoder and a corresponding momentum queue; inputting the training set into an unmanned aerial vehicle image-text retrieval model, training and optimizing the unmanned aerial vehicle image-text retrieval model to convergence by using a hybrid momentum loss function, and obtaining a trained model; and inputting paired image data and text data of an unmanned aerial vehicle to be identified into the trained model, so as to obtain unmanned aerial vehicle image-text retrieval results.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image-text retrieval, and in particular to a disaster-scene-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method. BACKGROUND

[0002] In recent years, unmanned aerial vehicle technology has ushered in a breakthrough, and its high-resolution optical, thermal infrared and multispectral sensing equipment makes large-scale and high-frequency data collection possible. This directly leads to an exponential explosion of geospatial data. At the same time, unmanned aerial vehicles, with their unique advantages of high flexibility and low cost, have rapidly expanded in the civilian sector, playing an increasingly important role in many fields such as large-scale environmental dynamic monitoring, precision agriculture of crop growth and pest assessment, urban planning management, and infrastructure inspection, and have become a key tool for driving these fields towards digital and intelligent management.

[0003] With the explosive growth of unmanned aerial vehicle data, the demand for analysis capabilities of unmanned aerial vehicle cross-modal data has also increased day by day. However, due to the inherent tilted view and low-altitude flight mode of unmanned aerial vehicles, a large number of difficult-to-distinguish negative samples (referred to as negative samples) are formed, which poses a formidable challenge to cross-modal data understanding and analysis in the unmanned aerial vehicle scene. This bottleneck problem is dramatically magnified in the disaster rescue scene, which is extremely demanding in terms of timeliness, and is particularly serious. In the face of these valuable but complex information, rescue command centers often have difficulty in extracting key decision-making information such as disaster-affected areas, building damage levels, road conditions, and trapped personnel locations in a timely and accurate manner. This lag in analysis capability not only may lead to a mismatch of rescue resources, but also may directly delay the best opportunity to save lives, so that the advantages brought by advanced data acquisition technology are difficult to translate into actual rescue effectiveness at critical moments.

[0004] To break this data analysis bottleneck, cross-modal image-text retrieval technology is highly expected, whose core mission is to build an intelligent association between natural language and visual content. The methods proposed so far can be divided into traditional methods and CLIP model (Contrastive Language-Image Pre-Training) based methods. Traditional methods dominated the early stage of cross-modal retrieval. Such methods often use ResNet (Residual Network) and BERT (Bidirectional Encoder Representations from Transformers) as visual and text feature extractors, respectively. Their performance is often subject to training data, resulting in poor performance in professional application fields. CLIP model based methods often have better performance due to the powerful generalization ability of CLIP model. However, they lack a targeted feature extraction mechanism for data captured by unmanned aerial vehicles in disaster scenarios, especially the processing means for difficult negative samples that severely affect the cross-modal retrieval ability in such scenarios, resulting in limited application performance in disaster scenarios. Specifically, these methods usually rely on a rough training strategy, that is, in the training batch, all non-paired image-text pairs are simply treated as negative samples, and they try to separate them from the query anchor in the hidden space. This strategy results in a significant lack of discrimination ability when facing difficult negative samples. As the model cannot perceive subtle semantic differences from this coarse-grained supervision signal, it is difficult for the model to focus on truly key discriminative features, resulting in a large number of ambiguous boundaries and confusion areas in the final learned joint embedding space, which directly restricts the stability of model convergence and the upper limit of final retrieval accuracy. SUMMARY

[0005] Therefore, the embodiments of the present application propose a disaster scene oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method, which can effectively overcome the problem of difficult negative samples being difficult to distinguish due to the chaotic characteristics of unmanned aerial vehicle flight modes and disaster scenes, thereby effectively improving the cross-modal retrieval ability and accuracy of the model.

[0006] To achieve the above object, the embodiment of the present application proposes a disaster scene-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method, which is realized based on a coarse-to-fine progressive discrimination framework, and comprises the following steps: preprocessing acquired paired image sample data and text sample data respectively, and constructing a training set based on the preprocessed image sample data and text sample data; constructing a coarse-grained cross-modal alignment module; wherein the coarse-grained cross-modal alignment module is composed of an image encoder containing a domain adapter and a text encoder containing a domain adapter, and is used to realize preliminary alignment in a regular scene by using domain adaptation technology, thereby providing a good feature space basis for difficult negative sample mining; constructing a fine-grained difficult negative sample mining module; wherein the fine-grained difficult negative sample mining module is composed of a momentum image encoder, a momentum text encoder and a corresponding momentum queue, and is used to gradually distinguish difficult negative samples with high similarity from false negative samples, thereby guiding learning of more discriminative feature representations; inputting the image sample data and text sample data in the training set into the unmanned aerial vehicle image-text retrieval model composed of the coarse-grained cross-modal alignment module and the fine-grained difficult negative sample mining module, training and optimizing the same to convergence using a hybrid momentum loss function, thereby obtaining a trained unmanned aerial vehicle image-text retrieval model; inputting paired image data and text data of an unmanned aerial vehicle to be identified into the trained unmanned aerial vehicle image-text retrieval model, thereby obtaining corresponding unmanned aerial vehicle image-text retrieval results.

[0007] To achieve the above object, the embodiment of the present application also proposes an unmanned aerial vehicle image-text retrieval system for disaster scene-oriented negative sample mining enhancement, which is realized based on the unmanned aerial vehicle image-text retrieval method for disaster scene-oriented negative sample mining enhancement as described above. The system comprises: a training set construction module, configured to respectively pre-process the obtained paired image sample data and text sample data, and construct a training set based on the pre-processed image sample data and text sample data; a model construction module, configured to construct an unmanned aerial vehicle image-text retrieval model composed of a coarse-grained cross-modal alignment module and a fine-grained difficult negative sample mining module, wherein the coarse-grained cross-modal alignment module is composed of an image encoder comprising a domain adapter and a text encoder comprising a domain adapter, configured to realize preliminary alignment in a regular scene by using a domain adaptation technology, and provide a good feature space basis for difficult negative sample mining, and the fine-grained difficult negative sample mining module is composed of a momentum image encoder, a momentum text encoder and a corresponding momentum queue, configured to gradually distinguish difficult negative samples with high similarity from false negative samples, and guide learning of more discriminative feature representations; a model training module, configured to input the image sample data and the text sample data in the training set into the unmanned aerial vehicle image-text retrieval model, and train and optimize the same to convergence using a hybrid momentum loss function, to obtain a trained unmanned aerial vehicle image-text retrieval model; and a model using module, configured to input paired image data and text data of an unmanned aerial vehicle to be identified into the trained unmanned aerial vehicle image-text retrieval model, to obtain corresponding unmanned aerial vehicle image-text retrieval results.

[0008] To achieve the above object, the embodiment of the present application also proposes an electronic device, comprising: a processor and a memory, the memory storing instructions executable by the processor, and the processor being configured to execute the instructions so that the electronic device can implement the unmanned aerial vehicle image-text retrieval method for disaster scene-oriented negative sample mining enhancement as described above.

[0009] To achieve the above object, the embodiment of the present application also proposes a computer-readable storage medium storing a computer program, the computer program being executable by a processor to implement the unmanned aerial vehicle image-text retrieval method for disaster scene-oriented negative sample mining enhancement as described above.

[0010] Optionally, the obtained paired image sample data and text sample data are respectively pre-processed, and a training set is constructed based on the pre-processed image sample data and text sample data, comprising:

[0011] Obtain an unmanned aerial vehicle image-text retrieval dataset for disaster scene, and the unmanned aerial vehicle image-text retrieval dataset comprises a plurality of paired image sample data and text sample data;

[0012] The resolution of all image sample data in the unmanned aerial vehicle image-text retrieval dataset is uniformly scaled to a preset standard resolution to obtain preprocessed image sample data.

[0013] All text sample data in the unmanned aerial vehicle image-text retrieval dataset is uniformly segmented and encoded to obtain preprocessed text sample data.

[0014] Based on the pair of preprocessed image sample data and preprocessed text sample data, the training set, test set and validation set are obtained by dividing according to a preset proportion.

[0015] Optionally, for the image encoder, the domain adapter is connected after the TransformerBlock of the original image encoder, and the two together constitute the same level of the image encoder, and for the text encoder, the domain adapter is connected after the Transformer Block of the original text encoder, and the two together constitute the same level of the text encoder.

[0016] The dimension change of the domain adapter follows a U-shaped structure of first reducing and then increasing, for realizing extraction and expression of deep features, and the domain adapter specifically consists of a dimension-reducing Linear layer, a GELU activation function, an LN layer normalization, an FC Layer full connection layer, a dimension-increasing Linear layer and a skip connection.

[0017] The domain adapter of the image encoder and the domain adapter of the text encoder share the parameters of the GELU activation function, the LN layer normalization and the FC Layer full connection layer, and the parameters of the dimension-reducing Linear layer and the dimension-increasing Linear layer are designed separately and not shared.

[0018] Optionally, the output of the image encoder is represented by the formula:

[0019] ;

[0020] ;

[0021] ;

[0022] wherein, represents the input of the image encoder, represents the dimension-reducing Linear layer of the domain adapter of the image encoder, represents the GELU activation function, represents the LN layer normalization, represents the FC Layer full connection layer, represents the dimension-increasing Linear layer of the domain adapter of the image encoder, an output of a dimension reduction Linear layer of a domain adapter of the image encoder, an output of a FC Layer of the image encoder, a result after the skip connection, an output of the image encoder;

[0023] an output of the text encoder is expressed by the following formula:

[0024] ;

[0025] ;

[0026] ;

[0027] wherein, an input of the text encoder, and a dimension reduction Linear layer and a dimension increase Linear layer of a domain adapter of the text encoder, an output of the dimension reduction Linear layer of the domain adapter of the text encoder, an output of a FC Layer of the domain adapter of the text encoder, a result after the skip connection, an output of the text encoder.

[0028] Optionally, the output of the image encoder is an input of a momentum image encoder, after the output of the image encoder is processed by the momentum image encoder, the output enters an image momentum queue to expand the perception ability to difficult negative samples by a larger batch, the output of the text encoder is an input of a momentum text encoder, after the output of the text encoder is processed by the momentum text encoder, the output enters a text momentum queue to expand the perception ability to difficult negative samples by a larger batch, and a size of the image momentum queue and a size of the text momentum queue are much larger than a single batch of the UAV image-text retrieval model.

[0029] Optionally, weights of the momentum image encoder and weights of the momentum text encoder are updated by the following formula:

[0030] ;

[0031] ;

[0032] wherein, is the weight of the momentum image encoder before updating, is the weight of the image encoder, is the weight of the momentum image encoder after updating, the weights of the momentum text encoder before the update, the weights of the text encoder, the weights of the momentum text encoder after the update, the preset balancing coefficient.

[0033] Optionally, the hybrid momentum loss function is represented by a formula as:

[0034] ;

[0035] ;

[0036] ;

[0037] wherein, denotes the hybrid momentum loss function, denotes a loss function of an image-to-text retrieval task, denotes a loss function of a text-to-image retrieval task, denotes a batch size, denotes a cross-entropy loss, denotes an image index, denotes a text index, a right lower subscript denotes a momentum matching probability of the image-to-text, denotes the image-to-text matching probability of the i-th sample data, a right lower subscript denotes a momentum matching probability of the text-to-image, denotes the text-to-image matching probability of the i-th sample data, a right lower subscript denotes a hybrid momentum label of the i-th sample data, denotes the hybrid momentum label of the i-th sample data, a right lower subscript denotes a momentum matching probability of the image-to-text, denotes the momentum matching probability of the image-to-text of the i-th sample data, a right lower subscript denotes a momentum matching probability of the text-to-image, denotes the momentum matching probability of the text-to-image of the i-th sample data, a right lower subscript

[0038] The embodiment of the present application proposes a disaster scene-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method. In the face of the problem that the traditional contrast learning method is inaccurate in disaster unmanned aerial vehicle scene and is prone to cause semantic confusion of the model, a unmanned aerial vehicle image-text retrieval model composed of a coarse-grained cross-modal alignment module and a fine-grained difficult negative sample mining module is designed, constructed, trained and used to perform the unmanned aerial vehicle image-text retrieval task. The coarse-grained cross-modal alignment module is used to realize preliminary alignment in the conventional scene by using the domain adaptation technology, and to provide a good feature space basis for subsequent difficult negative sample mining. Then, the fine-grained difficult negative sample mining module is used to gradually distinguish the difficult negative samples with high similarity from the false negative samples by constructing and using a specific screening mechanism, and to guide the learning of more discriminative feature representations. Through this progressive discrimination framework from coarse to fine, the influence of the chaotic characteristics of the unmanned aerial vehicle flight mode and the disaster scene can be effectively overcome, thereby effectively improving the cross-modal retrieval capability and retrieval precision of the unmanned aerial vehicle image-text retrieval model. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the following drawings are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings. The drawings described herein are only used to explain the present application and do not limit the present application.

[0040] Figure 1 is a flowchart of a disaster scene-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method provided in an embodiment of the present application;

[0041] Figure 2 is an iconic schematic diagram of a disaster scene-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method provided in an embodiment of the present application;

[0042] Figure 3 is a structure diagram of a unmanned aerial vehicle image-text retrieval model provided in an embodiment of the present application;

[0043] Figure 4 is a structure diagram of a disaster scene-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval system provided in another embodiment of the present application;

[0044] Figure 5 is a structure diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. Those skilled in the art can understand that, in the embodiments of the present application, many technical details are proposed in order to make the readers better understand. However, the technical solutions claimed by the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation modes of the present application. The following embodiments can be combined and referenced with each other without contradiction.

[0046] One embodiment of the present application proposes a disaster scenario-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method, which is realized based on a gradual discriminant framework from coarse to fine. The implementation details of the disaster scenario-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method proposed in the embodiment will be described in detail below. The following content only provides implementation details for the convenience of understanding, and is not essential for implementing the present solution.

[0047] The specific process of the disaster scenario-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method proposed in the embodiment can be as shown in Figure 1 The specific process of the disaster scenario-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method proposed in the embodiment can be as shown in Figure 2 The method comprises the following steps.

[0048] In step 11, the obtained paired image sample data and text sample data are preprocessed respectively, and a training set is constructed based on the preprocessed image sample data and text sample data.

[0049] In a specific implementation, whether there are sufficient training samples is very important for model training. Therefore, before model training, a large amount of paired image sample data and text sample data need to be obtained. The sources of these data are different, and the resolutions and qualities are uneven. Preprocessing including resolution normalization is needed before they can be used. After the preprocessing is completed, a training set can be constructed based on the preprocessed image sample data and text sample data (a test set and a validation set can also be constructed at the same time).

[0050] In one example, the server needs to obtain a disaster scenario-oriented unmanned aerial vehicle image-text retrieval dataset, such as the public unmanned aerial vehicle image-text retrieval dataset ERA dataset and UDV dataset. The ERA dataset contains 2864 images, the image resolution is 640pxx640px, and each image corresponds to five English text descriptions.

[0051] The preprocessing of the image sample data is to uniformly scale the resolution of all image sample data in the unmanned aerial vehicle image-text retrieval dataset to a preset standard resolution (such as 224pxx224px), to obtain preprocessed image sample data. The preprocessing of the text sample data is to uniformly segment and encode all text sample data in the unmanned aerial vehicle image-text retrieval dataset, to obtain preprocessed text sample data.

[0052] Finally, based on the pair of preprocessed image sample data and preprocessed text sample data, a training set, a test set and a validation set are obtained according to a preset proportion (which can be set according to actual needs). For the ERA dataset, 10260 pairs of image-text data are used as the training set, 2570 pairs of image-text data are used as the validation set, and 1480 pairs of image-text data are used as the test set.

[0053] Step 12, constructing a coarse-grained cross-modal alignment module, wherein the coarse-grained cross-modal alignment module is composed of an image encoder containing a domain adapter and a text encoder containing a domain adapter, and is used to realize preliminary alignment in a regular scene by using a domain adaptation technology, and to provide a good feature space basis for difficult negative sample mining.

[0054] In a specific implementation, the two cores of the progressive discriminative framework from coarse to fine are a coarse-grained cross-modal alignment module and a fine-grained difficult negative sample mining module. As shown in Figure 3 The coarse-grained cross-modal alignment module is composed of an image encoder containing a domain adapter and a text encoder containing a domain adapter, and the coarse-grained cross-modal alignment module is used to realize preliminary alignment in a regular scene by using a domain adaptation technology, and to provide a good feature space basis for difficult negative sample mining.

[0055] In one example, as shown in Figure 3 For the image encoder, the domain adapter is connected after the Transformer Block of the original image encoder, and the two together constitute the same level of the image encoder. For the text encoder, the domain adapter is connected after the Transformer Block of the original text encoder, and the two together constitute the same level of the text encoder.

[0056] The dimension change of the domain adapter follows a U-shaped structure of first reducing and then increasing, which is used to realize the extraction and expression of deep features. The domain adapter is specifically composed of a dimension reduction Linear layer, a GELU activation function, an LN layer normalization, an FC Layer full connection layer, a dimension increase Linear layer and two skip connections. The first skip connection is specifically connected between the output of the dimension reduction Linear layer and the output of the FC Layer full connection layer. The second skip connection is specifically connected between the input and the output of the dimension increase Linear layer.

[0057] The domain adapters of the image encoder and the text encoder share the GELU activation function, LN layer normalization, and FC layer parameters. The parameters of the dimension-reduced Linear layer and the dimension-upgraded Linear layer are designed separately and are not shared.

[0058] In one example, the output of the image encoder is expressed by the formula:

[0059] ;

[0060] ;

[0061] ;

[0062] in, This represents the input to the image encoder. This represents a dimension-reduced linear layer in the neighborhood adapter of an image encoder. This represents the GELU activation function. Indicates LN layer normalization, Indicates FC Layer (Fully Connected Layer). This represents an upgraded linear layer in the domain adapter of an image encoder. This represents the output of the dimension-reduced linear layer of the neighborhood adapter of the image encoder. The output of the FC Layer fully connected layer representing the domain adapter of the image encoder and The result after skip connections, This represents the output of the image encoder.

[0063] , The dimension is space, For sample batches, The feature dimension of the input feature. , The dimension is space, The compressed feature dimensions, In Under the influence of , it is projected onto a low-dimensional representation space. , The dimension is The space. , , The dimension is space, exist under the action of the dimensionality increasing layer, .

[0064] In one example, the output of the text encoder is represented by the formula:

[0065] ;

[0066] ;

[0067] ;

[0068] wherein, represents the input of the text encoder, and represent the dimensionality reducing Linear layer and the dimensionality increasing Linear layer of the domain adapter of the text encoder, respectively, represents the output of the dimensionality reducing Linear layer of the domain adapter of the text encoder, represents the output of the FC Layer of the domain adapter of the text encoder after the jump connection, and represents the output of the text encoder.

[0069] Step 13, constructing a fine-grained difficult negative sample mining module, wherein the fine-grained difficult negative sample mining module is composed of a momentum image encoder, a momentum text encoder and a corresponding momentum queue, and is used to gradually distinguish difficult negative samples with high similarity from false negative samples, and guide learning of more discriminative feature representations.

[0070] In a specific implementation, the fine-grained difficult negative sample mining module is composed of a momentum image encoder, a momentum text encoder and a corresponding momentum queue, and the fine-grained difficult negative sample mining module is used to gradually distinguish difficult negative samples with high similarity from false negative samples, and guide learning of more discriminative feature representations. The difficult negative sample refers to a training sample that is very similar to the positive sample (real target) in appearance, features or context in tasks such as target detection and image-text retrieval, but actually belongs to a negative sample (background or other categories). Since the difficult negative sample is easy to be misjudged as a positive sample by the model, it poses a challenge to model training. The false negative sample refers to a training sample that is actually a positive sample related to the query or anchor sample in tasks such as target detection and image-text retrieval, but is incorrectly labeled as a negative sample.

[0071] In one example, as Figure 3As shown, the output of the image encoder is the input of the momentum image encoder. After being processed by the momentum image encoder, the output of the image encoder enters the image momentum queue to expand the perception capability of hard-to-bear samples through larger batches. The output of the text encoder is the input of the momentum text encoder. After being processed by the momentum text encoder, the output of the text encoder enters the text momentum queue to expand the perception capability of hard-to-bear samples through larger batches. The size of the image momentum queue and the text momentum queue are much larger than the single batch of the UAV image-text retrieval model.

[0072] In one example, the weights of the momentum image encoder and the momentum text encoder are updated using the following formula:

[0073] ;

[0074] ;

[0075] in, The weights before the momentum image encoder update. For the weights of the image encoder, The weights are updated by the momentum image encoder. These are the weights before the momentum text encoder update. For the weights of the text encoder, The updated weights of the momentum text encoder This is the preset balance coefficient.

[0076] In one example, the input to the fine-grained hard-to-react sample mining module is denoted as image information. and text information , , , For dimension space, For dimension space, and These represent the batch size and feature dimension of the image information, respectively. and These represent the batch size and feature dimension of the text information, respectively. and After processing by the momentum encoder, the image momentum queues can be obtained respectively. and text momentum queue , , , For dimension space, The queue size is typically much larger than the batch size. .

[0077] Step 14: Input the image sample data and text sample data in the training set into the UAV image and text retrieval model, which consists of a coarse-grained cross-modal alignment module and a fine-grained hard negative sample mining module. Use the hybrid momentum loss function to train and optimize it until convergence, and obtain the trained UAV image and text retrieval model.

[0078] In the specific implementation, after the construction of the coarse-grained cross-modal alignment module and the fine-grained hard-negative sample mining module is completed, the image sample data and text sample data in the training set can be input into the UAV image-text retrieval model composed of the coarse-grained cross-modal alignment module and the fine-grained hard-negative sample mining module. The hybrid momentum loss function is used to train and optimize it until convergence, thereby obtaining the trained UAV image-text retrieval model.

[0079] In one example, the hybrid momentum loss function is constructed based on category-aware labels, which are first constructed using the following formula:

[0080] ;

[0081] in, Indicates the first The image sample data and the first Category-aware labels for text sample data For image index (the first) (image sample data) For text index (the first) (one text sample data) For indicator functions, and These are categories for images and categories for text. This represents logical AND. Indicates proportional to, The weights are assigned to the categories.

[0082] when When the text and image are completely corresponding, it has the highest weight of 1.

[0083] when In this case, it means that the text and image only correspond to the category, but not the instance, so the corresponding weight is only... .

[0084] when When the image and text do not correspond at all, the weight is 0.

[0085] Based on category-aware labels, hybrid momentum labels are constructed using the following formula:

[0086] ;

[0087] wherein, is the category-aware label (omitting the subscript), is the single-value label, single-value label weight, is the hybrid momentum label.

[0088] In one example, the hybrid momentum loss function is represented by the formula:

[0089] ;

[0090] ;

[0091] ;

[0092] wherein, denotes the hybrid momentum loss function, denotes the loss function for the image-to-text retrieval task, denotes the loss function for the text-to-image retrieval task, is the batch size, denotes the cross-entropy loss, is the image index (omitting the subscript), is the text index (omitting the subscript), the right lower subscript denotes the for the sample data, the right lower subscript denotes the for the sample data, denotes the for the sample data, the right lower subscript denotes the for the sample data, the right lower subscript denotes the image-to-text momentum matching probability, denotes the text-to-image momentum matching probability.

[0093] Step 15, input the image data and text data of the pair of unmanned aerial vehicles to be identified into the trained unmanned aerial vehicle image-text retrieval model to obtain the corresponding unmanned aerial vehicle image-text retrieval result.

[0094] In a specific implementation, after obtaining the trained unmanned aerial vehicle image-text retrieval model, it can be deployed where needed, and then the image data and text data of the pair of unmanned aerial vehicles to be identified are input into the trained unmanned aerial vehicle image-text retrieval model to obtain the corresponding unmanned aerial vehicle image-text retrieval result.

[0095] Therefore, the disaster scene-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method proposed in this embodiment can overcome the influence of the chaotic characteristics of the unmanned aerial vehicle flight mode and the disaster scene, thereby effectively improving the cross-modal retrieval capability and retrieval precision of the unmanned aerial vehicle image-text retrieval model.

[0096] The step division of the above methods is only for the purpose of clearly describing the technical solutions, and in actual implementation, one step can be combined or some steps can be decomposed into multiple steps, as long as the same logical relationship is included, and all are within the protection scope of the present application. Any insignificant modification or introduction of insignificant design in the algorithm or process, as long as the core of the algorithm and process is not changed, is within the protection scope of the present application.

[0097] In one embodiment, in order to verify the effectiveness of the disaster scene-oriented negative sample mining enhanced unmanned aerial vehicle image-text retrieval method (OURS or the method hereinafter) proposed in the present application, we performed a simulation experiment, the software framework of the simulation experiment was PyTorch1.8, and the hardware platform was 4*RTX3090 graphics cards. We performed experiments on the unmanned aerial vehicle disaster data set proposed by Huang et al., compared the performance between OURS and other reconstruction methods, and used R@1, R@5 and R@10 indexes, and the results are shown in Table 1.

[0098] Table 1: Performance index experimental results of the method and other image-text retrieval methods

[0099]

[0100] According to the results of Table 1, the method has advantages in text retrieval, image retrieval, and average retrieval accuracy compared to other image-text retrieval methods. This reflects the driving effect of the organic combination of the proposed lightweight domain adapter and difficult negative sample mining strategy on the unmanned aerial vehicle image-text retrieval task. The former provides a good cross-domain learning basis for the model, promoting the model to effectively recognize the key semantic targets in the disaster scene, and the latter further refines this recognition, strengthens the model's ability to distinguish difficult negative samples that are easily confused, and thus improves the overall retrieval accuracy of the model. The performance in this data particularly reflects the superior performance of the method in the unmanned aerial vehicle disaster scene.

[0101] Another embodiment of the present application proposes an unmanned aerial vehicle image-text retrieval system for disaster scene-oriented negative sample mining enhancement, which is implemented based on the unmanned aerial vehicle image-text retrieval method for disaster scene-oriented negative sample mining enhancement as described in the above method embodiment. The details of the unmanned aerial vehicle image-text retrieval system for disaster scene-oriented negative sample mining enhancement proposed in this embodiment are described below, and the following content is only provided for understanding the implementation details, and is not necessary for implementing the present solution.

[0102] The specific structure of the unmanned aerial vehicle image-text retrieval system for disaster scene-oriented negative sample mining enhancement proposed in this embodiment is shown in Figure 4 The specific structure of the unmanned aerial vehicle image-text retrieval system for disaster scene-oriented negative sample mining enhancement proposed in this embodiment is shown in

[0103] The training set construction module 21 is configured to pre-process the obtained paired image sample data and text sample data, and construct a training set based on the pre-processed image sample data and text sample data.

[0104] The model construction module 22 is configured to construct an unmanned aerial vehicle image-text retrieval model composed of a coarse-grained cross-modal alignment module and a fine-grained difficult negative sample mining module. The coarse-grained cross-modal alignment module is composed of an image encoder containing a domain adapter and a text encoder containing a domain adapter, which is used to perform preliminary alignment in a regular scene using domain adaptation technology, providing a good feature space basis for difficult negative sample mining. The fine-grained difficult negative sample mining module is composed of a momentum image encoder, a momentum text encoder, and corresponding momentum queues, which are used to gradually distinguish difficult negative samples with high similarity from false negative samples, and guide the learning of more discriminative feature representations.

[0105] The model training module 23 is configured to input the image sample data and text sample data in the training set into the unmanned aerial vehicle image-text retrieval model, and use a hybrid momentum loss function to train and optimize it to convergence, obtaining a trained unmanned aerial vehicle image-text retrieval model.

[0106] The model using module 24 is configured to input the image data and the text data of the pair of drones to be identified into the trained drone image-text retrieval model to obtain a corresponding drone image-text retrieval result.

[0107] It is worth mentioning that each module involved in the embodiment is a logical module. In actual application, one logical module can be one physical module, or a part of one physical module, or an organic combination of multiple physical modules. In addition, in order to highlight the innovative part of the present application, modules not closely related to solving the technical problems proposed in the present application are not introduced in the embodiment, but this does not mean that there are no other modules in the embodiment.

[0108] It can be found that the embodiment is a system embodiment corresponding to the above-mentioned method embodiment, and the embodiment can be implemented in cooperation with the above-mentioned method embodiment. The related technical details and technical effects mentioned in the above-mentioned method embodiment are still valid in the embodiment. In order to reduce repetition, they will not be described here. Correspondingly, the related technical details mentioned in the embodiment can also be applied to the above-mentioned method embodiment.

[0109] Another embodiment of the present application provides an electronic device, as shown in the figure, comprising a processor 31 and a memory 32, wherein the memory 32 stores instructions executable by the processor 31, and the processor 31 is configured to execute the instructions, so that the electronic device can implement a disaster scenario-oriented negative sample mining enhanced drone image-text retrieval method as described in the above-mentioned method embodiment. Figure 5

[0110] Wherein, the memory and the processor are connected in a bus mode, the bus includes any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage stabilizers and power management circuits together, which are well known in the art, and therefore will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements such as multiple receivers and transmitters, which provide a unit for communicating with various other devices on the transmission medium. The data processed by the processor is transmitted on the wireless medium through the antenna, and further, the antenna also receives data and transmits the data to the processor.

[0111] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management and other control functions. And the memory can be used to store the data used by the processor in the execution operation.

[0112] ​Another embodiment of the present application provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and the computer program, when executed by a processor, enables a disaster scenario-oriented negative sample mining enhanced UAV image-text retrieval method to be implemented.

[0113] That is, a person skilled in the art can understand that all or part of the steps in the above method embodiments can be completed by programs instructing related hardware, the programs are stored in a storage medium, and the programs include a plurality of instructions to make a device (such as a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the method described in the method embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.

[0114] A person skilled in the art can understand that the above embodiments are specific embodiments of the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application. For those skilled in the art, a number of improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements are also considered to be within the protection scope of the present application.

Claims

1. A UAV image and text retrieval method enhanced by negative sample mining for disaster scenarios, implemented based on a coarse-to-fine progressive discriminative framework, characterized in that... The method includes: The obtained paired image sample data and text sample data are preprocessed respectively, and a training set is constructed based on the preprocessed image sample data and text sample data; A coarse-grained cross-modal alignment module is constructed. The coarse-grained cross-modal alignment module consists of an image encoder containing a domain adapter and a text encoder containing a domain adapter. It is used to achieve preliminary alignment in normal scenarios by utilizing domain adaptation technology, providing a good feature space foundation for mining hard-burden samples. A fine-grained hard-negative sample mining module is constructed. This module consists of a momentum image encoder, a momentum text encoder, and a corresponding momentum queue. It is used to gradually distinguish between hard-negative samples with high similarity and false negative samples, and guide the learning of more discriminative feature representations. Image and text sample data from the training set are input into a UAV image-text retrieval model consisting of a coarse-grained cross-modal alignment module and a fine-grained hard-negative sample mining module. The model is trained and optimized using a hybrid momentum loss function until convergence, resulting in a trained UAV image-text retrieval model. The image and text data of the pairs of drones to be identified are input into the trained drone image-text retrieval model to obtain the corresponding drone image-text retrieval results.

2. The UAV image and text retrieval method for disaster scenarios with negative sample mining enhancement as described in claim 1, characterized in that, The acquired paired image and text sample data are preprocessed separately, and a training set is constructed based on the preprocessed image and text sample data, including: Obtain a drone image and text retrieval dataset for disaster scenarios. The drone image and text retrieval dataset includes several pairs of image sample data and text sample data. The resolution of all image sample data in the UAV image retrieval dataset is uniformly scaled to a preset standard resolution to obtain preprocessed image sample data. All text sample data in the UAV image and text retrieval dataset are uniformly segmented and encoded to obtain preprocessed text sample data; Based on pairs of preprocessed image sample data and preprocessed text sample data, the data is divided according to a preset ratio to obtain a training set, a test set, and a validation set.

3. The UAV image and text retrieval method for disaster scenarios enhanced by negative sample mining as described in claim 1, characterized in that, For an image encoder, the domain adapter is connected after the TransformerBlock of the original image encoder, and the two together constitute the same level of the image encoder. For a text encoder, the domain adapter is connected after the Transformer Block of the original text encoder, and the two together constitute the same level of the text encoder. The dimensionality of the domain adapter follows a U-shaped structure of first decreasing and then increasing dimensions, which is used to extract and express deep features. The domain adapter is specifically composed of a dimension-reducing Linear layer, a GELU activation function, an LN layer for normalization, a fully connected FC layer, a dimension-increasing Linear layer, and skip connections. The domain adapters of the image encoder and the text encoder share the GELU activation function, LN layer normalization, and FC layer parameters. The parameters of the dimension-reduced Linear layer and the dimension-upgraded Linear layer are designed separately and are not shared.

4. The UAV image and text retrieval method for disaster scenarios with negative sample mining enhancement as described in claim 3, characterized in that, The output of the image encoder is expressed by the formula: ； ； ； in, This represents the input to the image encoder. This represents a dimension-reduced linear layer in the neighborhood adapter of an image encoder. This represents the GELU activation function. Indicates LN layer normalization, Indicates FC Layer (Fully Connected Layer). This represents an upgraded linear layer in the domain adapter of an image encoder. This represents the output of the dimension-reduced linear layer of the neighborhood adapter of the image encoder. The output of the FC Layer fully connected layer representing the domain adapter of the image encoder and The result after skip connections, This represents the output of the image encoder; The output of the text encoder is expressed by the formula: ； ； ； in, This represents the input to the text encoder. and These represent the dimension-reduced Linear layer and the dimension-upgraded Linear layer of the domain adapter of the text encoder, respectively. This represents the output of the dimension-reduced linear layer of the domain adapter of the text encoder. The output of the FC Layer fully connected layer representing the domain adapter of the text encoder and The result after skip connections, This represents the output of the text encoder.

5. The UAV image and text retrieval method for disaster scenarios with negative sample mining enhancement as described in claim 1, characterized in that, The output of the image encoder is the input of the momentum image encoder. After processing by the momentum image encoder, the output of the image encoder enters the image momentum queue to expand the perception capability of hard-to-bear samples through larger batches. The output of the text encoder is the input of the momentum text encoder. After processing by the momentum text encoder, the output of the text encoder enters the text momentum queue to expand the perception capability of hard-to-bear samples through larger batches. The size of the image momentum queue and the text momentum queue are much larger than the single batch of the UAV image-text retrieval model.

6. The UAV image and text retrieval method for disaster scenarios with negative sample mining enhancement as described in claim 5, characterized in that, The weights of the momentum image encoder and the momentum text encoder are updated using the following formula: ； ； in, The weights before the momentum image encoder update. For the weights of the image encoder, The weights are updated by the momentum image encoder. These are the weights before the momentum text encoder update. For the weights of the text encoder, The updated weights of the momentum text encoder This is the preset balance coefficient.

7. A UAV image and text retrieval method for disaster scenarios enhanced by negative sample mining according to any one of claims 1 to 6, characterized in that, The mixed momentum loss function is expressed by the formula: ； ； ； in, This represents the mixed momentum loss function. The loss function represents the image-to-text retrieval task. The loss function representing the text-to-image retrieval task. For batch size, Represents cross-entropy loss, For image indexing, For text indexing, subscript in the lower right corner Indicates the first For sample data, the bottom right corner is labeled Indicates the first For sample data, Indicates the first Mixed momentum labels for sample data Indicates the first Mixed momentum labels for sample data This represents the momentum matching probability from image to text. This represents the momentum matching probability from text to image.

8. A UAV image and text retrieval system enhanced by negative sample mining for disaster scenarios, implemented based on the UAV image and text retrieval method enhanced by negative sample mining for disaster scenarios as described in any one of claims 1 to 7, characterized in that, The system includes: The training set construction module is used to preprocess the acquired paired image sample data and text sample data respectively, and construct the training set based on the preprocessed image sample data and text sample data. The model building module is used to construct a UAV image-text retrieval model consisting of a coarse-grained cross-modal alignment module and a fine-grained hard-negative sample mining module. The coarse-grained cross-modal alignment module consists of an image encoder with a domain adapter and a text encoder with a domain adapter. It is used to achieve initial alignment in normal scenarios using domain adaptation technology, providing a good feature space foundation for hard-negative sample mining. The fine-grained hard-negative sample mining module consists of a momentum image encoder, a momentum text encoder and a corresponding momentum queue. It is used to gradually distinguish between highly similar hard-negative samples and false negative samples, guiding the learning of more discriminative feature representations. The model training module is used to input image sample data and text sample data from the training set into the UAV image and text retrieval model, and use the hybrid momentum loss function to train and optimize it until convergence, thus obtaining the trained UAV image and text retrieval model. The model uses a module to input the image and text data of the pairs of drones to be identified into the trained drone image-text retrieval model, and obtain the corresponding drone image-text retrieval results.

9. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores instructions that the processor can execute, and the processor is configured to, when executing the instructions, enable the electronic device to implement a UAV image and text retrieval method for disaster scenarios with negative sample mining enhancement as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a UAV image and text retrieval method for disaster scenarios with negative sample mining enhancement as described in any one of claims 1 to 7.

Images