Cross-modal retrieval method and system for ocean remote sensing images and texts with embedded adaptive mask

Abstract

The present invention belongs to the technical field of cross-modal image and text retrieval, and discloses a cross-modal retrieval method and system for marine remote sensing images and texts with embedded adaptive masks. For input images and texts, image prompts and text prompts are defined, and feature extraction is performed through an encoder to obtain initial image features and initial text features. The initial image features and initial text features are used to update the image prompts and text prompts using a cross-guidance strategy. The updated prompts are input into the encoder to continue updating the image features and text features. Based on the initial image features and initial text features, significant image and text features are selected using an intra-modal importance matrix and a cross-modal importance matrix, and the selected features are masked to obtain mask features. The mask features are predicted using a mask loss. The present invention improves the robustness of prompt learning, achieves finer-grained feature learning, and can predict finer-grained information by introducing a masking mechanism.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-modal image and text retrieval, and in particular relates to a method and system for cross-modal retrieval of ocean remote sensing images and text with embedded adaptive masks. Background Art

[0002] Cross-modal retrieval of marine remote sensing data aims to establish matching relationships between data from different modalities and improve the representation of marine remote sensing objects through multimodal data fusion. Currently, one of the main challenges facing traditional cross-modal retrieval methods for marine remote sensing is the high computational cost. Therefore, cutting-edge image-text retrieval methods are focusing on using CLIP-based pre-trained networks to improve feature representation. The process involves pre-training CLIP's image and text encoders on large-scale datasets. These pre-trained image and text encoders are then directly transferred to the cross-modal retrieval network, using prompts as trainable parameters to better adapt the model to downstream tasks.

[0003] However, the above methods still have the following problems when applied to the ocean:

[0004] First, the weak robustness of the prompt design. The prompts designed by existing methods are separate and high-degree-of-freedom prompts. On the one hand, this method designs learnable prompts for images and text separately, lacking cross-modal information interaction, resulting in deficiencies in feature alignment and semantic understanding, making it difficult to fully capture the correlation between the two modalities, which in turn affects the retrieval effect and accuracy of the model. On the other hand, image and text prompts are learnable parameters with a high degree of freedom. Although they are supervised by the loss function in the subsequent process, the amount of information they contain is still insufficient. Marine remote sensing data involves complex content, such as multiple scenes and multi-scale targets. Therefore, adaptive prompts cannot fully mine and describe this complex information, resulting in limitations in the model's feature representation.

[0005] Second, the pre-training mechanism makes the model unsuitable for complex marine remote sensing data. Pre-training can transfer knowledge learned from large amounts of data to downstream tasks, but it cannot achieve fine-grained modeling. When faced with complex marine remote sensing data, the performance of cross-modal retrieval models remains unsatisfactory. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides a cross-modal retrieval method and system for marine remote sensing images with embedded adaptive masks. Among them, the present invention is committed to improving the robustness of prompt learning, and designs a fine-grained cross-prompt module. Through cross-guidance and feature fusion, the robustness of prompts is improved; the present invention is committed to designing a fine-grained feature modeling mechanism, and designs an embedded adaptive mask module. Through random mask areas, the prediction of fine-grained areas is realized, which improves the model's understanding of complex content in marine remote sensing data.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A cross-modal retrieval method for ocean remote sensing images and texts with embedded adaptive masks, including:

[0009] Steps of feature extraction:

[0010] For the input image data I and text data T, define a set of learnable parameters and As image prompts and text prompts, and feature extraction is performed through the image encoder and text encoder with k layers of Transformer respectively to obtain the initial image features and initial text features ;

[0011] Steps for fine-grained cross-hinting:

[0012] Using initial image features and initial text features , using a cross-guidance strategy to update image hints and text prompts In the cross-guidance strategy, cross-modal information is introduced into the prompt through the cross-attention mechanism to obtain updated image prompts and text prompts ;

[0013] The updated prompt is input into the image encoder and text encoder with the (k+1)th to Kth layer Transformer, and the image features and text features are continued to be updated to generate the updated image features. and text features ;

[0014] Steps to embed adaptive mask:

[0015] Based on the initial image features and initial text features , using the intra-modal importance matrix and the cross-modal importance matrix to select significant image and text features, and masking the selected features to obtain mask features; then using the mask loss to predict the mask features;

[0016] Steps for loss calculation and model training:

[0017] The updated image features and text features Map to the same dimension, calculate the triplet loss, and combine the triplet loss and mask loss for model training.

[0018] Furthermore, in the feature extraction step, text encoder and image encoder are used to extract features respectively. The specific steps are as follows:

[0019] In the text branch, for the text encoder, the sentence is first tokenized into a sequence T of words, which is then projected into word embeddings , As the input of the CLIP text encoder with k layers of Transformer, N is the number of texts, are the initial embedding vectors of the 1st, 2nd, ..., Nth text respectively; in addition, a set of learnable parameters is defined As a text prompt, it is introduced into each layer of Transformer.

[0020] ;

[0021] in, Represents a cascade operation, " indicates that the output at these specific positions is ignored or discarded. Transformer representing text, Indicates the number of layers of Transformer, Indicates the first The output of the layer Transformer, represents the output of the i-th layer Transformer in the text encoder, Represents the prompt of the i-th layer Transformer in the text encoder After the above operations, the features output by the kth layer are the initial text features ;

[0022] Similarly, in the vision branch, the image I is first segmented into M non-overlapping regions, which are then projected onto the image embedding middle, As the input of the image encoder with k layers of Transformer, M is the number of regions, are the initial embedding vectors of the 1st, 2nd, ..., Mth images respectively; define a set of learnable parameters As an image prompt, it is introduced into each layer of Transformer.

[0023] ;

[0024] in," " indicates that the output at these specific positions is ignored or discarded. Represents the Transformer of the image, i represents the number of layers of the Transformer, represents the output of the i-1th layer Transformer in the image encoder, represents the output of the i-th layer Transformer in the image encoder, Represents the hint of the i-th layer Transformer in the image encoder After the above operations, the features output by the kth layer are the initial image features .

[0025] Furthermore, the process of updating the prompts through the cross-attention mechanism is as follows: for text prompts, the cross-attention between them and the initial image features is calculated, and the updated text prompts are generated through layer normalization and a feed-forward network; for image prompts, the cross-attention between them and the initial text features is calculated, and the updated image prompts are generated through layer normalization and a feed-forward network.

[0026] Furthermore, the steps of the fine-grained cross-prompting are specifically:

[0027] For text prompts ,

[0028] ;

[0029] ;

[0030] Among them, CA is cross attention, LN is layer normalization, FFN stands for feedforward network, and i represents the number of layers of Transformer. Represents the intermediate prompt of the i-th layer Transformer in the text encoder Represents the text prompt of the i-th layer Transformer in the text encoder; the updated text prompt Passed into the (k+1)th layer to the Kth layer of Transformer; the features output by the Kth layer are the updated text features ;

[0031] ;

[0032] Similarly, for image prompts :

[0033] ;

[0034] ;

[0035] in, represents the intermediate prompt of the i-th layer Transformer in the image encoder, Represents the image prompt of the i-th layer Transformer in the image encoder; the updated image prompt Passed into the (k+1)th layer to the Kth layer of Transformer; the features output by the Kth layer are the updated image features ;

[0036] .

[0037] Furthermore, the step of embedding adaptive masking includes: an embedded masking mechanism and a masking strategy guided by cross-modal interaction, and the specific steps include:

[0038] Calculate the importance matrix of text and images within the modality and , and the cross-modal importance matrix and ;

[0039] Fuse the text and image importance matrices pairwise and filter out the valid areas and words ;

[0040] Record and The median value is 1, and and The corresponding positions in the random mask are generated to generate mask features and , the mask feature is supervised by mask image modeling loss and mask text modeling loss.

[0041] Furthermore, the specific process of embedded adaptive masking is as follows:

[0042] First, the noise filtering is realized by using the intramodal features. The similarity between the image and the text is calculated and the intramodal importance matrix of the text and image is obtained by summing the row vectors. and ;

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] Among them, L( ) represents the similarity operation, and is the position of the text feature, and express The position of the k-th layer feature in and The sub-features of and represents the location of image features, and express The position of the k-th layer feature in and sub-features of; Represents the similarity between two features in the text, Represents the similarity between two features in the image, M represents the number of regions, and N represents the number of texts;

[0048] Secondly, we use cross-modal features to achieve noise filtering. By calculating the similarity between images and texts, and summing the row vectors and column vectors respectively, we can get the cross-modal importance matrix of text and image features. and ;

[0049] ;

[0050] ;

[0051] ;

[0052] in, Represents the similarity between two features in the image and text;

[0053] Then, the text importance matrix and the image importance matrix are added together and the hyperparameters are designed. Filter out valid areas and words ;

[0054] ;

[0055] ;

[0056] in, Represents the conditional symbol, which takes 1 if satisfied and 0 if not.

[0057] Afterwards, record and The median value is 1, and and The corresponding positions in the random mask are generated to generate mask features , ; and Represents mask features, P and Q, and Indicates different mask positions, 、 、 Respectively expressed in In the example, the mask position is 1, , the sub-features corresponding to P, 、 、 Respectively expressed in In the example, the mask position is 1, , the sub-features corresponding to Q;

[0058] Mask loss includes mask image modeling loss and mask text modeling loss, mask feature and Relying on mask image modeling loss and mask text modeling loss supervision;

[0059] ;

[0060] ;

[0061] in Model the loss for the mask image, Modeling loss for masked text; Represented in word embedding Middle mask position The corresponding features, Represented in image embedding Middle mask position corresponding features; Represents the distance between two features; the final loss function is:

[0062] ;

[0063] Among them, L tri represents the triplet loss.

[0064] Furthermore, the triplet loss L tri The calculation formula is:

[0065] ;

[0066] in, and Represents the final text features and image features, and Mapping obtained, and Represents negative sample images and negative sample texts, Represents the similarity between the text and its corresponding positive sample image, The similarity between the text and its corresponding negative sample image, Represents the similarity between the image and its corresponding negative sample text; , Represents the interval parameter.

[0067] The present invention also provides a cross-modal retrieval system for marine remote sensing images and texts with an embedded adaptive mask, which is used to implement the cross-modal retrieval method for marine remote sensing images and texts with an embedded adaptive mask as described above. The system includes: a feature extraction module, a fine-grained cross-prompt module, an embedded adaptive mask module, and a loss calculation module.

[0068] The feature extraction module sets image prompts and text prompts for the input image data I and text data T, and obtains the initial image features through the image encoder and text encoder of the k-layer Transformer. and initial text features ;

[0069] The fine-grained cross-hinting module uses the initial image features and initial text features , using a cross-guidance strategy to update image hints and text prompts , get the updated image prompt and text prompts , input the updated prompt into the image encoder and text encoder with the (k+1)th layer to the Kth layer Transformer, continue to update the image features and text features, and generate the updated image features and text features ;

[0070] The embedded adaptive mask module is based on the initial image features and initial text features , using the intra-modal importance matrix and the cross-modal importance matrix to select significant image and text features, and masking the selected features to obtain mask features; then using the mask loss to predict the mask features;

[0071] The loss calculation module is used to calculate triplet loss and mask loss.

[0072] Compared with the prior art, the present invention has the following advantages:

[0073] First, the robustness of cue learning is improved. In the cross-guidance strategy, by incorporating intra-modal features into the cue, the cue learns the rich content information in marine remote sensing data. Furthermore, by leveraging cross-modal data for mutual guidance, the problem of cross-modal data alignment is alleviated, ultimately improving the robustness of cue learning.

[0074] Second, it enables finer-grained feature learning. By introducing a masking mechanism, the model predicts more fine-grained information, improving the robustness of feature representation. Furthermore, the embedded mechanism enables the model to have feature representations, thereby calculating effective intra-modal and inter-modal similarities, thereby filtering out valid and significant regions and words in complex ocean data. This approach, instead of random masking, specifically masks valid features, fully utilizing effective fine-grained features to optimize the retrieval network and improving network training speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0076] Figure 1 Schematic diagram of the method flow of the present invention;

[0077] Figure 2 Schematic diagram of the structure of the embedded adaptive mask module of the present invention. DETAILED DESCRIPTION

[0078] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0079] Combine Figure 1 、 Figure 2 As shown, this embodiment provides a cross-modal retrieval method for marine remote sensing images and texts with embedded adaptive masks, including: feature extraction, fine-grained cross-hinting, embedded adaptive masks, loss calculation and model training steps.

[0080] This paper follows the classic image-text cross-modal retrieval main framework and introduces text hints for image data I and text data T. and image prompts , extract features with the help of the k-layer Transformer pre-trained in CLIP, and obtain the initial image features and initial text features .

[0081] Different from previous image-text cross-modal retrieval methods, the present invention designs a fine-grained cross-hint module and an embedded adaptive mask module.

[0082] Among them, the fine-grained cross-prompt module is used in text prompts , image prompts , initial text features and initial image features Based on this, a cross-guidance strategy is adopted to obtain updated text prompts , image prompts After that, the (k+1)th to Kth layer Transformers are used to continue updating text features and image features to generate and , mapping these two to the same dimension, generating and , calculate the triplet loss.

[0083] In the cross-guidance strategy, by introducing intra-modal features into the prompts and leveraging the mutual guidance of cross-modal data, it enables the learning of rich content information while alleviating the problem of cross-modal data alignment, ultimately improving the robustness of prompt learning.

[0084] Among them, the embedded adaptive masking module designs an embedded masking mechanism, that is, masking several features in the internal features. Moreover, unlike the classic random masking, this method designs a masking strategy guided by cross-modal interaction. First, the text and importance matrix within the modality are calculated. and , and the cross-modal importance matrix and , and fuse the text and image importance matrices pairwise to design hyperparameters Filter out valid areas and words . The valid area and words are multiplied with the text features and image features respectively to obtain valid text features and image features. On this basis, features are randomly selected to mask, and the Transformer network is used to extract the features of the masked area. The generated mask features rely on the initial embedding and the supervision of the mask image modeling loss and the mask text modeling loss. By considering the effective information within and between modalities, the effective\significant areas and words are targetedly masked, so that fine-grained features can be trained more purposefully. This not only makes full use of the effective fine-grained features to optimize the retrieval network, but also improves the training speed of the network.

[0085] The following describes each step in detail.

[0086] 1. Steps of feature extraction:

[0087] For the input image data I and text data T, define a set of learnable parameters and As image prompts and text prompts, and feature extraction is performed through the image encoder and text encoder with k layers of Transformer respectively to obtain the initial image features and initial text features .

[0088] As a preferred embodiment, in the feature extraction step, CLIP text encoder and CLIP image encoder are used to extract features respectively. The specific steps are as follows:

[0089] In the text branch, for the text encoder, the sentence is first tokenized into a sequence T of words, which is then projected into word embeddings , As the input of the CLIP text encoder with k layers of Transformer, N is the number of texts, are the initial embedding vectors of the 1st, 2nd, ..., Nth text respectively; in addition, a set of learnable parameters is defined As a text prompt, it is introduced into each layer of Transformer.

[0090] ;

[0091] in, Represents a cascade operation, " indicates that the output at these specific positions is ignored or discarded, generating When the "The position that is useless is discarded. Represents the Transformer of the text, i represents the number of layers of the Transformer, represents the output of the i-th layer Transformer in the text encoder, represents the output of the i-th layer Transformer in the text encoder, Represents the prompt of the i-th layer Transformer in the text encoder After the above operations, the features output by the kth layer are the initial text features .

[0092] Similarly, in the vision branch, the image I is first segmented into M non-overlapping regions, which are then projected onto the image embedding middle, As the input of the CLIP image encoder with k layers of Transformer, M is the number of regions, are the initial embedding vectors of the 1st, 2nd, ..., Mth images respectively; define a set of learnable parameters As an image prompt, it is introduced into each layer of Transformer.

[0093] ;

[0094] in," " indicates that the output at these specific positions is ignored or discarded. Transformer representing an image, Indicates the number of layers of Transformer, Indicates the image encoder The output of the layer Transformer, represents the output of the i-th layer Transformer in the image encoder, Represents the hint of the i-th layer Transformer in the image encoder After the above operations, the features output by the kth layer are the initial image features .

[0095] 2. Steps for fine-grained cross-hinting:

[0096] Using initial image features and initial text features , using a cross-guidance strategy to update image hints and text prompts In the cross-guidance strategy, cross-modal information is introduced into the prompt through the cross-attention mechanism to obtain updated image prompts and text prompts .

[0097] As a preferred embodiment, the process of updating prompts through the cross-attention mechanism is as follows: for text prompts, the cross-attention between them and the initial image features is calculated, and the updated text prompts are generated through layer normalization and a feed-forward network; for image prompts, the cross-attention between them and the initial text features is calculated, and the updated image prompts are generated through layer normalization and a feed-forward network.

[0098] The updated prompt is input into the image encoder and text encoder with the (k+1)th to Kth layers of the Transformer, that is, input into the (k+1)th to Kth layers of the image encoder and text encoder, where K is the total number of layers of the Transformer and k is the current layer index. The image features and text features are continued to be updated to generate the updated image features. and text features .

[0099] As a preferred embodiment, the steps of fine-grained cross-prompting are specifically:

[0100] For text prompts ,

[0101] ;

[0102] ;

[0103] in, represents cascade operation, CA is cross attention, LN is layer normalization, FFN represents feedforward network, i represents the number of layers of Transformer, Represents the intermediate prompt of the i-th layer Transformer in the text encoder Represents the text prompt of the i-th layer Transformer in the text encoder. The updated text prompt Passed into the (k+1)th layer to the Kth layer of Transformer; the features output by the Kth layer are the updated text features ;

[0104] ;

[0105] Similarly, for image prompts :

[0106] ;

[0107] ;

[0108] Among them, i represents the number of layers of Transformer, represents the intermediate prompt of the i-th layer Transformer in the image encoder, Represents the image prompt of the i-th layer Transformer in the image encoder. The updated image prompt Passed into the (k+1)th layer to the Kth layer of Transformer; the features output by the Kth layer are the updated image features ;

[0109] .

[0110] 3. Steps for embedding adaptive mask:

[0111] Based on the initial image features and initial text features , the intra-modal importance matrix and the cross-modal importance matrix are used to select significant image and text features, and the selected features are masked to obtain mask features; then, the mask loss is used to predict the mask features, that is, the mask image modeling loss and the mask text modeling loss are used to predict the mask features. Through the transmission mechanism of the neural network, this loss further optimizes the image prompts and text prompts.

[0112] The steps of embedding adaptive masking include: an embedded masking mechanism and a masking strategy guided by cross-modal interaction. The specific steps include:

[0113] Calculate the importance matrix of text and images within the modality and , and the cross-modal importance matrix and ;

[0114] Fuse the text and image importance matrices pairwise and filter out the valid areas and words ;

[0115] Record and The median value is 1, and and The corresponding positions in the random mask are generated to generate mask features and , the mask feature is supervised by mask image modeling loss and mask text modeling loss.

[0116] As an example, the specific process of embedded adaptive masking is as follows:

[0117] First, the noise filtering is realized by using the intramodal features. The similarity between the image and the text is calculated and the intramodal importance matrix of the text and image is obtained by summing the row vectors. and ;

[0118] ;

[0119] ;

[0120] ;

[0121] ;

[0122] Among them, L( ) represents the similarity operation, and is the position of the text feature, and express The position of the k-th layer feature in and The sub-features of and represents the location of image features, and express The position of the k-th layer feature in and sub-features of; Represents the similarity between two features in the text, Represents the similarity between two features in the image, M represents the number of regions, and N represents the number of texts.

[0123] Secondly, we use cross-modal features to achieve noise filtering, by calculating the similarity between images and texts, and deriving the cross-modal importance matrix of text and image features by summing up the row vectors and column vectors respectively. and ;

[0124] ;

[0125] ;

[0126] ;

[0127] in, Represents the similarity between two features in image and text.

[0128] Then, the text importance matrix and the image importance matrix are added together and the hyperparameters are designed. Filter out valid areas and words ;

[0129] ;

[0130] ;

[0131] in, Represents the conditional symbol, which takes 1 if satisfied and 0 if not.

[0132] Afterwards, record and The median value is 1, and and The corresponding positions in the random mask are generated to generate mask features , ; and Represents mask features, P and Q, and Indicates different mask positions, 、 、 Respectively expressed in In the example, the mask position is 1, , the sub-features corresponding to P, 、 、 Respectively expressed in In the example, the mask position is 1, , the sub-features corresponding to Q.

[0133] Mask loss includes mask image modeling loss and mask text modeling loss, mask feature and Relying on mask image modeling loss and mask text modeling loss supervision;

[0134]

[0135]

[0136] in Model the loss for the mask image, Modeling loss for masked text; Represented in word embedding Middle mask position The corresponding features, Represented in image embedding Middle mask position corresponding features; Represents the distance between two features; the final loss function is:

[0137] ;;

[0138] Among them, L tri represents the triplet loss.

[0139] Triplet loss The calculation formula is:

[0140] ;

[0141] in, and Represents the final text features and image features, and Mapping obtained, and Represents negative sample images and negative sample texts, Represents the similarity between the text and its corresponding positive sample image, The similarity between the text and its corresponding negative sample image, Represents the similarity between the image and its corresponding negative sample text; , Represents the interval parameter.

[0142] 4. Steps for loss calculation and model training:

[0143] The updated image features and text features Map to the same dimension, calculate the triplet loss, and combine the triplet loss and mask loss for model training.

[0144] As a preferred embodiment, this embodiment also provides a cross-modal retrieval system for marine remote sensing images and texts with an embedded adaptive mask, which can implement the cross-modal retrieval method for marine remote sensing images and texts with an embedded adaptive mask as described above. The system includes a feature extraction module, a fine-grained cross-hinting module, an embedded adaptive mask module, and a loss calculation module.

[0145] The feature extraction module sets image prompts and text prompts for the input image data I and text data T, and obtains the initial image features through the image encoder and text encoder of the k-layer Transformer and initial text features .

[0146] Fine-grained cross-hinting module, leveraging initial image features and initial text features , using a cross-guidance strategy to update image hints and text prompts , get the updated image prompt and text prompts , input the updated prompt into the image encoder and text encoder with the (k+1)th layer to the Kth layer Transformer, continue to update the image features and text features, and generate the updated image features and text features .

[0147] Embedded adaptive mask module based on initial image features and initial text features , the intra-modal importance matrix and the cross-modal importance matrix are used to select significant image and text features, and the selected features are masked to obtain mask features; then the mask loss is used to predict the mask features.

[0148] The loss calculation module is used to calculate triplet loss and mask loss.

[0149] The functions and specific implementation methods of each module can be found in the steps of the previous cross-modal retrieval method and will not be repeated here.

[0150] In summary, the present invention (1) designs a fine-grained cross-prompt module and adopts a cross-guidance strategy to obtain updated text prompts. , image prompts After that, the (k+1)th to Kth layer Transformers are used to continue updating text features and image features to generate and , mapping these two to the same dimension, generating and , calculate the triplet loss. (2) Design an embedded adaptive mask module, design an embedded mask mechanism, that is, mask several features in the internal features. Moreover, unlike the classic random mask, this method designs a mask strategy guided by cross-modal interaction. First, calculate the text and importance matrix within the modality and , and the cross-modal importance matrix and , and fuse the text and image importance matrices pairwise to design hyperparameters Filter out valid areas and words The valid region and word are multiplied by the text features and image features respectively to obtain valid text features and image features. Based on this, features are randomly selected to mask, and the Transformer network is used to extract the features of the masked region. The generated mask features are supervised by the initial embedding and the mask image modeling loss and mask text modeling loss.

[0151] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the scope of protection of the present invention.

Claims

1. A cross-modal retrieval method for ocean remote sensing images and texts with embedded adaptive masks, characterized by: include: Steps of feature extraction: For the input image data I and text data T, define a set of learnable parameters and As image prompts and text prompts, and feature extraction is performed through the image encoder and text encoder with k layers of Transformer respectively to obtain the initial image features and initial text features ; Steps for fine-grained cross-hinting: Using initial image features and initial text features , using a cross-guidance strategy to update image hints and text prompts ; In the cross-guidance strategy, cross-modal information is introduced into the prompt through the cross-attention mechanism to obtain updated image prompts and text prompts ; The updated prompt is input into the image encoder and text encoder with the (k+1)th to Kth layer Transformer, and the image features and text features are continued to be updated to generate the updated image features. and text features ; Steps to embed adaptive mask: Based on the initial image features and initial text features , using the intra-modal importance matrix and the cross-modal importance matrix to select significant image and text features, and mask the selected features to obtain mask features; Then the mask loss is used to predict the mask features; Steps for loss calculation and model training: The updated image features and text features Map to the same dimension, calculate the triplet loss, and combine the triplet loss and mask loss for model training.

2. The cross-modal retrieval method for ocean remote sensing images and texts with embedded adaptive masks according to claim 1 is characterized in that: In the feature extraction step, text encoder and image encoder are used to extract features respectively. The specific steps are as follows: In the text branch, for the text encoder, the sentence is first tokenized into a sequence T of words, which is then projected into word embeddings , As the input of the text encoder with k layers of Transformer, N is the number of texts, are the initial embedding vectors of the 1st, 2nd, ..., Nth texts respectively; In addition, we define a set of learnable parameters As a text prompt, it is introduced into each layer of Transformer. ； in, Represents a cascade operation, " indicates that the output at these specific positions is ignored or discarded. Represents the Transformer of the text, i represents the number of layers of the Transformer, Represents the output of the i-1th layer Transformer in the text encoder, represents the output of the i-th layer Transformer in the text encoder, Represents the prompt of the i-th layer Transformer in the text encoder; after the above operations, the features output by the k-th layer are the initial text features ; Similarly, in the vision branch, the image I is first segmented into M non-overlapping regions, which are then projected onto the image embedding middle, As the input of the image encoder with k layers of Transformer, M is the number of regions, are the initial embedding vectors of the 1st, 2nd, ..., Mth images respectively; define a set of learnable parameters As an image prompt, it is introduced into each layer of Transformer. ； in, Transformer representing an image, represents the output of the i-1th layer Transformer in the image encoder, represents the output of the i-th layer Transformer in the image encoder, Represents the prompt of the i-th layer Transformer in the image encoder; after the above operations, the features output by the k-th layer are the initial image features .

3. The cross-modal retrieval method for ocean remote sensing images and texts with embedded adaptive masks according to claim 1, characterized in that: The process of updating the prompts through the cross-attention mechanism is as follows: for text prompts, the cross-attention with the initial image features is calculated, and the updated text prompts are generated through layer normalization and feed-forward network; for image prompts, the cross-attention with the initial text features is calculated, and the updated image prompts are generated through layer normalization and feed-forward network.

4. The cross-modal retrieval method for ocean remote sensing images and texts with embedded adaptive masks according to claim 2, characterized in that: The steps of the fine-grained cross-prompt are specifically: For text prompts , ； ； Among them, CA is cross attention, LN is layer normalization, FFN stands for feedforward network, and i represents the number of layers of Transformer. Represents the intermediate prompt of the i-th layer Transformer in the text encoder Represents the text prompt of the i-th layer Transformer in the text encoder; the updated text prompt The feature passed to the (k+1)th layer to the Kth layer of the Transformer is the updated text feature. ; ； Similarly, for image prompts : ； ； in, represents the intermediate prompt of the i-th layer Transformer in the image encoder, Represents the image prompt of the i-th layer Transformer in the image encoder; the updated image prompt Passed into the (k+1)th layer to the Kth layer of Transformer; the features output by the Kth layer are the updated image features ; 。 5. The cross-modal retrieval method for ocean remote sensing images and texts with embedded adaptive masks according to claim 1, characterized in that: The step of embedding adaptive masking includes: an embedded masking mechanism and a masking strategy guided by cross-modal interaction, and the specific steps include: Calculate the importance matrix of text and images within the modality and , and the cross-modal importance matrix and ; Fuse the text and image importance matrices pairwise and filter out the valid areas and words ; Record and The median value is 1, and and The corresponding positions in the random mask are generated to generate mask features and , the mask feature is supervised by mask image modeling loss and mask text modeling loss.

6. The cross-modal retrieval method for ocean remote sensing images and texts with embedded adaptive masks according to claim 5, characterized in that: The specific process of embedded adaptive masking is as follows: First, the noise filtering is realized by using the intramodal features. The similarity between the image and the text is calculated and the intramodal importance matrix of the text and image is obtained by summing the row vectors. and ; ； ； ； ； Among them, L( ) represents the similarity operation, and is the position of the text feature, and express The position of the k-th layer feature in and The sub-features of and represents the location of image features, and express The position of the k-th layer feature in and sub-features of; Represents the similarity between two features in the text, Represents the similarity between two features in the image, M represents the number of regions, and N represents the number of texts; Secondly, we use cross-modal features to achieve noise filtering. By calculating the similarity between images and texts, and summing the row vectors and column vectors respectively, we can get the cross-modal importance matrix of text and image features. and ; ； ； ； in, Represents the similarity between two features in the image and text; Then, the text importance matrix and the image importance matrix are added together and the hyperparameters are designed. Filter out valid areas and words ; ； ； in, Represents the conditional symbol, which takes 1 if satisfied and 0 if not. Afterwards, record and The median value is 1, and and The corresponding positions in the random mask are generated to generate mask features , ; and Represents mask features, P and Q, and Indicates different mask positions, 、 、 Respectively expressed in In the example, the mask position is 1, , the sub-features corresponding to P, 、 、 Respectively expressed in In the example, the mask position is 1, , the sub-features corresponding to Q; Mask loss includes mask image modeling loss and mask text modeling loss, mask feature and Relying on mask image modeling loss and mask text modeling loss supervision; ； ； in Model the loss for the mask image, Modeling loss for masked text; Represented in word embedding Middle mask position The corresponding features, Represented in image embedding Middle mask position corresponding features; Represents the distance between two features; the final loss function is: ； Among them, L tri represents the triplet loss.

7. The cross-modal retrieval method for ocean remote sensing images and texts with embedded adaptive masks according to claim 6, characterized in that: Triplet loss The calculation formula is: ； in, and Represents the final text features and image features, and Mapping obtained, and Represents negative sample images and negative sample texts, Represents the similarity between the text and its corresponding positive sample image, The similarity between the text and its corresponding negative sample image, Represents the similarity between the image and its corresponding negative sample text; , Represents the interval parameter.

8. A cross-modal retrieval system for ocean remote sensing images and texts with embedded adaptive masks, characterized by: A method for cross-modal retrieval of marine remote sensing images and texts with embedded adaptive masks according to any one of claims 1 to 7 is implemented, wherein the system comprises: a feature extraction module, a fine-grained cross-prompt module, an embedded adaptive mask module, and a loss calculation module. The feature extraction module sets image prompts and text prompts for the input image data I and text data T, and obtains the initial image features through the image encoder and text encoder of the k-layer Transformer. and initial text features ; The fine-grained cross-hinting module uses the initial image features and initial text features , using a cross-guidance strategy to update image hints and text prompts , get the updated image prompt and text prompts , input the updated prompt into the image encoder and text encoder with the (k+1)th layer to the Kth layer Transformer, continue to update the image features and text features, and generate the updated image features and text features ; The embedded adaptive mask module is based on the initial image features and initial text features , using the intra-modal importance matrix and the cross-modal importance matrix to select significant image and text features, and masking the selected features to obtain mask features; then using the mask loss to predict the mask features; The loss calculation module is used to calculate triplet loss and mask loss.

Images