A multimodal unstructured content association retrieval method

Abstract

The present invention discloses a multimodal unstructured content association retrieval method, comprising extracting features from multimodal data to obtain features of data in different modalities; aligning the multimodal features to obtain multimodal alignment features; randomly masking the multimodal alignment features and feeding them into a cross-modal self-attention model for fusion to obtain a masked multimodal fusion feature vector; extracting enhanced features from each image; processing the enhanced features through a cross-attention network to obtain cosine similarity between different images; matching image features to obtain image association results; obtaining text features by inputting retrieval text into a large language model, and matching the most similar images using cosine similarity in a multimodal data embedding space to obtain image retrieval results. The present invention improves the accuracy and stability of multimodal feature fusion and improves the accuracy and efficiency of image retrieval.

Description

Technical Field

[0001] The present invention relates to the field of retrieval technology, and in particular to a multimodal unstructured content association retrieval method. Background Art

[0002] Multimodal data association retrieval requires the ability to input an image or text and find the corresponding associated images and search content that matches the text. Currently, multimodal image data includes visible light imaging, infrared imaging, and SAR imaging. Compared to infrared and visible light imaging, SAR imaging has stable detection capabilities. In contrast, the other two types of imaging are susceptible to high-dynamic environments (high-speed targets can cause blurring and poor image quality). The unstable output of these sensors can lead to unpredictable modal structuring, resulting in significant errors in the acquired information.

[0003] Existing technologies for multimodal data association retrieval have shortcomings, such as difficulty matching image content with retrieval text, data distribution discrepancies, and ambiguous cross-modal feature expressions. Traditional methods typically only consider feature alignment between two modalities and fail to fully exploit intermodal associations when fusing multimodal features. This results in low image association and retrieval accuracy, making them inadequate for complex scenarios. Summary of the Invention

[0004] To solve the above problems, the present invention provides a multimodal unstructured content association retrieval method, which makes full use of multimodal feature information, aligns the multimodal data obtained by the visual language large model, eliminates the problem of data unstructuredness, and fuses the multimodal data to obtain comprehensive information perception. It also uses cross-attention network processing and cosine similarity comparison to realize the association and retrieval of multimodal data content, improves the accuracy and efficiency of image retrieval, and better adapts to the needs of multimodal image data processing and application scenarios in complex scenarios.

[0005] The present invention provides a multimodal unstructured content association retrieval method, the specific technical solution is as follows:

[0006] S1: Extract features from multimodal data to obtain features of data in different modalities;

[0007] S2: Align the multimodal features according to the obtained multimodal features to obtain multimodal alignment features;

[0008] S3: Randomly mask the multimodal alignment features, and send the masked feature vector to the cross-modal self-attention Transformer model for fusion to obtain the masked multimodal fusion feature vector;

[0009] S4: Process the masked multimodal fusion features to extract enhanced features of each image;

[0010] S5: The enhanced features are processed through the cross attention network to obtain the cosine similarity between different images;

[0011] S6: Matching image features by performing metric calculation based on the obtained cosine similarity to obtain an image association result;

[0012] S7: Based on the image association results obtained, the text features are obtained by inputting the retrieval text into the language model. Through the multimodal data embedding space, the cosine similarity is used to match the most similar images to obtain the image retrieval results.

[0013] Furthermore, in step S1, the multimodal data are images of different modalities, including visible light, infrared and SAR images.

[0014] Furthermore, in step S2, the multimodal features are aligned across modalities using modality center perception. The specific process is as follows:

[0015] S201: Taking the same pair of multimodal features as a positive sample pair and the cross-domain multimodal features of different objects as a negative sample pair, constructing positive and negative sample pairs;

[0016] S202: Using the multimodal feature centers of the same object as anchor points, perform comparative learning and update model parameters;

[0017] S203: Mapping different modal features into a common feature space through the trained model;

[0018] S204: Restore the original dimension of the feature through multimodal feature inverse mapping to obtain aligned multimodal features.

[0019] Furthermore, the multimodal feature centers of the same object are used as anchor points and as positive samples in the contrastive learning process. The contrastive learning loss function is used to perform model update training. The loss function is expressed as follows:

[0020]

[0021] in, is the similarity measure score between the current input modality feature and the multimodal feature center, is the similarity measurement score between the current input object feature and other object features, is the temperature coefficient, which is used to adjust the degree of attention paid to positive and negative sample pairs during model learning. is the set of all object samples, is the set of all modes.

[0022] Furthermore, in step S3, multimodal features are fused, and the specific process is as follows:

[0023] Randomly mask one modality alignment feature and perform complementary masking on the other modality alignment features to generate an incomplete unimodal feature vector after masking;

[0024] The masked incomplete unimodal feature vector is used as the query input, and the multimodal feature vectors are concatenated and fused as the key and value input to obtain the multimodal complementary features after cross-modal attention weighting. The features are then input into the ViT model in the cross-modal self-attention Transformer model for fusion, and the multimodal fusion features are output.

[0025] Furthermore, the multimodal complementary feature is obtained and calculated as follows:

[0026]

[0027] in, Compute cross-modal attention, is the feature vector of the visible part after masking, and are other modal eigenvectors, express and Feature dimension, T represents the transpose of the matrix.

[0028] Furthermore, in step S5, during the training of the cross-attention network, the loss function is composed of a contrast loss based on cosine similarity, which is expressed as follows:

[0029]

[0030] in, represents the cosine similarity, is the target feature sample, samples can be obtained through data augmentation samples, is the cosine similarity between positive sample pairs, represents the i-th target feature sample, Represents Matched feature samples, Indicates the kth feature sample in 2N samples, is a piecewise function, which is 1 when k≠i and 0 when k=i.

[0031] Furthermore, in step S6, image association results are obtained through feature matching, as follows:

[0032] Cosine similarity matching is performed between features, and the target with the highest matching degree is output as the association result of the most relevant image.

[0033] Furthermore, in step S7, when the user enters the search text, the visual language model CLIP is used to extract text features. ,Then the text features are mapped to the multimodal embedding space, and the image with the highest cosine similarity matching is output as the retrieval result.

[0034] The beneficial effects of the present invention are as follows:

[0035] 1. The present invention adopts the modal center perception method to align the features of multimodal data, performs comparative learning through the twin network, optimizes cross-modal feature matching, and improves the expression consistency of data in different modalities; at the same time, the original dimension of the feature is restored through feature inverse mapping; during the training process, the contrast loss function is used to optimize the feature distribution, so that the features of different modalities maintain similarity in the common feature space and reduce the error of cross-modal matching.

[0036] 2. This invention uses random masked cross-modal feature fusion to mask some modal features and use other modal features to supplement information, achieving modal complementary enhancement. Combined with the cross-modal self-attention Transformer model, the masked unimodal features are used as queries, and the other modal features are fused as keys and values. This allows efficient feature expression to be maintained even in the absence of information, improving the robustness and fusion effect of unimodal features and enhancing the fusion capability of cross-modal features.

[0037] 3. The present invention constructs a feature matching network based on the cross-attention mechanism, calculates the cross-modal attention score through Softmax normalization, and adjusts the feature weighting to improve the matching accuracy; combines cosine similarity calculation to achieve efficient matching of target images, and improves the accuracy and robustness of the retrieval system; during the training process, combines contrast loss and symmetric cross-entropy loss to optimize the matching network, so as to maximize the correct text-image pair matching score, while reducing false matches and improving the retrieval ability of the model; based on the multimodal feature enhancement method of the large visual language model (CLIP), it extracts high-level semantic features through the visual language model, and combines cosine similarity matching to optimize the retrieval model, thereby improving the accuracy and generalization ability of cross-modal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0039] The following description clearly and completely describes the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0040] In the description of the embodiments of the present invention, it should be noted that the indicated orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, or the orientations or positional relationships in which the inventive product is typically placed when in use, or the orientations or positional relationships commonly understood by those skilled in the art, or the orientations or positional relationships in which the inventive product is typically placed when in use. These are merely for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as limiting the present invention. In addition, the terms "first" and "second" are used only to distinguish descriptions and should not be understood as indicating or implying relative importance.

[0041] In describing the embodiments of the present invention, it should be noted that, unless otherwise specified or limited, the terms "disposed" and "connected" should be understood broadly. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to direct connections or indirect connections through an intermediary. Those skilled in the art will understand the specific meanings of these terms in the present invention based on the specific circumstances.

[0042] Example 1

[0043] Embodiment 1 of the present invention discloses a multimodal unstructured content association retrieval method, such as Figure 1 As shown, the details are as follows:

[0044] S1: Extract features from multimodal data to obtain features of data in different modalities;

[0045] The multimodal data are images of different modalities, including images of different modalities such as visible light, infrared, and SAR.

[0046] In this embodiment, feature extraction is performed based on image data of three different modalities: visible light, infrared, and SAR, and multimodal features are output. ,in, represent visible light images, infrared images, and SAR images, respectively.

[0047] S2: Based on the obtained multimodal features, cross-modal global feature alignment is performed, and cross-modal pre-alignment of features of different modalities is performed in multiple feature spaces. Then, modal center perception learning is performed to achieve cross-modal fine alignment. Finally, the required aligned multimodal features are obtained through multimodal feature inverse mapping.

[0048] The specific process is as follows:

[0049] To enable cross-modal alignment of multimodal features, it is necessary to construct positive and negative sample pairs. Using a twin network structure, the model learns to discriminate the relationship between positive and negative samples, thereby performing comparative learning.

[0050] In this embodiment, during the multimodal feature alignment process, the same pair of multimodal features is used as a positive sample pair, and cross-domain multimodal features of different objects constitute a negative sample pair, thereby constructing positive and negative sample pairs.

[0051] The model maps different modal features into a common feature space, shortening the distance between positive sample pairs (multimodal features of the same object) while making the negative sample pairs as evenly distributed as possible.

[0052] In this embodiment, to guide the unified alignment of multimodal features, a unified alignment target is provided for features of two or more modalities. The center of the multimodal features of the same object is used as an anchor point and used as a positive sample in the contrastive learning process. The contrastive learning loss function is used for calculation. As the model learning process is continuously updated, the loss function is expressed as follows:

[0053]

[0054] in, is the similarity metric score between the current input modality feature and the multimodal feature center (positive sample pair), is the similarity measurement score between the current input object feature and other object features (negative sample pairs), is the temperature coefficient, which is used to adjust the degree of attention paid to positive and negative sample pairs during model learning. is the set of all object samples, is the set of all modes.

[0055] Contrastive Learning Alignment Loss It will shorten the distance between different modal features of the same object and improve the difference between features of different objects, thus achieving feature alignment;

[0056] Finally, the original dimension of the feature is restored by inverse mapping of the multimodal feature to obtain the aligned feature .

[0057] S3: Randomly mask one modality alignment feature and perform complementary masking on the other modality alignment features to generate an incomplete unimodal feature vector after masking;

[0058] A cross-modal self-attention Transformer model is used to fuse the masked multimodal feature vectors.

[0059] In the multimodal feature fusion process, due to the limited representation capabilities of a single modality, the complementary relationship between modalities is exploited to enrich and enhance the information-deficient single modal features, and then fuse the enhanced features. By using a masked autoencoder representation learning strategy and designing mutual masking between different modalities, the missing parts of the masked modality are predicted and restored using complementary information from other modalities, thereby improving the expressiveness of single modal features.

[0060] The specific process is as follows:

[0061] By randomly masking a portion of the input data, the remaining masked portion is fed into the autoencoder as visible data. The masked portion of the input is then reconstructed using the visible data to learn more expressive latent feature representations, improving the model's ability to learn local features and enabling prediction of missing information. Furthermore, in multi-modal scenarios, a multimodal mutual masking strategy is employed to simulate the incompleteness of some modal information that may exist in real environments. The complementarity between different modalities is then leveraged to supplement and enrich the modality with missing information, thereby achieving feature enhancement and improving multi-modal balance. Through intra-modal masking, an incomplete single-modal feature vector is obtained after masking.

[0062] In the cross-modal attention feature fusion process, taking unimodal features as an example, the incomplete unimodal feature vector after masking is used as the query input, and the multimodal feature vector is concatenated and fused as the key and value input to obtain the multimodal complementary features after cross-modal attention weighting, providing supplementary information for the masked modality;

[0063] In this embodiment, the multimodal complementary feature is obtained and calculated as follows:

[0064]

[0065] in, Compute cross-modal attention, is the feature vector of the visible part after masking, and are other modal eigenvectors, express and Feature dimension, T represents the transpose of the matrix.

[0066] The multimodal complementary features after cross-modal attention weighting are input into the ViT model in the cross-modal self-attention Transformer model and fused with the masked unimodal features, and the fused feature vector is used to predict the original content of the masked part. Since the original feature vector before the mask is known, there is no need to provide manual annotation information during the training process. The prediction results of the masked autoencoder can be supervised directly by reading the original feature data, and the reconstruction loss between the prediction results and the original features is calculated. By using the complementary information of cross-domain multimodal to provide guidance for the mask reconstruction process, the in-domain feature enhancement of modal complementarity is achieved, and finally the multimodal fusion features are output. .

[0067] Among them, ViT can be divided into three layers: Embedding layer, Transformer Encoder layer and MLPHead layer.

[0068] S4: Process the masked multimodal fusion features to extract enhanced features of each image;

[0069] In this embodiment, a large visual language model is used to process multimodal fusion features. This is different from the image feature processing method in existing image association retrieval algorithms. It can fully extract multimodal fusion feature information and obtain enhanced features for each image.

[0070] S5: The enhanced features are processed through the cross attention network to obtain the cosine similarity between different images;

[0071] The cosine similarity is calculated as follows:

[0072]

[0073] in, is the dot product of the vectors, and The cosine similarity value ranges from -1 to 1. The closer the value is to 1, the more similar the features of the two images are after the cross-attention network processing. The closer the value is to -1, the less similar they are. The closer the value is to 0, the weaker the correlation is.

[0074] S6: Matching image features by performing metric calculation based on the obtained cosine similarity, performing cosine similarity matching between features, and outputting the target with the highest matching degree as the association result of the most relevant image.

[0075] During training, the loss function for image association based on the crisscross attention network consists of a contrast loss based on cosine similarity. The loss function formula is as follows:

[0076]

[0077] in, represents the cosine similarity, is the target feature sample, samples can be obtained through data augmentation samples, is the cosine similarity between positive sample pairs, represents the i-th target feature sample, Represents The matching feature sample is the other sample in the positive sample pair. Indicates the kth feature sample in the 2N samples, which needs to be considered when calculating the loss function Cosine similarity with all other feature samples, It is a piecewise function, which is 1 when k≠i and 0 when k=i. The i-th function is excluded to prevent the denominator from being 0.

[0078] This loss function makes the similarity between matching feature samples higher and reduces the similarity between unmatched feature samples.

[0079] By minimizing the above loss function, the model can capture the relationship between images and compare the target feature matching to obtain the associated image .

[0080] S7: Based on the image association results obtained, the text features are obtained by inputting the retrieval text into the language model. Through the multimodal data embedding space, the cosine similarity is used to match the most similar images to obtain the image retrieval results.

[0081] When the user enters the search text, the visual model and language model are used to process the image multimodal fusion features and text data respectively to extract text features. and image features ,Then the text features are mapped to the multimodal embedding space, and the image with the highest cosine similarity matching is output as the retrieval result.

[0082] During the training process, in order to optimize the accuracy of text-image matching, maximize The cosine similarity of the matching image and text embeddings is minimized The cosine similarity of the embeddings of incorrect image and text pairs is calculated. These cosine similarity scores are optimized by the symmetric cross entropy loss, which is formulated as follows:

[0083]

[0084] in, represents the number of categories, is a sign function, if the sample The true category is equal to Take 1, otherwise take 0, Is the observed sample belonging to the category By minimizing the above loss function, the model can capture the relationship between text and image, and the image retrieval results can be obtained based on the multimodal embedding results. , Represents the loss value calculated for the i-th pair among N matching image-text pairs.

[0085] The present invention is not limited to the aforementioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A multimodal unstructured content association retrieval method, characterized in that: include: S1: Extract features from multimodal data to obtain features of data in different modalities; S2: Based on the obtained multimodal features, the multimodal features are aligned across modalities through modal center perception to obtain multimodal aligned features; The specific process is as follows: S201: Taking the same pair of multimodal features as a positive sample pair and the cross-domain multimodal features of different objects as a negative sample pair, constructing positive and negative sample pairs; S202: Using the multimodal feature centers of the same object as anchor points and as positive samples in the contrastive learning process, and using the contrastive learning loss function to perform contrastive learning to update model parameters; The loss function is expressed as follows: in, is the similarity measure score between the current input modality feature and the multimodal feature center, is the similarity measurement score between the current input object feature and other object features, is the temperature coefficient, which is used to adjust the degree of attention paid to positive and negative sample pairs during model learning. is the set of all object samples, is the set of all modes; S203: Mapping different modal features into a common feature space through the trained model; S204: Restoring the original dimension of the feature through inverse mapping of the multimodal feature to obtain aligned multimodal features; S3: Randomly mask the multimodal alignment features, and send the masked feature vector to the cross-modal self-attention Transformer model for fusion to obtain the masked multimodal fusion feature vector; S4: Process the masked multimodal fusion features to extract enhanced features of each image; S5: The enhanced features are processed through the cross attention network to obtain the cosine similarity between different images; S6: Matching image features by performing metric calculation based on the obtained cosine similarity to obtain an image association result; S7: Based on the image association results obtained, the text features are obtained by inputting the retrieval text into the language model. Through the multimodal data embedding space, the cosine similarity is used to match the most similar images to obtain the image retrieval results.

2. The multimodal unstructured content association retrieval method according to claim 1, characterized in that: In step S1, the multimodal data are images of different modalities, including visible light, infrared, and SAR images.

3. The multimodal unstructured content association retrieval method according to claim 1, characterized in that: In step S3, multimodal feature fusion is performed. The specific process is as follows: Randomly mask one modality alignment feature and perform complementary masking on the other modality alignment features to generate an incomplete unimodal feature vector after masking; The masked incomplete unimodal feature vector is used as the query input, and the multimodal feature vectors are concatenated and fused as the key and value input to obtain the multimodal complementary features after cross-modal attention weighting. The features are then input into the ViT model in the cross-modal self-attention Transformer model for fusion, and the multimodal fusion features are output.

4. The multimodal unstructured content association retrieval method according to claim 3, characterized in that: The multimodal complementary feature is obtained and calculated as follows: in, Compute cross-modal attention, is the feature vector of the visible part after masking, and are other modal eigenvectors, express and Feature dimension, T represents the transpose of the matrix.

5. The multimodal unstructured content association retrieval method according to claim 1, characterized in that: In step S5, during the training of the cross-attention network, the loss function is composed of a contrast loss based on cosine similarity, which is expressed as follows: in, represents the cosine similarity, is the target feature sample, samples can be obtained through data augmentation samples, is the cosine similarity between positive sample pairs, represents the i-th target feature sample, Represents Matched feature samples, Indicates the kth feature sample in 2N samples, is a piecewise function, which is 1 when k≠i and 0 when k=i.

6. The multimodal unstructured content association retrieval method according to claim 1, characterized in that: In step S6, the image association result is obtained through feature matching, which is as follows: Cosine similarity matching is performed between features, and the target with the highest matching degree is output as the association result of the most relevant image.

7. The multimodal unstructured content association retrieval method according to claim 1, characterized in that: In step S7, when the user enters the search text, the visual language model CLIP is used to extract text features ,Then the text features are mapped to the multimodal embedding space, and the image with the highest cosine similarity matching is output as the retrieval result.

Images