Multi-modal unstructured content association retrieval method

Abstract

The invention discloses a multi-modal unstructured content association retrieval method, which comprises the following steps of: performing feature extraction on multi-modal data to obtain features of the data in different modals; the multi-modal features are aligned, and multi-modal alignment features are obtained; random masking is carried out on the multi-modal alignment features, the multi-modal alignment features are sent into a cross-modal self-attention model to be fused, and multi-modal fusion feature vectors after masking are obtained; extracting enhanced features of each image; processing the enhanced features through a cross attention network to obtain cosine similarity between different images; performing image feature matching to obtain an image association result; a retrieval text is input into a large language model to obtain text features, through a multi-modal data embedding space, the most similar image is matched by using cosine similarity, and a retrieval result of the image is obtained. According to the method, the accuracy and the stability of multi-modal feature fusion are improved, and the accuracy and the efficiency of image retrieval are improved.

Description

Technical Field

[0001] The present invention relates to the field of retrieval technology, and in particular to a multimodal unstructured content association retrieval method. Background Art

[0002] Multimodal data association retrieval requires the ability to input an image or text and find the corresponding associated images and search content that matches the text. Currently, multimodal image data includes visible light imaging, infrared imaging, and SAR imaging. Compared to infrared and visible light imaging, SAR imaging has stable detection capabilities. In contrast, the other two types of imaging are susceptible to high-dynamic environments (high-speed targets can cause blurring and poor image quality). The unstable output of these sensors can lead to unpredictable modal structuring, resulting in significant errors in the acquired information.

[0003] Existing technologies for multimodal data association retrieval have shortcomings, such as difficulty matching image content with retrieval text, data distribution discrepancies, and ambiguous cross-modal feature expressions. Traditional methods typically only consider feature alignment between two modalities and fail to fully exploit intermodal associations when fusing multimodal features. This results in low image association and retrieval accuracy, making them inadequate for complex scenarios. Summary of the Invention

[0004] To solve the above problems, the present invention provides a multimodal unstructured content association retrieval method, which makes full use of multimodal feature information, aligns the multimodal data obtained by the visual language large model, eliminates the problem of data unstructuredness, and fuses the multimodal data to obtain comprehensive information perception. It also uses cross-attention network processing and cosine similarity comparison to realize the association and retrieval of multimodal data content, improves the accuracy and efficiency of image retrieval, and better adapts to the needs of multimodal image data processing and application scenarios in complex scenarios.

[0005] The present invention provides a multimodal unstructured content association retrieval method, the specific technical solution is as follows: S1: Extract features from multimodal data to obtain features of data in different modalities; S2: Align the multimodal features according to the obtained multimodal features to obtain multimodal alignment features; S3: Randomly mask the multimodal alignment features, and send the masked feature vector to the cross-modal self-attention Transformer model for fusion to obtain the masked multimodal fusion feature vector; S4: Process the masked multimodal fusion features to extract enhanced features of each image; S5: Process the enhanced features through a cross-attention network to obtain the cosine similarity between different images; S6: According to the obtained cosine similarity, perform metric calculation to match the image features and obtain the association result of the images; S7: According to the obtained image association result, input the retrieval text into the language large model to obtain text features. Through the multi-modal data embedding space, use cosine similarity to match the most similar image and obtain the image retrieval result.

[0006] Further, in step S1, the multi-modal data is images of different modalities, including visible light, infrared, and SAR images.

[0007] Further, in step S2, cross-modal global feature alignment of multi-modal features is performed through modality center perception. The specific process is as follows: S201: Use the same pair of multi-modal features as the positive sample pair, and form negative sample pairs between cross-domain multi-modal features of different objects to construct positive and negative sample pairs; S202: Use the multi-modal feature center of the same object as the anchor point and perform contrast learning to update the model parameters; S203: Map different modality features into a common feature space through the trained model; S204: Restore the original dimension of the features through multi-modal feature inverse mapping to obtain aligned multi-modal features.

[0008] Further, use the multi-modal feature center of the same object as the anchor point and as the positive sample in the contrast learning process. Adopt the contrast learning loss function to perform model update training. The loss function is expressed as follows:

[0009] Among them, is the similarity measurement score between the current input modality feature and the multi-modal feature center, is the similarity measurement score between the current input object feature and the features of other objects, is the temperature coefficient, which is used to adjust the attention degree of the model to positive and negative sample pairs during learning, is the set of all object samples, is the set of all modalities.

[0010] Further, in step S3, multi-modal feature fusion is performed. The specific process is as follows: Randomly mask one modality aligned feature and perform complementary masking on the aligned features of other modalities to generate masked incomplete single-modal feature vectors; The masked incomplete unimodal feature vector is used as the query input, and the multimodal feature vectors are concatenated and fused as the key and value input to obtain the multimodal complementary features after cross-modal attention weighting. The features are then input into the ViT model in the cross-modal self-attention Transformer model for fusion, and the multimodal fusion features are output.

[0011] Furthermore, the multimodal complementary feature is obtained and calculated as follows:

[0012] in, Compute cross-modal attention, is the feature vector of the visible part after masking, and are other modal eigenvectors, express and Feature dimension, T represents the transpose of the matrix.

[0013] Furthermore, in step S5, during the training of the cross-attention network, the loss function is composed of a contrast loss based on cosine similarity, which is expressed as follows:

[0014] in, represents the cosine similarity, is the target feature sample, samples can be obtained through data augmentation samples, is the cosine similarity between positive sample pairs, represents the i-th target feature sample, Represents Matched feature samples, Indicates the kth feature sample in 2N samples, is a piecewise function, which is 1 when k≠i and 0 when k=i.

[0015] Furthermore, in step S6, image association results are obtained through feature matching, as follows: Cosine similarity matching is performed between features, and the target with the highest matching degree is output as the association result of the most relevant image.

[0016] Furthermore, in step S7, when the user enters the search text, the visual language model CLIP is used to extract text features. ,Then the text features are mapped to the multimodal embedding space, and the image with the highest cosine similarity matching is output as the retrieval result.

[0017] The beneficial effects of the present invention are as follows: 1. The present invention uses a modal center perception method to perform feature alignment on multi-modal data, conducts contrastive learning through a siamese network, optimizes cross-modal feature matching, and improves the expression consistency of different modal data; at the same time, it restores the original dimension of the features through feature inverse mapping; during the training process, a contrastive loss function is used to optimize the feature distribution, so that different modal features maintain similarity in the common feature space and reduce the error of cross-modal matching.

[0018] 2. The present invention is based on cross-modal feature fusion with random masking. It masks some modal features and uses other modal features for information supplementation to achieve modal complementary enhancement. Combining with the cross-modal self-attention Transformer model, the masked single-modal features are used as Query, and other modal features are used as Key and Value for fusion, so as to maintain efficient feature expression even in the case of missing information, improve the robustness and fusion effect of single-modal features, and enhance the cross-modal feature fusion ability. 3. The present invention constructs a feature matching network based on the cross-attention mechanism, calculates the cross-modal attention score through Softmax normalization, and adjusts feature weighting to improve the matching accuracy; combined with cosine similarity calculation, it realizes the efficient matching of the target image and improves the accuracy and robustness of the retrieval system; during the training process, the matching network is optimized by combining contrastive loss and symmetric cross-entropy loss, so that the matching score of the correct text-image pair is maximized, while reducing incorrect matches and improving the retrieval ability of the model; based on the multi-modal feature enhancement method of the vision-language large model (CLIP), it extracts high-level semantic features through the vision-language model and optimizes the retrieval model by combining cosine similarity matching to improve the accuracy and generalization ability of cross-modal retrieval. Description of the Drawings

[0019] Figure 1 It is a schematic flowchart of the method of the present invention. Detailed Embodiments

[0020] In the following description, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0021] In the description of the embodiments of the present invention, it should be noted that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed during use, or the orientation or positional relationship commonly understood by those skilled in the art, or the orientation or positional relationship in which the invention product is usually placed during use. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention. In addition, the terms "first" and "second" are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0022] In the description of the embodiments of the present invention, it should also be noted that unless otherwise clearly specified and limited, the terms "set" and "connect" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] Embodiment 1 Embodiment 1 of the present invention discloses a multi-modal unstructured content association retrieval method, as Figure 1 shown below: S1: Extract features from multi-modal data to obtain features of data in different modalities; The multi-modal data are images in different modalities, including images in different modalities such as visible light, infrared, and SAR.

[0024] In this embodiment, according to the image data in three different modalities of visible light, infrared, and SAR, feature extraction is performed, and multi-modal features are output where, respectively represent visible light images, infrared images, and SAR images.

[0025] S2: According to the obtained multi-modal features, perform cross-modal global feature alignment. Pre-align the features in different modalities in multiple feature spaces, then perform modality center perception learning to achieve cross-modal fine alignment, and finally obtain the required aligned multi-modal features through multi-modal feature inverse mapping.

[0026] The specific process is as follows: To enable cross-modal alignment of multi-modal features, it is necessary to construct positive and negative sample pairs, and through the siamese network structure, enable the model to learn the ability to distinguish the relationship between positive and negative samples for contrast learning; In this embodiment, during the multi-modal feature alignment process, the same pair of multi-modal features is used as a positive sample pair, and the cross-domain multi-modal features of different objects form a negative sample pair to construct positive and negative sample pairs.

[0027] The model maps different modal features into a common feature space, reducing the distance between positive sample pairs (multi-modal features of the same object), and at the same time making the negative sample pairs as evenly distributed as possible. In this embodiment, to guide the unified alignment between multi-modal features, a unified alignment objective is provided for features of more than two modalities. The center of the multi-modal features of the same object is used as an anchor point, which serves as the positive sample during the contrastive learning process, and a contrastive learning loss function is used for calculation. As the model learning process progresses, the loss function is expressed as follows:

[0028] Where, is the similarity metric score between the current input modal feature and the multi-modal feature center (positive sample pair), is the similarity metric score between the current input object feature and other object features (negative sample pair), is the temperature coefficient, which is used to adjust the attention degree of the model to positive and negative sample pairs during learning, is the set of all object samples, is the set of all modalities.

[0029] Contrastive learning alignment loss will reduce the distance between different modal features of the same object and increase the difference between features of different objects, thereby achieving feature alignment; Finally, the original dimension of the feature is restored through the inverse mapping of the multi-modal feature to obtain the aligned feature .

[0030] S3: Randomly mask one modal alignment feature and perform complementary masking on other modal alignment features to generate masked incomplete single-modal feature vectors; Use a cross-modal self-attention Transformer model to fuse the masked multi-modal feature vectors.

[0031] During the multi-modal feature fusion process, due to the limited single-modal representation ability, by utilizing the complementary relationship between modalities, the insufficient single-modal features are enriched and enhanced, and then the enhanced features are fused. By using the masked autoencoder representation learning strategy and designing mutual masking between different modalities, the missing part of the masked modality is predicted and restored using complementary information from other modalities, thereby improving the expressiveness of single-modal features.

[0032] The specific process is as follows: Randomly mask a part of the input data, take the remaining unmasked part as visible data and input it into the autoencoder, and use the visible part of the data to reconstruct the masked part of the input, learn a more expressive latent feature representation, improve the local feature learning ability of the model, and realize the prediction of the missing part of the information. And in multiple modal scenarios, adopt a multi-modal mutual masking strategy to simulate the situation where some modal information may be incomplete in the actual environment, and then use the complementarity between different modalities to provide supplementation and enrichment for the modality with missing information, so as to achieve feature enhancement and improve the balance between multiple modalities. Through intra-modal masking, an incomplete single-modal feature vector after masking is obtained.

[0033] In the process of cross-modal attention feature fusion, taking the single-modal feature as an example, the incomplete single-modal feature vector after masking is used as the Query input, and the multi-modal feature vector after splicing and fusion is used as the Key and Value inputs to obtain the multi-modal complementary feature after cross-modal attention weighting, providing supplementary information for the masked modality; In this embodiment, the multi-modal complementary feature is obtained and calculated as follows:

[0034] Among them, Calculate cross-modal attention, is the visible part feature vector after masking, and are other modal feature vectors, represents and The feature dimension, and T represents the transpose of the matrix.

[0035] Input the multi-modal complementary feature after cross-modal attention weighting into the ViT model in the cross-modal self-attention Transformer model and fuse it with the masked single-modal feature, and use the fused feature vector to predict the original content of the masked part. Since the original feature vector before masking is known, there is no need to provide artificial annotation information during the training process, and the prediction result of the masked autoencoder can be supervised directly by reading the original feature data, and the reconstruction loss between the prediction result and the original feature is calculated. By using the complementary information of cross-domain multi-modalities to guide the masking reconstruction process, the in-domain feature enhancement of modal complementarity is realized, and finally the multi-modal fusion feature is output. .

[0036] Among them, ViT can be divided into three layers, namely the Embedding layer, the Transformer Encoder layer, and the MLPHead layer.

[0037] S4: Process the masked multi-modal fusion features to extract the enhanced features of each image; In this embodiment, a vision-language large model is used to process the multi-modal fusion features. Different from the methods for processing image features in existing image correlation retrieval algorithms, it can fully extract the information of the multi-modal fusion features and obtain the enhanced features of each image.

[0038] S5: Process the enhanced features through a cross-attention network to obtain the cosine similarity between different images; The cosine similarity is calculated as follows:

[0039] where, is the dot product of vectors, and are the norms of two vectors respectively. The value of this cosine similarity is between -1 and 1. The closer the value is to 1, the more similar the features of the two images are after being processed by the cross-attention network. The closer it is to -1, the less similar they are. Being close to 0 indicates weak correlation.

[0040] S6: According to the obtained cosine similarity, perform metric calculation to match the image features, perform cosine similarity matching between the features, and output the target with the highest matching degree as the correlation result of the most relevant image.

[0041] During the training process, the loss function of image correlation based on the cross-attention network consists of a contrast loss based on cosine similarity. The formula of the loss function is expressed as follows:

[0042] where, represents the cosine similarity, is the target feature sample. By data augmentation of samples, samples can be obtained. is the cosine similarity between positive sample pairs, represents the i-th target feature sample, represents the feature sample that matches , that is, the other sample in the positive sample pair. represents the k-th feature sample among the 2N samples. When calculating the loss function, the cosine similarity between and all other feature samples needs to be considered. is a piecewise function, which is 1 when k≠i and 0 when k = i, excluding the i-th to prevent the denominator from being 0.

[0043] This loss function makes the similarity between matching feature samples higher and reduces the similarity between mismatching feature samples.

[0044] By minimizing the above loss function, the model can capture the relationships between images and obtain associated images by comparing target feature matches. 。

[0045] S7: According to the obtained image association results, input the retrieval text into the language large model to obtain text features. Through the multi-modal data embedding space, use cosine similarity to match the most similar image to obtain the image retrieval result.

[0046] When the user inputs the retrieval text, the visual large model and the language large model are used to process the image multi-modal fusion features and text data respectively to extract text features and image features , and then map the text features to the multi-modal embedding space, and output the image with the highest cosine similarity matching degree as the retrieval result.

[0047] During the training process, to optimize the accuracy of text-image matching, maximize the cosine similarity of the embeddings of mutually matching images and texts, and at the same time minimize the cosine similarity of the embeddings of incorrect image and text pairs. These cosine similarity scores are optimized by the symmetric cross-entropy loss, and the formula is as follows:

[0048] where, represents the number of categories, is the sign function. If the true category of the sample is equal to , take 1, otherwise take 0, is the predicted probability that the observed sample belongs to category . By minimizing the above loss function, the model can capture the relationship between text and image, and the image retrieval result can be obtained according to the multi-modal embedding result , represents the loss value calculated for the i-th pair among N matching image-text pairs.

[0049] The present invention is not limited to the foregoing specific embodiments. The present invention extends to any new feature or any new combination disclosed in this specification, as well as any new method or process step or any new combination disclosed.

Claims

1. A multimodal unstructured content associated retrieval method, characterized in that, Including: S1: Extract features from multi-modal data to obtain features of data in different modalities; S2: Align the multi-modal features according to the obtained multi-modal features to obtain multi-modal aligned features; S3: Randomly mask the multi-modal aligned features, and send the masked feature vectors into a cross-modal self-attention Transformer model for fusion to obtain masked multi-modal fusion feature vectors; S4: Process the masked multi-modal fusion features to extract enhanced features of each image; S5: Process the enhanced features through a cross-attention network to obtain the cosine similarity between different images; S6: According to the obtained cosine similarity, perform metric calculation to match image features to obtain the association result of the images; S7: According to the obtained image association result, input the retrieved text into a language large model to obtain text features, and use cosine similarity matching in the multi-modal data embedding space to find the most similar image to obtain the retrieval result of the image.

2. The multimodal unstructured content associated retrieval method according to claim 1, wherein In step S1, the multi-modal data is images in different modalities, including visible light, infrared, and SAR images.

3. The multimodal unstructured content correlation retrieval method according to claim 1, characterized in that In step S2, cross-modal global feature alignment of multi-modal features is performed through modal center perception. The specific process is as follows: S201: Use the same pair of multi-modal features as positive sample pairs, and form negative sample pairs between cross-domain multi-modal features of different objects to construct positive and negative sample pairs; S202: Use the multi-modal feature center of the same object as an anchor point to perform contrastive learning to update the model parameters; S203: Map different modality features into a common feature space through the trained model; S204: Restore the original dimension of the features through inverse mapping of multi-modal features to obtain aligned multi-modal features.

4. The multimodal unstructured content association retrieval method according to claim 3, characterized in that Use the multi-modal feature center of the same object as an anchor point and as a positive sample in the contrastive learning process. Adopt a contrastive learning loss function to perform model update training. The loss function is expressed as follows: wherein, is the similarity metric score between the current input modal feature and the multi-modal feature center, is the similarity metric score between the current input object feature and other object features, is the temperature coefficient, which is used to adjust the attention degree of the model to positive and negative sample pairs during learning, is the set of all object samples, is the set of all modalities.

5. The multimodal unstructured content correlation retrieval method according to claim 1, characterized in that In step S3, multi-modal feature fusion is performed. The specific process is as follows: Randomly mask one modality of aligned features and perform complementary masking on other modality aligned features to generate masked incomplete single-modal feature vectors; Use the masked incomplete single-modal feature vectors as Query inputs, and use the concatenated and fused multi-modal feature vectors as Key and Value inputs to obtain multi-modal complementary features weighted by cross-modal attention, and input them into the ViT model in the cross-modal self-attention Transformer model for fusion to output multi-modal fusion features.

6. The multimodal unstructured content associated retrieval method according to claim 5, wherein Obtain the multi-modal complementary features, and the calculation is as follows: Among them, Calculate cross-modal attention, is the visible part feature vector after masking, and is the feature vector of other modalities, denotes and feature dimensions, and T represents the transpose of the matrix.

7. The multimodal unstructured content associated retrieval method according to claim 1, wherein In step S5, during the training process of the cross-attention network, the loss function consists of a contrastive loss based on cosine similarity, and is expressed as follows: Among them, represents the cosine similarity, is the target feature sample. By data augmentation on samples, the cosine similarity between positive sample pairs, represents the i-th target feature sample, represents the feature sample that matches represents the k-th feature sample among the 2N samples, is a piecewise function, which is 1 when k≠i and 0 when k = i.

8. The multimodal unstructured content associated retrieval method according to claim 1, wherein In step S6, through feature matching, obtain the image association result, specifically as follows: Perform cosine similarity matching between features, and output the target with the highest matching degree as the association result of the most relevant image.

9. The multimodal unstructured content associated retrieval method according to claim 1, wherein In step S7, when the user inputs the retrieval text, the visual language model CLIP is used to extract the text features , and then the text features are mapped to the multi-modal embedding space, and the image with the highest cosine similarity matching degree is output as the retrieval result.

Images