Training method and retrieval method of cross-modal feature fusion model

Abstract

The invention discloses a training method of a cross-modal feature fusion model and a retrieval method based on the cross-modal feature fusion model. The cross-modal feature fusion model comprises an LLM, a CLIP, a linear layer and a cross-modal attention layer. The output end of the LLM is connected with the input end of the cross-modal attention layer through the linear layer; the output end of the CLIP is connected with the input end of the cross-modal attention layer; according to the method, the accuracy and robustness of the knowledge recall task are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and more specifically to a training method and a retrieval method for a cross-modal feature fusion model. Background Art

[0002] In recent years, with the rapid development of deep learning and large-scale pre-training models, cross-modal retrieval technology has gradually become a research hotspot in the field of artificial intelligence, especially in the joint understanding and information retrieval of text and images. Cross-modal retrieval aims to achieve information exchange and intelligent matching between modalities by representing and aligning the features of different modal data (such as text and images).

[0003] At present, most mainstream cross-modal retrieval methods adopt an architecture that encodes text and images separately, extracts features from text and images through independent encoders, and aligns them in a high-dimensional feature space. For example, a research paper related to the present invention, "LLM2CLIP: POWERFUL LANGUAGE MODEL UNLOCK RICHER VISUALREPRESENTATION", proposes a cross-modal representation model based on separate encoding of text and images. The model uses the powerful features of a large language model (LLM) and a visual encoder (CLIP) to extract semantic features of text and images respectively. However, this separate encoding method still has the following shortcomings in specific application scenarios:

[0004] Insufficient semantic fusion capability: Since text and image features are extracted independently, they fail to achieve deep fusion in the encoding stage. This lack of interaction limits the performance of the model when high semantic consistency or fine-grained feature understanding of complex scenes is required, resulting in a decrease in the accuracy and reliability of the recall results.

[0005] Insufficient expression of fine-grained features: In the recall stage, existing methods are limited by the resolution of the visual encoding model and find it difficult to effectively retain and extract detailed information in the image. For example, in scenes where small targets or complex backgrounds need to be focused on, fine-grained features are often ignored during the encoding process, affecting the effect and accuracy of knowledge recall.

[0006] Therefore, how to provide a training method and a retrieval method for a cross-modal feature fusion model with better semantic fusion capability and can effectively retain and extract fine-grained features of images is an urgent problem that needs to be solved by technical personnel in this field. Summary of the invention

[0007] In view of this, an object of the present invention is to provide a training method and a retrieval method for a cross-modal feature fusion model.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A training method for a cross-modal feature fusion model, including a training method for LLM and a joint training method for CLIP, a linear layer, and a cross-modal attention layer:

[0010] The LLM training method specifically includes:

[0011] Get A training images and generate W descriptive captions for each training image;

[0012] Select S descriptive captions from the same training image as positive sample pairs; where 2≤S≤W;

[0013] Select S descriptive captions from different training images as negative sample pairs;

[0014] Input the positive sample pairs and the negative sample pairs into the LLM, and use the first simCSE loss function to adjust the parameters of the LLM to obtain the pre-trained LLM;

[0015] The joint training method of CLIP, linear layer, and cross-modal attention layer specifically includes:

[0016] Acquire a joint training sample set; wherein each training sample in the joint training sample set includes a training image and one of S descriptive titles selected from the training image;

[0017] Freeze the gradients of the pre-trained LLM;

[0018] Input the training images in the training samples into CLIP, mask the descriptive titles in the training samples and input them into the pre-trained LLM, and use the second simCSE loss function to adjust the parameters of CLIP, linear layer, and cross-modal attention layer to obtain pre-trained CLIP, pre-trained linear layer, and pre-trained cross-modal attention layer;

[0019] Among them, the cross-modal feature fusion model includes LLM, CLIP, linear layer and cross-modal attention layer;

[0020] The output end of the LLM is connected to the input end of the cross-modal attention layer through the linear layer;

[0021] The output end of the CLIP is connected to the input end of the cross-modal attention layer;

[0022] The pre-trained LLM, the pre-trained CLIP, the pre-trained linear layer, and the pre-trained cross-modal attention layer constitute a pre-trained cross-modal feature fusion model.

[0023] Preferably, the expression of the first simCSE loss function is:

[0024]

[0025]

[0026]

[0027] Among them, L simCSE represents the first simCSE loss function; sim represents cosine similarity; h i represents the embedding value obtained by inputting one of the S descriptive captions selected from the i-th training image into the LLM; represents the embedding value obtained by inputting another descriptive caption from the S descriptive captions selected from the i-th training image into the LLM; represents the embedding value obtained by inputting the jth descriptive title into LLM, where the jth descriptive title represents a descriptive title in all positive and negative sample pairs, and the jth descriptive title does not include h i Corresponding descriptive title; τ represents the temperature coefficient.

[0028] Preferably, the expression of the second simCSE loss function is:

[0029]

[0030]

[0031]

[0032] Among them, L simCSE ' represents the second simCSE loss function; sim represents the cosine similarity; H i represents the embedding value obtained by inputting one of the S descriptive captions selected from the i-th training image into the cross-modal feature fusion model; represents the embedding value obtained by inputting another descriptive caption from the S descriptive captions selected from the i-th training image into the cross-modal feature fusion model; represents the embedding value obtained by inputting the jth descriptive title into the cross-modal feature fusion model, wherein the jth descriptive title represents a descriptive title in all positive and negative sample pairs, and the jth descriptive title does not include H i Corresponding descriptive title; τ represents the temperature coefficient.

[0033] Preferably, the causal attention machine of LLM is replaced with a bidirectional attention mechanism.

[0034] Preferably, the bidirectional attention mechanism is trained using the MNTP method.

[0035] Preferably, the LLM includes a first prompt template and a second prompt template;

[0036] The first prompt template is: given a title, retrieve a detailed related title;

[0037] The second prompt template is: given a detailed title, retrieve a short related title.

[0038] A retrieval method based on a cross-modal feature fusion model is implemented using the pre-trained cross-modal feature fusion model described above, and includes the following steps:

[0039] S1: Input the image to be retrieved and the question to be retrieved into the pre-trained cross-modal feature fusion model to obtain the text-image mixed embedding value;

[0040] Inputting the image to be retrieved into the target detection model to obtain N target images;

[0041] Input N target images into the pre-trained LLM; input the question to be retrieved into the pre-trained CLIP to obtain N target embedding values;

[0042] S2: Use the mixed embedding value of the text image and the N target embedding values ​​to perform knowledge retrieval to obtain relevant knowledge corresponding to the image to be retrieved and the question to be retrieved.

[0043] Preferably, the relevant knowledge is obtained based on the following formula:

[0044]

[0045]

[0046] D final =D IT ∪D OT ;

[0047] Among them, max Top-M Indicates taking The first M maximum values ​​in ; Q represents the mixed embedding value of the text image; Indicates D j The transpose of D j Represents the embedding value of the jth text in the database document; max Top-K Indicates taking The first K largest values ​​in Qi represents the i-th target embedding value; D final Represents the relevant knowledge.

[0048] Preferably, the database document comprises: inputting W descriptive captions of each training image into a pre-trained LLM, and outputting embedding values ​​of A*W descriptive captions from the LLM.

[0049] Preferably, LLM is Llama-38B; CLIP is CLIP-ViT-L; and the target detection model is YOLO World.

[0050] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a training method for a cross-modal feature fusion model and a retrieval method based on the cross-modal feature fusion model, which can achieve the following beneficial technical effects:

[0051] 1) This paper introduces a cross-attention layer to significantly improve the depth of feature interaction between text and image. Compared with the traditional separation encoding method, this paper can more accurately capture the complex semantic associations between text and image, and construct a more integrated cross-modal feature space, thereby significantly improving the accuracy and robustness of the knowledge recall task.

[0052] 2) This paper combines target detection technology and proposes a fine-grained retrieval strategy based on target detection. By detecting and segmenting key objects in the image, the detailed features contained in the image are fully retained and utilized. This strategy enables the model to perform better when dealing with complex scenes and hidden targets, especially in tasks that require detail distinction.

[0053] 3) This paper adopts a dual-path retrieval strategy, combining the query results based on the original image and the target detection image, and integrating the recall results through the knowledge fusion mechanism, thus improving the overall accuracy of the retrieval task. The dual-path design not only makes up for the possible shortcomings of single-path retrieval, but also provides higher reliability and efficiency for knowledge recall tasks in complex scenarios.

[0054] 4) In actual tests of firefighting scenarios, the present invention significantly surpassed the baseline model and demonstrated extremely high scene adaptation capabilities

[0055] 5) The present invention is not only applicable to firefighting scenarios, but also has strong versatility and can be widely used in various scenarios such as intelligent monitoring, risk identification, multimodal retrieval, etc. Its innovative design provides efficient and reliable technical solutions for complex tasks and has significant advantages in multimodal tasks that require high precision and high robustness.

[0056] 6) The present invention significantly reduces the computational cost of task execution and improves the overall processing efficiency through optimized cross-modal interaction mechanism, target detection technology, and efficient knowledge recall method, providing important support for the intelligent development of multimodal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0058] Figure 1 A schematic diagram of the structure of the cross-modal feature fusion model provided by the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The embodiment of the present invention discloses a training method for a cross-modal feature fusion model, including a training method for an LLM (large language model) and a joint training method for a CLIP (visual encoder), a linear layer, and a cross-modal attention layer:

[0061] The LLM training method specifically includes:

[0062] Get A training images and generate W descriptive captions for each training image;

[0063] Select S descriptive captions from the same training image as positive sample pairs; where 2≤S≤W;

[0064] Select S descriptive captions from different training images as negative sample pairs;

[0065] In one embodiment:

[0066] Apply ShareGPT4V to each training image to generate five descriptive captions that describe the semantic information of the image from different perspectives;

[0067] Select the top-2 descriptive captions from the same training image as positive sample pairs;

[0068] Randomly select two different training images and select their top-1 descriptive captions from these two different training images as negative sample pairs;

[0069] The present invention constructs a comparative text learning task through positive sample pairs and negative sample pairs, and designs a title-to-title retrieval experiment in the model evaluation stage. Through comparative learning, the titles of the same image are more closely combined in the semantic space, and the titles of different images are separated as much as possible, thereby optimizing the retrieval results.

[0070] Input the positive sample pairs and the negative sample pairs into the LLM, and use the first simCSE loss function to adjust the parameters of the LLM to obtain the pre-trained LLM;

[0071] In one embodiment, the expression of the first simCSE loss function is:

[0072]

[0073]

[0074]

[0075] Among them, L simCSE represents the first simCSE loss function; sim represents cosine similarity; h i represents the embedding value obtained by inputting one of the S descriptive captions selected from the i-th training image into the LLM; represents the embedding value obtained by inputting another descriptive caption from the S descriptive captions selected from the i-th training image into the LLM; represents the embedding value obtained by inputting the jth descriptive title into LLM, where the jth descriptive title represents a descriptive title in all positive and negative sample pairs, and the jth descriptive title does not include h i Corresponding descriptive title; τ represents the temperature coefficient.

[0076] The joint training method of CLIP, linear layer, and cross-modal attention layer specifically includes:

[0077] Acquire a joint training sample set; wherein each training sample in the joint training sample set includes a training image and one of S descriptive titles selected from the training image;

[0078] In one embodiment, one of the S descriptive captions selected from the training image is a Top-1 descriptive caption generated for the training image;

[0079] Freeze the gradients of the pre-trained LLM;

[0080] In order to maintain the inherent capabilities of the large language model, the present invention freezes the gradient of the LLM during the training phase so that its core properties and pre-trained semantic knowledge can be fully preserved.

[0081] Input the training images in the training samples into CLIP, mask the descriptive titles in the training samples (specifically, mask the nouns in the post-descriptive titles) and then input them into the pre-trained LLM, and use the second simCSE loss function to adjust the parameters of CLIP, linear layer, and cross-modal attention layer to obtain pre-trained CLIP, pre-trained linear layer, and pre-trained cross-modal attention layer;

[0082] During the training process, the present invention jointly optimizes CLIP, linear layer, and cross-modal attention layer to ensure the collaborative working efficiency of each module in the cross-modal feature space. By optimizing the second simCSE loss function, the model can improve the alignment effect of image and text.

[0083] In one embodiment, the expression of the second simCSE loss function is:

[0084]

[0085]

[0086]

[0087] Among them, L simCSE ' represents the second simCSE loss function; sim represents the cosine similarity; H i represents the embedding value obtained by inputting one of the S descriptive captions selected from the i-th training image into the cross-modal feature fusion model; represents the embedding value obtained by inputting another descriptive caption from the S descriptive captions selected from the i-th training image into the cross-modal feature fusion model; represents the embedding value obtained by inputting the jth descriptive title into the cross-modal feature fusion model, wherein the jth descriptive title represents a descriptive title in all positive and negative sample pairs, and the jth descriptive title does not include H i Corresponding descriptive title; τ represents the temperature coefficient.

[0088] The present invention adopts a supervised contrastive learning strategy (Supervised SimCSE Loss) to fine-tune the LLM, further improving the alignment ability of the model.

[0089] Among them, cross-modal feature fusion models (such as Figure 1 As shown) includes LLM, CLIP, linear layer and cross-modal attention layer;

[0090] The output end of the LLM is connected to the input end of the cross-modal attention layer through the linear layer;

[0091] The output end of the CLIP is connected to the input end of the cross-modal attention layer;

[0092] It is understandable that in order to compensate for the insufficient cross-modal feature alignment capability that may be caused by freezing LLM, the present invention introduces a new linear layer as an adapter at the back end of LLM. The learnable parameters of the linear layer can effectively improve the alignment capability between LLM and CLIP by optimizing.

[0093] To further enhance the semantic interaction between text and image features, the present invention introduces a cross-attention mechanism after the linear layer and the visual encoder. This mechanism captures the deep semantic relationship between text and image, realizes the hybrid encoding of the two, and fully captures the correspondence between image and text in the semantic space, thereby improving the ability to understand complex scenes and fine-grained features.

[0094] The pre-trained LLM, the pre-trained CLIP, the pre-trained linear layer, and the pre-trained cross-modal attention layer constitute a pre-trained cross-modal feature fusion model.

[0095] In one embodiment, the causal attention machine of the LLM is replaced with a bidirectional attention mechanism.

[0096] In order to enhance the text encoding capability, the present invention transforms the causal attention mechanism of LLM (specifically Llama-38B in one embodiment) into a bidirectional attention mechanism. This modification enables the model to not only encode the input sequence using contextual relationships, but also improves its robustness in contrastive learning tasks.

[0097] In one embodiment, the bidirectional attention mechanism is trained using the MNTP method.

[0098] Specifically, for a given sequence of N tokens, a subset of them is randomly masked and their values ​​are predicted, similar to the training method of BERT.

[0099] In a certain embodiment, the LLM includes a first prompt template and a second prompt template;

[0100] The first prompt template is: given a title, retrieve a detailed related title;

[0101] The second prompt template is: given a detailed title, retrieve a short related title.

[0102] The two prompt templates provided by the present invention can enhance the model's ability to understand the semantics of the title:

[0103] In one embodiment, in order to maintain the text function advantage of LLM, the present invention not only uses the self-made dataset, but also introduces the WikiText-103 dataset for training. This combination of multi-source data effectively avoids overfitting of the model in specific fields and ensures the versatility of LLM.

[0104] In one embodiment, in order to reduce the training cost, the present invention adopts the LoRA (Low-Rank Adaptation) method in the process of training LLM.

[0105] The embodiment of the present invention further provides a retrieval method based on a cross-modal feature fusion model, which is implemented using the pre-trained cross-modal feature fusion model described above and includes the following steps:

[0106] S1: Input the image to be retrieved and the question to be retrieved into the pre-trained cross-modal feature fusion model to obtain the text-image mixed embedding value;

[0107] Inputting the image to be retrieved into the target detection model to obtain N target images;

[0108] Input N target images into the pre-trained LLM; input the question to be retrieved into the pre-trained CLIP to obtain N target embedding values;

[0109] S2: Use the mixed embedding value of the text image and the N target embedding values ​​to perform knowledge retrieval to obtain relevant knowledge corresponding to the image to be retrieved and the question to be retrieved.

[0110] In one embodiment, the relevant knowledge is obtained based on the following formula:

[0111]

[0112]

[0113] D final =D IT ∪D OT ;

[0114] Among them, maxTop-M Indicates taking The first M maximum values ​​in ; Q represents the mixed embedding value of the text image; Indicates D j The transpose of D j Represents the embedding value of the jth text in the database document; max Top-K Indicates taking The first K largest values ​​in Q i represents the i-th target embedding value; D final Represents the relevant knowledge.

[0115] In one embodiment,

[0116] 1) Input the image to be retrieved and the question to be retrieved into the pre-trained cross-modal feature fusion model to obtain the text-image mixed embedding value e IT ;

[0117] Specifically: the image to be retrieved is input into the pre-trained CLIP, the question to be retrieved is input into the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the text-image mixed embedding value e IT ;

[0118] 2) Input the image to be retrieved into the target detection model to obtain 5 target images;

[0119] Specifically, according to the target probability values ​​output by the target detection model (YOLO World), the top five target images with the highest probability are selected as key target objects for further retrieval;

[0120] 3) Input the segmented 5 target images and the query to be retrieved into the pre-trained cross-modal feature fusion model to obtain the embedding values ​​e of the 5 target images. ot1, e ot2........ e ot5 .

[0121] Specifically: Assume that the five target images are target image ot1, target image ot2, target image ot3, target image ot4, and target image ot5;

[0122] Input the target image ot1 into the pre-trained CLIP, input the question to be retrieved into the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the embedding value e of the target image ot1 ot1 ;

[0123] Input the target image ot2 into the pre-trained CLIP, input the question to be retrieved into the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the embedding value e of the target image ot2 ot2 ;

[0124] Input the target image ot3 into the pre-trained CLIP, input the question to be retrieved into the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the embedding value e of the target image ot3 ot3 ;

[0125] Input the target image ot4 into the pre-trained CLIP, input the query to be retrieved into the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the embedding value e of the target image ot4 ot4 ;

[0126] Input the target image ot5 into the pre-trained CLIP, input the query to be retrieved into the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the embedding value e of the target image ot5 ot5 ;

[0127] Using text image mixed embedding value e IT Search, recall and e IT Related Top-5 Knowledge Items D IT ={d1,d2,d3,d4,d5}.

[0128] Using e ot1、 e ot2、 e ot3、 e ot4、 e ot4 Retrieve and recall separately with e ot1、 e ot2、 e ot3、 e ot4、 e ot5 Related Top-1 Knowledge D OT ={d OT1 ,d OT2 ,d OT3 ,d OT4 ,d OT5}.

[0129] D final =D IT ∪D OT ={d1,d2,d3,d4,d5,d OT1 ,d OT2 ,d OT3 ,d OT4 ,d OT5};

[0130] In one embodiment, the database document includes: W descriptive captions of each training image are input into a pre-trained LLM, and the LLM outputs A*W embedding values ​​of the descriptive captions.

[0131] In a certain embodiment, LLM is Llama-38B; CLIP is CLIP-ViT-L; and the target detection model is YOLOWorld.

[0132] To verify the advantages of the present invention, we selected BAAI / bge-visualized-m3 as the benchmark model for comparative testing, and statistically analyzed the accuracy of the knowledge recall task of the two models in the fire scene. The specific results are as follows:

[0133] Model R@10 R@20 Model of the present invention 79.46％ 90.03％ BAAI / bge-visualized-m3 52.71％ 68.25％

[0134] The experimental results show that the performance of the present invention in the knowledge recall task in the fire scene is significantly better than the baseline model. The Recall@10 (recall 10 relevant knowledge) and Recall@20 (recall 20 relevant knowledge) indicators have been improved by 26.75% and 21.78% respectively. This shows that the present invention can more accurately recall knowledge related to images and texts, and has demonstrated high accuracy and reliability in the task of identifying safety hazards in fire scenes.

[0135] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0136] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a cross-modal feature fusion model, characterized in that: Including the training method of LLM and the joint training method of CLIP, linear layer, and cross-modal attention layer: The LLM training method specifically includes: Get A training images and generate W descriptive captions for each training image; Select S descriptive captions from the same training image as positive sample pairs; where 2≤S≤W; Select S descriptive captions from different training images as negative sample pairs; Input the positive sample pairs and the negative sample pairs into the LLM, and use the first simCSE loss function to adjust the parameters of the LLM to obtain the pre-trained LLM; The joint training method of CLIP, linear layer, and cross-modal attention layer specifically includes: Acquire a joint training sample set; wherein each training sample in the joint training sample set includes a training image and one of S descriptive titles selected from the training image; Freeze the gradients of the pre-trained LLM; Input the training images in the training samples into CLIP, mask the descriptive titles in the training samples and input them into the pre-trained LLM, and use the second simCSE loss function to adjust the parameters of CLIP, linear layer, and cross-modal attention layer to obtain pre-trained CLIP, pre-trained linear layer, and pre-trained cross-modal attention layer; Among them, the cross-modal feature fusion model includes LLM, CLIP, linear layer and cross-modal attention layer; The output end of the LLM is connected to the input end of the cross-modal attention layer through the linear layer; The output end of the CLIP is connected to the input end of the cross-modal attention layer; The pre-trained LLM, the pre-trained CLIP, the pre-trained linear layer, and the pre-trained cross-modal attention layer constitute a pre-trained cross-modal feature fusion model.

2. The method for training a cross-modal feature fusion model according to claim 1, characterized in that: The expression of the first simCSE loss function is: Among them, L simCSE represents the first simCSE loss function; sim represents cosine similarity; h i represents the embedding value obtained by inputting one of the S descriptive captions selected from the i-th training image into the LLM; represents the embedding value obtained by inputting another descriptive caption from the S descriptive captions selected from the i-th training image into the LLM; represents the embedding value obtained by inputting the jth descriptive title into LLM, where the jth descriptive title represents a descriptive title in all positive and negative sample pairs, and the jth descriptive title does not include h i Corresponding descriptive title; τ represents the temperature coefficient.

3. The method for training a cross-modal feature fusion model according to claim 1, characterized in that: The expression of the second simCSE loss function is: Among them, L simCSE ' represents the second simCSE loss function; sim represents the cosine similarity; H i represents the embedding value obtained by inputting one of the S descriptive captions selected from the i-th training image into the cross-modal feature fusion model; represents the embedding value obtained by inputting another descriptive caption from the S descriptive captions selected from the i-th training image into the cross-modal feature fusion model; represents the embedding value obtained by inputting the jth descriptive title into the cross-modal feature fusion model, wherein the jth descriptive title represents a descriptive title in all positive and negative sample pairs, and the jth descriptive title does not include H i Corresponding descriptive title; τ represents the temperature coefficient.

4. The method for training a cross-modal feature fusion model according to claim 1, characterized in that: Replace the causal attention machine of LLM with a bidirectional attention mechanism.

5. The method for training a cross-modal feature fusion model according to claim 4, characterized in that: The bidirectional attention mechanism is trained using the MNTP method.

6. The method for training a cross-modal feature fusion model according to claim 1, characterized in that: The LLM includes a first prompt template and a second prompt template; The first prompt template is: given a title, retrieve a detailed related title; The second prompt template is: given a detailed title, retrieve a short related title.

7. A retrieval method based on a cross-modal feature fusion model, characterized in that: The method is implemented by using the pre-trained cross-modal feature fusion model described in any one of claims 1 to 6, comprising the following steps: S1: Input the image to be retrieved and the question to be retrieved into the pre-trained cross-modal feature fusion model to obtain the text-image mixed embedding value; Inputting the image to be retrieved into the target detection model to obtain N target images; Input N target images into the pre-trained LLM; input the question to be retrieved into the pre-trained CLIP to obtain N target embedding values; S2: Use the mixed embedding value of the text image and the N target embedding values ​​to perform knowledge retrieval to obtain relevant knowledge corresponding to the image to be retrieved and the question to be retrieved.

8. The retrieval method based on a cross-modal feature fusion model according to claim 7, characterized in that: The relevant knowledge is obtained based on the following formula: D final =D IT ∪D OT ; Among them, max Top-M Indicates taking The first M maximum values ​​in ; Q represents the mixed embedding value of the text image; Indicates D j The transpose of D j Represents the embedding value of the jth text in the database document; max Top-K Indicates taking The first K largest values ​​in Q i represents the i-th target embedding value; D final Represents the relevant knowledge.

9. The retrieval method based on a cross-modal feature fusion model according to claim 8, characterized in that: The database document includes: W descriptive captions of each training image are input into the pre-trained LLM, and the LLM outputs A*W embedding values ​​of the descriptive captions.

10. The retrieval method based on a cross-modal feature fusion model according to claim 7, characterized in that: LLM is Llama-38B; CLIP uses CLIP-ViT-L; the target detection model is YOLO World.

Images