A training method and a retrieval method of a cross-modal feature fusion model

Abstract

The application discloses a training method of a cross-modal feature fusion model and a retrieval method based on the cross-modal feature fusion model, wherein the cross-modal feature fusion model comprises an LLM, a CLIP, a linear layer and a cross-modal attention layer; an output end of the LLM is connected with an input end of the cross-modal attention layer through the linear layer; and an output end of the CLIP is connected with the input end of the cross-modal attention layer. The application significantly improves the accuracy and robustness of a knowledge recall task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and more particularly to a training method and a retrieval method of a cross-modal feature fusion model. BACKGROUND

[0002] In recent years, with the rapid development of deep learning and large-scale pre-training models, cross-modal retrieval technology has gradually become a research hotspot in the field of artificial intelligence, especially in the joint understanding and information retrieval of text and images. Cross-modal retrieval aims to realize information intercommunication and intelligent matching between modalities through feature representation and alignment of different modal data (such as text and image).

[0003] At present, the mainstream cross-modal retrieval method mostly adopts a text and image separate coding architecture, extracts features of text and image through independent encoders, and realizes alignment in a high-dimensional feature space. For example, the research paper related to the present application, “LLM2CLIP: POWERFUL LANGUAGE MODEL UNLOCK RICHER VISUAL REPRESENTATION”, proposes a cross-modal representation model based on text and image separate coding. The model uses the powerful characteristics of large language models (LLM) and visual encoders (CLIP) to extract semantic features of text and image respectively. However, this separate coding method still has the following shortcomings in certain application scenarios:

[0004] Insufficient semantic fusion capability: Since text and image features are extracted independently, they cannot be deeply fused in the coding stage. This lack of interaction limits the performance of the model in scenarios that require high semantic consistency or complex scene fine-grained feature understanding, resulting in reduced accuracy and reliability of the recall results.

[0005] Insufficient fine-grained feature expression: In the recall stage, existing methods are limited by the resolution of the visual coding model and cannot effectively retain and extract detailed information from images. For example, in scenarios that require attention to small targets or complex backgrounds, fine-grained features are often ignored in the coding process, affecting the effectiveness and accuracy of knowledge recall.

[0006] Therefore, how to provide a training method and a retrieval method of a cross-modal feature fusion model, which has better semantic fusion capability and can effectively retain and extract fine-grained features of images, is a problem that needs to be solved by those skilled in the art. SUMMARY

[0007] Therefore, the purpose of the present application is to provide a training method and a retrieval method of a cross-modal feature fusion model.

[0008] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0009] The training method of the cross-modal feature fusion model comprises the training method of the LLM and the joint training method of the CLIP, the linear layer and the cross-modal attention layer:

[0010] The training method of the LLM specifically comprises:

[0011] A plurality of training images are obtained, and W descriptive titles are generated for each training image;

[0012] S descriptive titles are selected from the same training image as a positive sample pair; wherein, 2≤S≤W;

[0013] S descriptive titles are selected from different training images as a negative sample pair;

[0014] The positive sample pair and the negative sample pair are input into the LLM, and the parameters of the LLM are adjusted by using a first simCSE loss function, to obtain a pre-trained LLM;

[0015] The joint training method of the CLIP, the linear layer and the cross-modal attention layer specifically comprises:

[0016] A joint training sample set is obtained; wherein, each training sample in the joint training sample set comprises a training image and one of S descriptive titles selected from the training image;

[0017] The gradient of the pre-trained LLM is frozen;

[0018] The training image in the training sample is input into the CLIP, and the descriptive title in the training sample is input into the pre-trained LLM after being masked, and the parameters of the CLIP, the linear layer and the cross-modal attention layer are adjusted by using a second simCSE loss function, to obtain a pre-trained CLIP, a pre-trained linear layer and a pre-trained cross-modal attention layer;

[0019] The cross-modal feature fusion model comprises the LLM, the CLIP, the linear layer and the cross-modal attention layer;

[0020] The output end of the LLM is connected to the input end of the linear layer and the cross-modal attention layer;

[0021] The output end of the CLIP is connected to the input end of the cross-modal attention layer;

[0022] The pre-trained LLM, the pre-trained CLIP, the pre-trained linear layer and the pre-trained cross-modal attention layer constitute a pre-trained cross-modal feature fusion model.

[0023] Preferably, the expression of the first simCSE loss function is:

[0024]

[0025]

[0026]

[0027] wherein, L simCSE represents the first simCSE loss function; sim represents cosine similarity; h i represents an embedding value obtained by inputting a certain one of S descriptive titles selected from the i-th training image into the LLM; represents an embedding value obtained by inputting another one of S descriptive titles selected from the i-th training image into the LLM; represents an embedding value obtained by inputting the j-th descriptive title into the LLM, wherein the j-th descriptive title represents a certain one of all positive and negative sample pairs, and the j-th descriptive title does not include h i the corresponding descriptive title; τ represents a temperature coefficient.

[0028] Preferably, the expression of the second simCSE loss function is:

[0029]

[0030]

[0031]

[0032] wherein, L simCSE ' represents the second simCSE loss function; sim represents cosine similarity; H i represents an embedding value obtained by inputting a certain one of S descriptive titles selected from the i-th training image into the cross-modal feature fusion model; represents an embedding value obtained by inputting another one of S descriptive titles selected from the i-th training image into the cross-modal feature fusion model; represents an embedding value obtained by inputting the j-th descriptive title into the cross-modal feature fusion model, wherein the j-th descriptive title represents a certain one of all positive and negative sample pairs, and the j-th descriptive title does not include H i the corresponding descriptive title; τ represents a temperature coefficient.

[0033] Preferably, the causal attention mechanism of the LLM is replaced by a bidirectional attention mechanism.

[0034] Preferably, the bidirectional attention mechanism is trained using the MNTP method.

[0035] Preferably, the LLM includes a first prompt template and a second prompt template;

[0036] The first prompt template is: Given a title, retrieve a detailed related title;

[0037] The second prompt template is: Given a detailed title, retrieve a short, relevant title.

[0038] A retrieval method based on a cross-modal feature fusion model, implemented using the aforementioned pre-trained cross-modal feature fusion model, includes the following steps:

[0039] S1: Input the image to be retrieved and the question to be retrieved into a pre-trained cross-modal feature fusion model to obtain the text-image hybrid embedding value;

[0040] The image to be retrieved is input into the target detection model to obtain N target images;

[0041] Input N target images into a pre-trained LLM; input the retrieval question into a pre-trained CLIP to obtain N target embedding values;

[0042] S2: Use the text image mixed embedding value and the N target embedding values ​​to perform knowledge retrieval, and obtain relevant knowledge corresponding to the image to be retrieved and the question to be retrieved.

[0043] Preferably, the relevant knowledge is obtained based on the following formula:

[0044]

[0045]

[0046] D final =D IT ∪D OT ;

[0047] Where, max Top-M Indicates taking The first M maximum values ​​in the text image; Q represents the text image blending embedding value; D represents j The transpose of D j This represents the embedding value of the j-th text in the database document; max Top-K Indicates taking The top K maximum values ​​in Q;i represents the i-th target embedding value; D final represents the relevant knowledge.

[0048] Preferably, the database document comprises: inputting W descriptive titles of each training image into a pre-trained LLM, and the LLM outputs embedding values of A*W descriptive titles.

[0049] Preferably, the LLM is Llama-38B; the CLIP is CLIP-ViT-L; and the target detection model is YOLO World.

[0050] According to the technical solutions described above, compared with the prior art, the present disclosure provides a training method of a cross-modal feature fusion model and a retrieval method based on the cross-modal feature fusion model, which can obtain the following beneficial technical effects:

[0051] 1) The present application greatly improves the feature interaction depth between text and image by introducing a cross-modal attention mechanism (Cross-Attention Layer). Compared with traditional separate encoding methods, the present application can more accurately capture the complex semantic association of text and image, and construct a more integrated cross-modal feature space, thereby significantly improving the accuracy and robustness of the knowledge recall task.

[0052] 2) The present application combines target detection technology and proposes a fine-grained retrieval strategy based on target detection. By detecting and blocking the key objects in the image, the detailed features contained in the image are fully retained and utilized. This strategy makes the model perform better in handling complex scenes and hidden targets, especially in tasks that require detailed differentiation.

[0053] 3) The present application adopts a dual-path retrieval strategy, which combines the query results of the image based on the original image and target detection, and integrates the recall results through a knowledge fusion mechanism, improving the overall accuracy of the retrieval task. The dual-path design not only makes up for the shortcomings of single-path retrieval, but also provides higher reliability and efficiency for knowledge recall tasks in complex scenarios.

[0054] 4) In the actual test of the fire scene, the present application significantly outperforms the benchmark model, showing extremely high scene adaptation ability

[0055] 5) The present application is not only suitable for fire scenes, but also has strong versatility and can be widely applied to intelligent monitoring, risk identification, multi-modal retrieval and other scenarios. Its innovative design provides an efficient and reliable technical solution for complex tasks, and has a significant advantage in multi-modal tasks that require high precision and high robustness.

[0056] 6) The present application significantly reduces the computational cost of task execution and improves overall processing efficiency through an optimized cross-modal interaction mechanism, target detection technology, and efficient knowledge recall method, providing important support for the intelligent development of multi-modal tasks. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0058] Figure 1 The structure diagram of the cross-modal feature fusion model provided by the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0060] The embodiments of the present application disclose a training method of a cross-modal feature fusion model, including a training method of LLM (large language model) and a joint training method of CLIP (visual encoder), linear layer and cross-modal attention layer:

[0061] The training method of the LLM specifically includes:

[0062] Obtain A training images, and generate W descriptive titles for each training image;

[0063] Select S descriptive titles from the same training image as positive sample pairs; wherein, 2≤S≤W;

[0064] Select S descriptive titles from different training images as negative sample pairs;

[0065] In an embodiment:

[0066] Apply ShareGPT4V to each training image to generate five descriptive titles, which depict the semantic information of the image from different angles;

[0067] Select the top-2 (top 2) descriptive titles from the same training image as positive sample pairs;

[0068] Two different training images are randomly selected, and Top-1 descriptive titles are selected from the two different training images respectively as negative sample pairs;

[0069] The present application constructs a contrastive learning task through positive sample pairs and negative sample pairs, and designs a title-to-title retrieval experiment in the model evaluation stage. Through contrastive learning, the titles of the same image are more closely combined in the semantic space, and the titles of different images are separated as much as possible, thereby optimizing the retrieval results.

[0070] The positive sample pairs and the negative sample pairs are input into the LLM, and the parameters of the LLM are adjusted by using the first simCSE loss function to obtain a pre-trained LLM;

[0071] In an embodiment, the expression of the first simCSE loss function is:

[0072]

[0073]

[0074]

[0075] Wherein, L simCSE represents the first simCSE loss function; sim represents the cosine similarity; h i represents the embedding value obtained by inputting a certain one of the S descriptive titles selected from the i-th training image into the LLM; represents the embedding value obtained by inputting another one of the S descriptive titles selected from the i-th training image into the LLM; represents the embedding value obtained by inputting the j-th descriptive title into the LLM, wherein the j-th descriptive title represents a certain one of all positive and negative sample pairs, and the j-th descriptive title does not include h i corresponding descriptive title; τ represents the temperature coefficient.

[0076] The joint training method of the CLIP, the linear layer and the cross-modal attention layer specifically includes:

[0077] Obtain a joint training sample set; each training sample in the joint training sample set includes a training image and a certain one of S descriptive titles selected from the training image;

[0078] In an embodiment, the certain one of the S descriptive titles selected from the training image is a Top-1 descriptive title generated for the training image.

[0079] freezing the gradient of the pre-trained LLM;

[0080] In order to maintain the inherent ability of the large language model, the gradient of the LLM is frozen during the training stage, so that the core characteristics and pre-training semantic knowledge are completely retained.

[0081] The training image in the training sample is input into the CLIP, and the descriptive title in the training sample is input into the pre-trained LLM after being masked (specifically, the nouns in the latter descriptive title are masked), and the parameters of the CLIP, the linear layer and the cross-modal attention layer are adjusted using the second simCSE loss function, to obtain the pre-trained CLIP, the pre-trained linear layer and the pre-trained cross-modal attention layer.

[0082] During the training process, the CLIP, the linear layer and the cross-modal attention layer are jointly optimized to ensure the collaborative working efficiency of each module in the cross-modal feature space. By optimizing the second simCSE loss function, the model can improve the alignment effect of images and texts.

[0083] In an embodiment, the expression of the second simCSE loss function is:

[0084]

[0085]

[0086]

[0087] wherein, L simCSE represents the second simCSE loss function; sim represents the cosine similarity; H i represents the embedding value obtained by inputting a certain one of the S descriptive titles selected from the i-th training image into the cross-modal feature fusion model; represents the embedding value obtained by inputting another one of the S descriptive titles selected from the i-th training image into the cross-modal feature fusion model; represents the embedding value obtained by inputting the j-th descriptive title into the cross-modal feature fusion model, wherein the j-th descriptive title represents a certain one of all positive and negative sample pairs, and the j-th descriptive title does not include H i the corresponding descriptive title; τ represents the temperature coefficient.

[0088] The present application adopts a supervised contrast learning strategy (Supervised SimCSE Loss) to fine-tune the LLM, further improving the alignment ability of the model.

[0089] wherein the cross-modal feature fusion model (as shown in Figure 1 includes the LLM, the CLIP, a linear layer, and a cross-modal attention layer;

[0090] An output end of the LLM is connected to an input end of the cross-modal attention layer through the linear layer;

[0091] An output end of the CLIP is connected to an input end of the cross-modal attention layer;

[0092] It can be understood that, in order to compensate for the insufficient cross-modal feature alignment capability caused by freezing the LLM, the present application introduces a new linear layer as an adapter at the back end of the LLM. The learnable parameters of the linear layer can effectively improve the alignment capability between the LLM and the CLIP through optimization.

[0093] In order to further improve the semantic interaction capability of text and image features, the present application introduces a cross-modal attention (Cross-Attention) mechanism after the linear layer and the visual encoder. This mechanism captures the deep semantic relationship between text and image, realizes the hybrid coding of the two, fully captures the corresponding relationship of image and text in the semantic space, and thus improves the understanding capability of complex scenes and fine-grained features.

[0094] The pre-trained LLM, the pre-trained CLIP, the pre-trained linear layer, and the pre-trained cross-modal attention layer constitute a pre-trained cross-modal feature fusion model.

[0095] In an embodiment, the causal attention mechanism of the LLM is replaced by a bidirectional attention mechanism.

[0096] In order to enhance the text encoding capability, the present application converts the causal attention mechanism (Causal Attention) of the LLM (in an embodiment, specifically Llama-38B) into a bidirectional attention mechanism (Bidirectional Attention). This modification enables the model not only to encode input sequences using context relationships, but also to improve its robustness in contrast learning tasks.

[0097] In an embodiment, the MNTP method is used to train the bidirectional attention mechanism.

[0098] Specifically, for a given sequence of N tokens, a subset of them is randomly masked, and their values are predicted, similar to the training method of BERT.

[0099] In an embodiment, the LLM includes a first prompt template and a second prompt template;

[0100] The first prompt template is: given a title, retrieve a detailed related title;

[0101] The second prompt template is: given a detailed title, retrieve a short related title.

[0102] The two prompt templates provided by the present application can enhance the understanding ability of the model to the semantic of title:

[0103] In an embodiment, in order to maintain the text function advantage of LLM, the present application introduces WikiText-103 dataset for training in addition to using self-made dataset. The combination of multiple source data effectively avoids overfitting of the model in a specific field, and ensures the universality of LLM.

[0104] In an embodiment, in order to reduce the training cost, the present application adopts LoRA (Low-Rank Adaptation) method in the process of training LLM.

[0105] The embodiment of the present application also provides a retrieval method based on a cross-modal feature fusion model, which is realized by using the pre-trained cross-modal feature fusion model described above, and includes the following steps:

[0106] S1: input the image to be retrieved and the question to be retrieved into the pre-trained cross-modal feature fusion model to obtain a text-image hybrid embedding value;

[0107] input the image to be retrieved into a target detection model to obtain N target images;

[0108] input the N target images into the pre-trained LLM, and input the question to be retrieved into the pre-trained CLIP to obtain N target embedding values;

[0109] S2: use the text-image hybrid embedding value and the N target embedding values for knowledge retrieval to obtain related knowledge corresponding to the image to be retrieved and the question to be retrieved.

[0110] In an embodiment, the related knowledge is obtained based on the following formula:

[0111]

[0112]

[0113] D final =D IT ∪D OT ;

[0114] wherein, maxTop-M Indicates taking The first M maximum values ​​in the text image; Q represents the text image blending embedding value; D represents j The transpose of D j This represents the embedding value of the j-th text in the database document; max Top-K Indicates taking The top K maximum values ​​in Q; i D represents the embedding value of the i-th target; final This refers to the relevant knowledge mentioned above.

[0115] In one embodiment,:

[0116] 1) Input the image to be retrieved and the question to be retrieved into a pre-trained cross-modal feature fusion model to obtain the text-image hybrid embedding value e. IT ;

[0117] Specifically: the image to be retrieved is input into a pre-trained CLIP, the retrieval question is input into a pre-trained LLM, and the pre-trained cross-modal attention layer outputs the text-image hybrid embedding value e. IT ;

[0118] 2) Input the image to be retrieved into the object detection model to obtain 5 object images;

[0119] Specifically, based on the target probability values ​​output by the object detection model (YOLO World), the top 5 target images with the highest probabilities are selected as key target objects for further retrieval;

[0120] 3) Input the five segmented target images and the retrieval question into a pre-trained cross-modal feature fusion model to obtain the embedding values ​​e of the five target images. ot1, e ot2........ e ot5 .

[0121] Specifically: Assume the five target images are target image ot1, target image ot2, target image ot3, target image ot4, and target image ot5;

[0122] The target image ot1 is input into the pre-trained CLIP, the retrieval question is input into the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the embedding value e of the target image ot1. ot1 ;

[0123] The target image ot2 is input to the pre-trained CLIP, the to-be-retrieved question is input to the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the embedding value e of the target image ot2 ot2 ;

[0124] The target image ot3 is input to the pre-trained CLIP, the to-be-retrieved question is input to the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the embedding value e of the target image ot3 ot3 ;

[0125] The target image ot4 is input to the pre-trained CLIP, the to-be-retrieved question is input to the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the embedding value e of the target image ot4 ot4 ;

[0126] The target image ot5 is input to the pre-trained CLIP, the to-be-retrieved question is input to the pre-trained LLM, and the pre-trained cross-modal attention layer outputs the embedding value e of the target image ot5 ot5 ;

[0127] The text-image hybrid embedding value e is utilized IT Retrieval is performed in the database document, and Top-5 knowledge items D related to e IT are recalled IT ={d1,d2,d3,d4,d5}.

[0128] The e ot1、 is utilized ot2、 The e ot3、 is utilized ot4、 The e ot4 is respectively utilized, and Top-1 knowledge D respectively related to e ot1、 e ot2、 e ot3、 e ot4、 e ot5 is recalled OT ={d OT1 ,d OT2 ,d OT3 ,d OT4 ,d OT5}。

[0129] D final =D IT ∪D OT ={d1,d2,d3,d4,d5,d OT1 ,d OT2 ,d OT3 ,d OT4 ,d OT5};

[0130] In a certain embodiment, the database document comprises: inputting the W descriptive titles of each training image into a pre-trained LLM, and the LLM outputs embedding values of A*W descriptive titles.

[0131] In a certain embodiment, the LLM is Llama-38B; the CLIP is CLIP-ViT-L; and the target detection model is YOLOWorld.

[0132] To verify the advantages of the present application, we selected BAAI / bge-visualized-m3 as the benchmark model for comparison test, and counted the accuracy of the knowledge recall task of the two models in the fire scene, and the specific results are as follows:

[0133] Model R@10 R@20 Model of the invention 79.46％ 90.03％ BAAI / bge-visualized-m3 52.71％ 68.25％

[0134] The experimental results show that the knowledge recall task of the present application in the fire scene is significantly better than the benchmark model. In the Recall@10 (recalling 10 related knowledge) and Recall@20 (recalling 20 related knowledge) indicators, it is increased by 26.75% and 21.78% respectively. This shows that the present application can more accurately recall the knowledge related to images and texts, and has high accuracy and reliability in the safety hazard identification task in the fire scene.

[0135] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0136] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method of a cross-modal feature fusion model, characterized in that, The training method of the LLM and the joint training method of the CLIP, the linear layer and the cross-modal attention layer are included. The training method of the LLM specifically includes: A plurality of training images are obtained, and W descriptive titles are generated for each training image; S descriptive titles are selected from the same training image as positive sample pairs; wherein, 2≤S≤W; S descriptive titles are selected from different training images as negative sample pairs; The positive sample pairs and the negative sample pairs are input into the LLM, and the parameters of the LLM are adjusted by using a first simCSE loss function to obtain a pre-trained LLM; The joint training method of the CLIP, the linear layer and the cross-modal attention layer specifically includes: A joint training sample set is obtained; wherein each training sample in the joint training sample set includes a training image and one of S descriptive titles selected from the training image; The gradient of the pre-trained LLM is frozen; The training image in the training sample is input into the CLIP, and the descriptive title in the training sample is input into the pre-trained LLM after being masked, and the parameters of the CLIP, the linear layer and the cross-modal attention layer are adjusted by using a second simCSE loss function to obtain a pre-trained CLIP, a pre-trained linear layer and a pre-trained cross-modal attention layer; The cross-modal feature fusion model includes the LLM, the CLIP, the linear layer and the cross-modal attention layer; The output end of the LLM is connected to the input end of the linear layer and the cross-modal attention layer; The output end of the CLIP is connected to the input end of the cross-modal attention layer; The pre-trained LLM, the pre-trained CLIP, the pre-trained linear layer and the pre-trained cross-modal attention layer constitute a pre-trained cross-modal feature fusion model. 2.The method of claim 1, wherein, The expression of the first simCSE loss function is: wherein, L simCSE represents the first simCSE loss function; sim represents cosine similarity; h i represents an embedding value obtained by inputting a certain one of S descriptive titles selected from the i-th training image into the LLM; represents an embedding value obtained by inputting another one of S descriptive titles selected from the i-th training image into the LLM; represents an embedding value obtained by inputting the j-th descriptive title into the LLM, wherein the j-th descriptive title represents a certain one of all positive and negative sample pairs, and the j-th descriptive title does not include h i the corresponding descriptive title; τ represents a temperature coefficient. 3.The method of claim 1, wherein, The expression of the second simCSE loss function is: wherein, L simCSE simCSEloss represents the second simCSE loss function; sim represents cosine similarity; H i represents an embedding value obtained by inputting one of S descriptive titles selected from the i-th training image into the cross-modal feature fusion model; represents an embedding value obtained by inputting another of S descriptive titles selected from the i-th training image into the cross-modal feature fusion model; represents an embedding value obtained by inputting the j-th descriptive title into the cross-modal feature fusion model, wherein the j-th descriptive title represents one of all positive and negative sample pairs, and the j-th descriptive title does not include H i the corresponding descriptive title; τ represents a temperature coefficient. 4.The method of claim 1, wherein, The causal attention mechanism of the LLM is replaced by a bidirectional attention mechanism. 5.The method of Claim 4, wherein, The MNTP method is used to train the bidirectional attention mechanism. 6.The method of Claim 1, wherein, The LLM includes a first prompt template and a second prompt template; The first prompt template is: given a title, retrieve a detailed related title; The second prompt template is: given a detailed title, retrieve a short related title.

7. A retrieval method based on a cross-modal feature fusion model, characterized in that, The pre-trained cross-modal feature fusion model of any one of claims 1-6 is implemented, including the following steps: S1: input the to-be-retrieved image and the to-be-retrieved question into the pre-trained cross-modal feature fusion model to obtain a text-image hybrid embedding value; Input the to-be-retrieved image into a target detection model to obtain N target images; Input the N target images into the pre-trained LLM, and input the to-be-retrieved question into the pre-trained CLIP to obtain N target embedding values; S2: use the text-image hybrid embedding value and the N target embedding values for knowledge retrieval to obtain related knowledge corresponding to the to-be-retrieved image and the to-be-retrieved question. 8.The retrieval method based on the cross-modal feature fusion model according to claim 7, characterized in that, The related knowledge is obtained based on the following formula: D final = D IT U D OT ; where max Top-M denotes taking the first M largest values in ; Q denotes the text image mixed embedding value; denotes the transpose of D j , D j denotes the embedding value of the jth text in the database document; max Top-K denotes taking the first K largest values in ; Q i denotes the ith target embedding value; D final denotes the relevant knowledge. 9.The retrieval method based on the cross-modal feature fusion model according to claim 8, characterized in that, The database document includes: inputting W descriptive titles of each training image into a pre-trained LLM, and the LLM outputs embedding values of A*W descriptive titles. 10.The retrieval method based on the cross-modal feature fusion model according to claim 7, characterized in that, The LLM is Llama-38B; the CLIP is CLIP-ViT-L; and the target detection model is YOLO World.

Images