A domain incremental graph-text retrieval method based on banzhaf coalition evaluation distillation

Abstract

The application discloses a domain incremental graph-text retrieval method based on Banzhaf coalition evaluation distillation, which models the graph-text retrieval as a multi-element cooperative game process through Banzhaf coalition evaluation, calculates the contribution of image tokens and text tokens to the retrieval accuracy, so as to realize more efficient cross-modal comparison and improve the model accuracy. Through contrast learning distillation and Banzhaf coalition evaluation distillation, structured semantic information is retained, cross-task structural knowledge is constrained from sample and token levels, and catastrophic forgetting is reduced. The application also proposes that token-level dynamic semantic prompts are added to adaptively compensate for the semantic information difference between modes, and the knowledge of the task is retained by retaining domain-independent semantic prompts. The main technical problems solved by the method include the loss of structured semantic information of different modal characteristics and the task knowledge deviation during cross-modal knowledge transfer.

Description

TECHNICAL FIELD

[0001] The application is suitable for domain incremental image-text retrieval in the field of computer vision, and particularly relates to a domain incremental image-text retrieval method based on Banzhaf coalition evaluation distillation. BACKGROUND

[0002] In the past few decades, multimedia data has been growing explosively, greatly expanding the human vision. Among them, visual and textual content are two main forms on the Internet. In practical applications such as intelligent search engines, multimedia digital libraries, there is a huge demand for retrieval of multimedia resources such as images and texts. Therefore, cross-modal image-text retrieval has become an important research topic.

[0003] In many practical scenarios, cross-modal data arrives sequentially rather than all at once, resulting in new data and even new classes in different domains. For example, doctors diagnose patients based on daily updated medical records and images; they are required to diagnose new and old diseases. The Internet generates images and texts every day, and network monitoring departments need to timely discover new or previously negative content. In this case, training a separate model for each new domain will result in unacceptable storage consumption. On the other hand, direct retraining or fine-tuning on data in the new domain may reduce performance in the new and old domains. Therefore, domain incremental image-text retrieval is very important for practical applications with limited storage and computing power.

[0004] The domain incremental image-text retrieval task wants to sequentially learn a model on a series of datasets from different task domains. At each sub-task stage, the model must be able to retrieve all seen image-texts. Formally, given a dataset sequence D 1 ,…,D K , where denote the jth image and its corresponding text description in the kth (k = 1…K) task, respectively. The purpose of domain incremental image-text retrieval is to build a model that sequentially trains all tasks, and the model can retrieve the test set containing all seen domains without access to (or only a small part of) the previous sub-task data. In recent years, with the rapid development of deep learning, some domain incremental image-text retrieval methods have been proposed. Some representative methods are listed as follows.

[0005] VLKD effectively preserves the semantic knowledge of the old domain and transfers it to the new domain by keeping the relative distance of the feature centers of the new domain data and the old domain data in the public space unchanged. Meanwhile, VLKD constructs a hierarchical recursive network that can adaptively expand the network to utilize the knowledge at the semantic and attention levels when learning new domain data, preventing catastrophic forgetting. KDR decomposes the multi-modal features into general and task-specific features and stores a task-specific knowledge extractor for each task to support long sequence continual learning. In addition, KDR uses knowledge distillation to constantly replay the previous cross-modal general knowledge during the learning process, simulating the human knowledge replay behavior, thereby maintaining the memory of old knowledge when learning new knowledge. 2 MR resists catastrophic forgetting by constraining the new model to be consistent with the old model in terms of feature and sample relationship. 2 MR also designs a relationship-aware replay strategy to replay the most representative image-text pairs in the previously constructed feature space, further constraining the consistency of cross-modal relationships. CLCMR designs multi-layer domain-selective attention to automatically select parameters specific to previous tasks, and then uses weight regularization methods to limit the update of corresponding parameters. Strong constraints are imposed on task-specific key weight parameters to reduce the excessive drift of sample features and sample relationships.

[0006] The above methods have achieved success in the domain incremental image-text retrieval task, but still have the following shortcomings: (a) the structured semantic information between image data and text data is missing. Existing domain incremental image-text retrieval methods are performed at a coarse granularity, i.e., using global vectors of images and texts for retrieval. This approach can lead to the loss of semantic information, causing semantic confusion and other problems during the domain incremental retrieval process. (b) The inter-task knowledge bias is ignored. Traditional knowledge transfer methods often retain a large amount of intra-task knowledge while ignoring most inter-task knowledge. The lack of inter-task knowledge can lead to a decrease in model stability and exacerbate the phenomenon of catastrophic forgetting.

[0007] With the successful application of cooperative game in cross-modal retrieval tasks, the present application proposes a domain incremental image-text retrieval network based on Banzhaf coalition evaluation distillation to implement the domain incremental image-text retrieval task. The present application captures the fine-grained semantics between modalities through Banzhaf coalition evaluation, achieving accurate cross-modal comparison. At the same time, the Banzhaf coalition evaluation distillation loss is used to preserve the structured semantic information between modalities, achieving the transfer of fine-grained knowledge between tasks and effectively reducing catastrophic forgetting. SUMMARY

[0008] The present application proposes a Banzhaf coalition evaluation distillation network (Coalition Banzhaf Distillation Network, CBDNet) to complete the domain incremental image-text retrieval task. The method models the image-text retrieval as a multi-element cooperative game process through Banzhaf coalition evaluation, calculates the contribution of image tokens and text tokens to retrieval accuracy, realizes more efficient cross-modal comparison, and improves model accuracy. At the same time, the structured semantic information is retained by using contrastive learning distillation and Banzhaf coalition evaluation distillation, and the cross-task structural knowledge is constrained from sample and token levels respectively to alleviate catastrophic forgetting. In addition, the present application also proposes to add dynamic semantic prompts at the token level to adaptively compensate for the semantic information difference between modalities, and to retain the knowledge of the task by retaining domain-independent semantic prompts. The present application mainly solves the technical problems of structured semantic information loss of different modalities and task knowledge deviation during cross-modal knowledge transfer.

[0009] The domain incremental image-text retrieval method based on Banzhaf coalition evaluation distillation proposed by the present application can effectively avoid the shortcomings of traditional domain incremental image-text retrieval methods. By adding dynamic semantic prompts to image tokens and text tokens, the semantic information difference between the two modalities is adaptively compensated, and the retrieval accuracy is improved. By modeling the image-text retrieval as a multi-element cooperative game process through Banzhaf coalition evaluation, the structured semantic information between modalities is effectively extracted, and semantic confusion in the incremental process is avoided. Through contrastive learning and distillation and Banzhaf coalition evaluation distillation, multi-granularity knowledge transfer between tasks is realized, and the catastrophic forgetting phenomenon is alleviated.

[0010] The present application is implemented by the following technical solutions:

[0011] The model proposed by the present application takes RGB images and image corresponding text descriptions as input, first divides the data set by class increment, then inputs the divided image and text data into the Banzhaf coalition evaluation distillation image-text retrieval network, trains the established model parameters, then increases the image and text data of the new domain to train the entire model again, repeats this step until there is no new domain to increase, finally tests the trained model and outputs the domain incremental image-text retrieval result. The domain incremental image-text retrieval network model based on Banzhaf distillation is established.

[0012] Training data set input and domain incremental division:

[0013] In order to simulate the real scene of continuous increase of new class images on the network, the cross-modal image-text data set will be randomly divided into different sub-data sets (denoted as different sub-tasks) according to different domains to represent newly added image-text data at different times.

[0014] Design image and text encoder:

[0015] To deal with the difference between modalities, a double-branch encoder is designed to extract image and text features respectively. For each image and its corresponding text (I, T) of the kth task, the input image is first flattened Then it is input into the embedding layer to get the image embedding feature I, and then I is input into L self-attention layers linear layers to get the image feature where L I is the number of image patches, p is the size of image patches, d I is the dimension of image embedding feature, and d is the dimension of image feature output by the encoder.

[0016]

[0017] where ° represents the connection between different operations, is the learnable class token of the image, and each transformer layer Trans (l) (the superscript l represents the layer index) adopts the traditional multi-head self-attention structure, that is:

[0018]

[0019] where X is the input of Trans (l) , X' is the result after multi-head self-attention, X" is the output of Trans (l) , LN is layer normalization, MSA is multi-head self-attention, and MLP is multi-layer perceptron. Similar to the image modality, the text description T is encoded by a Byte-Pair encoder and then input into the BERT model to get the final text feature where L T represents the number of text words, and d represents the dimension of text feature. The specific formula is as follows:

[0020] T' = BERT ([t cls ; BPE(T)]) (4)

[0021] where is the learnable class token of the text, T is the input text description, and T' is the output of the text encoder.

[0022] Design dynamic semantic prompt module:

[0023] ​Due to the inter-modal difference of image-text data and the information contained in image-text is not completely consistent, there is often a large semantic information difference between the image-text features extracted by the encoder, which leads to the fact that the data between modalities cannot be accurately matched. Therefore, a dynamic semantic prompt module is proposed. Here, prompt refers to a learnable token. By introducing multiple learnable tokens between image and text modalities, the semantic difference is made up. These tokens can dynamically aggregate image and text information, and then achieve more accurate fine-grained matching. Specifically, learnable image prompts and text prompts are added to the image embedding layer and Byte-Pair encoder layer, respectively. The specific formula is as follows:

[0024]

[0025] where L'I: = LI+N+1, L'T: = LT+N+1 represent the image and text embedding feature dimensions after adding prompts and class tokens, respectively. N is the number of image prompts and text prompts, I prompt , T prompt represent the image and text embedding features after adding prompts, respectively. Then I prompt , T prompt are input into the L-layer self-attention module SA and the BERT model, respectively, to obtain the final image and text features I", T":

[0026]

[0027] In order to alleviate catastrophic forgetting, a set of dynamic semantic prompts is reserved for each task, and it is hoped that the correct prompts for different tasks will be selected by samples in the test stage. Therefore, the prompt-key matching strategy is used to learn task-related keys. Specifically, let denote the image (text) prompt of the k-th task, denote the key corresponding to the prompt, and the similarity between the image (text) category vector I' cls (I) (T' cls (T)) without adding prompts and is minimized to associate the prompt and the key:

[0028]

[0029] where denotes a training batch, and <·,·> denotes the cosine similarity calculation.

[0030] After all K tasks are trained, the prompt-key pair is obtained. For each test image Itest The corresponding prompt is selected by finding the key that is most similar to the image features without adding the prompt. Then the prompt image embedding is obtained:

[0031]

[0032] where I test is the test image, I(I test ) is the output of I test after the embedding layer, I prompt (I test ) represents the image embedding features after adding the prompt, I”(I test ) is the test image feature output by the encoder, k * represents the selected key, is the prompt corresponding to k * . T test The process of prompt embedding is similar to that of images.

[0033] Design of contrast learning and distillation modules

[0034] In the process of incremental image-text retrieval, two tasks need to be completed: the first is the alignment of image and text modalities, and the second is to retain old knowledge to alleviate catastrophic forgetting. For these two tasks, contrast learning and distillation modules are designed. Specifically, for the final obtained image and text features, the similarity of (I, T) is evaluated through image-text similarity at the token level:

[0035]

[0036] where u indexes the number of image tokens, and v indexes the number of text tokens. A (u,:) represents the u-th row of matrix A, and A (v,:) represents the v-th row of matrix A. I”(I), T”(T) represent the image and text features, respectively.

[0037] The image-text contrast learning loss is as follows:

[0038]

[0039]

[0040] For the second task, since cross-modal knowledge reflects the semantic similarity between images and texts, this knowledge can be preserved by maintaining the invariance of the image-text similarity matrix of different tasks. Specifically, the consistency of the similarity matrix S output by task k and task k-1 is constrained: k-1 where​ denotes the total number of buffer samples, and the specific formula is as follows:

[0041]

[0042] where Sim k-1 i , Sim j k i j represent the token-level image-text similarity obtained by the previous model and the current model, respectively. Then, an image-text contrastive distillation loss is designed:

[0043]

[0044] where softmax(·) represents the softmax operation row by row, and KL represents the Kullback-Leibler divergence.

[0045] Designing Banzhaf coalition evaluation distillation module

[0046] The current incremental image-text retrieval method uses global features for matching, which leads to the lack of structured semantic information and produces serious inter-task knowledge bias. To solve this problem, a joint Banzhaf distillation module is proposed. Joint Banzhaf is a concept in cooperative game theory, specifically, cooperative game theory contains a set of players and a characteristic function φ. The characteristic function maps each group of players to a real number, which represents the benefit obtained by all players in completing the task. Joint Banzhaf measures the additional benefit of the target coalition and the cost of the players losing other people's coalition. Given a coalition The Banzhaf coalition evaluation for the two players {u, v} is:

[0047]

[0048] where denotes all participants except players {u, v}, and S is a subset of . Φ(u, v) reflects the tendency of players {u, v} to interact with any subset S. In the image-text retrieval task, the higher the value of Φ(u, v), the more the image u-th patch and the text description v-th word form a coalition. For the image-text retrieval task The contribution of image token and text word token coalition to the retrieval task is evaluated through the Banzhaf coalition evaluation value. The joint Banzhaf network is used to evaluate the token similarity matrix of image-text pairs. ψ, ψ * ​​​​Let represent the learnable and pre-trained joint Banzhaf networks, respectively. For each image-text pair (I,T), define random variables. Let u be the Banzhaf alliance evaluation value of the u-th token of the input image I relative to all tokens of the input text T (image-text Banzhaf alliance evaluation value):

[0049]

[0050] Where p u,v for The value of the v-th element, Φ(I”) (u,:) ,T” (v,:) ) indicates I” (u,:) and T” (v,:) Evaluation value of the Banzhaf alliance between them. Learning random variables. and pre-trained network ψ * The obtained joint Banzhaf value The comparison Represented as:

[0051]

[0052] Where f * (I” (u,:) ,T” (v,:) ) represents the pre-trained network ψ * The obtained I” (u,:) and T” (v,:) The evaluation value of the Banzhaf alliance between them. Constrained by KL divergence. and The distribution of [the data]. Similarly, for each image-text pair (I,T), a random variable is defined. Let T be the Banzhaf federation evaluation of the v-th token relative to all tokens in I (text-image Banzhaf federation evaluation value):

[0053]

[0054] We define the Banzhaf alliance evaluation loss to characterize the approximation of the distribution of fine-grained image-text similarity based on Banzhaf alliance evaluation compared to that based on pre-trained networks:

[0055]

[0056] To preserve the structured semantic information and fine-grained cross-modal relationships learned from the old domains, a joint Banzhaf-based knowledge distillation is proposed. Specifically, the image-text and text-image Banzhaf coalition evaluation values obtained on the kth and (k-1)th task models are calculated using the data in the buffer, respectively:

[0057]

[0058] wherein denote the image-text Banzhaf coalition evaluation values obtained on the (k-1)th and kth task models, respectively, for the data in the buffer, denote the values of the vth element of and , respectively. denote the text-image Banzhaf coalition evaluation values obtained on the (k-1)th and kth task models, respectively, for the data in the buffer, denote the values of the uth element of and , respectively.

[0059] Based on formula (18), the Banzhaf coalition evaluation distillation loss is designed to depict the distribution approximation of the fine-grained cross-modal similarity between the new and old tasks based on the Banzhaf coalition evaluation:

[0060] BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 FIG. 1 is a general framework diagram of a Banzhaf coalition evaluation distillation network (CBDNet).

[0062] Figure 2 FIG. 2 shows the visualization of the text attention (a) and token similarity (b), (c) on the image-text pair “plane on runway with buildings in the background”. Each column of (b) and (c) corresponds to the average similarity on the region of the test image (a).

[0063] Figure 3 FIG. 3 shows the visualization of the text attention (a) and token similarity (b), (c) on the image-text pair “Two trains inside a station”. Each column of (b) and (c) corresponds to the average similarity on the region of the test image (a). DETAILED DESCRIPTION

[0064] The present application will be described in detail below with reference to the accompanying drawings and examples.

[0065] Figure 1 The overall structure of the Banzhaf coalition evaluation distillation network is given.

[0066] The Pascal Sentence and Wikipedia data sets are used for experiments, and three experimental settings are proposed based on the two data sets, and the specific experimental settings are shown in Table 2.

[0067] 1. Qualitative evaluation

[0068] Table 1 gives three experimental scenarios based on the Pascal Sentence and Wikipedia data sets. Scene one and scene two are experiments on a single data set, and Pascal Sentence and Wikipedia are divided into three domains as subtasks, respectively. Scene three is an experiment on multiple data sets, and different domains of two data sets are mixed to form a more difficult and more realistic scene. Then the model is input according to the task order of the three scenes in the table for domain incremental image-text retrieval training.

[0069] Adjust the network parameters, and train and test the neural network

[0070] 1. Parameter setting

[0071] The present application adopts the Pytorch 1.8.6 architecture, which is implemented on a single NVIDIA A100 GPU, and uses the Adam optimizer to realize gradient descent. The VIT-B-32 weight is used to initialize the image and text encoder. For all experimental settings, the initial learning rate is set to 0.00001, the batch size is 32, and the iteration number is 30. The specific hyperparameter settings are shown in Table 1, wherein the output dimension d, the number of Transformer layers L and the number of prompts N share the same parameters in the two encoders.

[0072] Table 1: Banzhaf coalition evaluation distillation network hyperparameter settings

[0073]

[0074]

[0075] 2. Input test data set

[0076] For the Pascal Sentence data set, 300 pictures and corresponding text descriptions in three tasks are used for the test set; for the Wikipedia data set, a total of 474 pictures and corresponding text descriptions in three tasks are used for the test set.

[0077] Table 3 gives the experimental results on scenario one and scenario two (the best results are in bold, and the second is underlined). By analyzing the results, it is found that for scenario one and scenario two, the average accuracy of the present invention on the three tasks is 0.6751, 0.8212 and 0.8067 respectively, which is 0.0222, 0.1370 and 0.0697 higher than the most advanced method respectively. In addition, as shown in Table 5, the present invention achieves the best value of accuracy and forgetting rate indicators, which is 0.0833 and 0.0202 higher than the most advanced method respectively. This shows that after learning all the tasks, the present invention maintains better performance on historical tasks and new tasks.

[0078] The tasks across datasets can better reflect the ability of the model to resist forgetting and generalization. Based on this understanding, scenario three is designed, and the results are shown in Table 4. The results show that in the average accuracy indicator of the last five tasks, the present invention is 0.0292, 0.0582, 0.0734, 0.0324 and 0.0115 higher than the second place respectively. For the accuracy and forgetting rate indicators, the present invention is 0.0483 and 0.0222 higher than the second place respectively. Overall, the present invention has better anti-forgetting ability and generalization performance than existing methods.

[0079] Table 2: Domain-incremental task division of Pascal Sentence and Wikipedia datasets

[0080]

[0081]

[0082] Table 3: Image-text retrieval results on scenario one and scenario two

[0083]

[0084] Table 4: Image-text retrieval results on scenario three

[0085]

[0086] Table 5: Accuracy and forgetting rate on three experimental scenarios

[0087]

[0088]

[0089] 2、Subjective effect

[0090] Figure 2 Table 3 gives the experimental results on scenario one and scenario two (the best results are in bold, and the second is underlined). By analyzing the results, it is found that for scenario one and scenario two, the average accuracy of the present invention on the three tasks is 0.6751, 0.8212 and 0.8067 respectively, which is 0.0222, 0.1370 and 0.0697 higher than the most advanced method respectively. In addition, as shown in Table 5, the present invention achieves the best value of accuracy and forgetting rate indicators, which is 0.0833 and 0.0202 higher than the most advanced method respectively. This shows that after learning all the tasks, the present invention maintains better performance on historical tasks and new tasks. Figure 2(a) As can be seen, the model using the Banzhaf coalition evaluation distillation module produces three large attention values on "plane", "runway", "buildings" which represent the main semantics of the image, ranking 2nd in text-image retrieval. The model without using the Banzhaf coalition evaluation distillation module produces only two large attention values on "plane", "runway" and little attention to other key components, ranking 7th in text-image retrieval. From Figure 2 (b) and 2(c) the token similarity matrices of the two models can be seen, the model using the Banzhaf coalition evaluation distillation module assigns larger similarities to the correct regions "plane", "runway", "buildings", while the model without using the Banzhaf coalition evaluation distillation module is not as accurate in similarity matching.

Claims

1. A domain incremental image-text retrieval method based on Banzhaf coalition evaluation distillation, characterized by: First, the image and text are input into the encoder to obtain image and text features; then, through the prompt-key matching strategy, the dynamic semantic prompt is selected using the image and text features, and the dynamic semantic prompt and the image and text embedding features are spliced and then input into the encoder to obtain the final image and text features; then, the similarity of the paired image and text features is constrained through the contrast learning loss and the Banzhaf coalition evaluation loss; finally, the similarity matrix and the Banzhaf coalition evaluation value between the new task and the old task are constrained to achieve the domain incremental image-text retrieval task through the contrast distillation loss and the Banzhaf coalition evaluation distillation loss; Specifically, the following steps are included: Step 1: design an image and text encoder to input the RGB image and the corresponding text into the encoder to obtain fine-grained image and text features; Step 2: design a dynamic semantic module to select the corresponding prompt for the input image and text through the prompt-key matching strategy; then, the selected prompt and the image and text embedding features are spliced and input into the image and text encoder again; Step 3: design a contrast learning and distillation module to constrain the image and text features output by the encoder through the contrast loss, so that the similarity of the paired image and text features increases, and the similarity of the unpaired image and text features decreases; through the contrast distillation, the consistency of the similarity matrix of the new model and the old model on the replay data is constrained to alleviate the catastrophic forgetting; Step 4: design a Banzhaf coalition evaluation distillation module to evaluate the contribution of the image features and the text features to the retrieval task through the Banzhaf coalition evaluation value between them to explore the structured semantic information between modalities; through the Banzhaf coalition evaluation distillation, the distribution approximation of the image and text fine-grained similarity based on the Banzhaf coalition evaluation between the new and old tasks is constrained to realize the propagation of structured semantic information between tasks. 2.The domain incremental image-text retrieval method based on Banzhaf coalition evaluation distillation according to claim 1, characterized by: 1) image and text encoder: To address the differences in data between modalities, a dual-branch encoder was designed to extract image and text features separately. For each image and its corresponding text (I,T) in the k-th task, the input image is first processed... Flatten Then input it into the embedding layer The image embedding feature I1 is obtained, and then I1 is sequentially input into L self-attention layers. linear layer Obtain image features Where L I d represents the number of image patches, p represents the size of the image patch, and d represents the size of the image patch. I d represents the image embedding feature dimension, and d represents the image feature dimension output by the encoder. wherein denotes the concatenation between different operations, is the learnable class token for the image, each transformer layer Trans (l) The superscript l represents the layer index, and a multi-head self-attention structure is adopted, that is: X" = Trans (l) (X) = MLP(LN(X')) + X' X' = MSA(LN(X)) + X (3), where X is the input of Trans (l) , X' is the result after multi-head self-attention, X" is the output of Trans (l) , LN is layer normalization, MSA is multi-head self-attention, and MLP is multi-layer perception; similar to the image modality, the text description T is encoded by a Byte-Pair encoder which is then fed into a BERT model to obtain the final text feature where L T represents the number of text words, and d1 represents the dimension of the text feature; the specific formula is as follows: T' = BERT([t cls ; BPE(T)) (4), wherein is a learnable class token for text, T is the input text description, and T' is the output of the text encoder; 2) dynamic semantic prompt module; The prompt refers to a learnable token; the semantic difference is made up by introducing multiple learnable tokens between the image and text modalities; the token can dynamically aggregate image and text information, thereby achieving more accurate fine-grained matching; specifically, learnable image prompts are added to the image embedding layer and the Byte-Pair encoder layer, respectively and text prompts The specific formula is as follows: where L I := L I + N + 1, L T := L T + N + 1, respectively denote the image and text embedding feature dimensions with added prompt and category tokens; N is the number of image prompts and text prompts, I prompt , T prompt denote the image and text embedding features after adding the prompts, respectively. Then I prompt , T prompt are input into the L-layer self-attention module SA and the BERT model respectively to obtain the final image and text features I", T": To mitigate catastrophic forgetting, a dynamic set of semantic cues is maintained for each task, and the cues are expected to be correctly selected from samples of different tasks during the testing phase; a cue-key matching strategy is used to learn task-related keys; and... An image / text prompt representing the Kth task. This represents the key corresponding to the prompt, determined by minimizing the image / text category vector I' without the prompt. cls (I)(T' cls (T)) and Similarity is used to link hints and keys: wherein denotes a training batch, <·,·> denotes a cosine similarity computation; After all K tasks are trained, get prompt-key pairs, For each test image I test , select the corresponding prompt by finding the key that is most similar to the image features without adding the prompt; then get prompt image embedding: where I test is the test image, I(I test ) is the image test output after the embedding layer, I prompt (I test ) represents the image embedding feature after adding the cue, I”(I test ) is the test image feature output by the encoder, k * represents the selected key, and T * is the cue corresponding to k; T test The process of cue embedding is similar to that of images; 3) contrast learning and distillation module; In the process of incremental image-text retrieval, two tasks need to be completed: first, the alignment of the image modality and the text modality, and second, the preservation of old knowledge to alleviate catastrophic forgetting; for these two tasks, a contrast learning and distillation module is designed; specifically, for the final obtained image features and text features, the similarity of (I, T) is evaluated through the token-level image-text similarity: where u indexes the number of image tokens and v indexes the number of text tokens; A (u,:) denotes the u-th row of matrix A, A (v,:) denotes the v-th row of matrix A; I”(I), T”(T) denote the image and text features, respectively; the image-text contrastive learning loss is as follows: For the second task, since cross-modal knowledge reflects the semantic similarity between images and texts, the knowledge is preserved by keeping the invariance of the image-text similarity matrix of different tasks; the samples in the constraint buffer The consistency of the similarity matrix S k-1 , output by task k and task k-1, where represents the total number of buffer samples; The specific formula is as follows: Sim k-1 (I i ,T j Sim k (I i ,T j The symbols ) represent the token-level image-text similarity obtained by the previous model and the current model, respectively; then, the image-text contrastive distillation loss was designed: 4) Banzhaf coalition evaluation distillation module; The cooperative game theory with Banzhaf includes a set of participants and a characteristic function φ; The characteristic function maps each coalition of players to a real number, the number representing the payoff to all players in the coalition for completing a task; the joint Banzhaf measure captures the extra payoff of a target coalition and the cost to a participant of losing other people's coalitions; given a coalition The Banzhaf coalition evaluation for the pair of players {u, v} is: wherein denotes all participants except players {u,v}, S is a subset of ; Φ(u,v) reflects the tendency of {u,v} to interact with any subset S, in the image-text retrieval task, the higher the value of Φ(u,v), the more the u-th patch of the image and the v-th word of the text description form an alliance; for the image-text retrieval task the contribution to the retrieval task is evaluated by the Banzhaf coalition evaluation value of the image token and the text word token alliance; the joint Banzhaf network is used to evaluate the token similarity matrix of the image-text pair; ψ, ψ * respectively represent the learnable and pre-trained joint Banzhaf network; for each image-text pair (I, T), define the random variable is the Banzhaf coalition evaluation value of the u-th token of I relative to all tokens of T, and the image-text Banzhaf coalition evaluation value: Φ(I" (15), T" (15)) = ψ(Sim(I, T)) (u,:) (v,:) (u,v) (15),​​ where p u,v is the value of the vth element, Φ(I (u,:) , T (v,:) ) denotes the Banzhaf coalition evaluation value between I (u,:) and T (v,:) ; learning random variables and pre-trained network ψ * resulting joint Banzhaf value of the contrast, is represented as: f * (I″ (u,: ),T″ (v,:) )=ψ * (Sim(I,T)) (u,v) (16), where f * (I” (u,:) ,T” (v,:) ) denotes the Banzhaf coalition evaluation between I” * and T” (u,:) obtained from the pre-trained network ψ (v,:) ; the distributions of and are constrained by the KL divergence; similarly, for each image-text pair (I, T), define the random variable as the Banzhaf coalition evaluation of the v-th token of T with respect to all tokens of I, i.e., the text-image Banzhaf coalition evaluation value: Finally, the Banzhaf coalition evaluation loss is defined to depict the distribution approximation of the image and text fine-grained similarity based on the Banzhaf coalition evaluation and the image and text fine-grained similarity based on the pre-trained network: Finally, in order to preserve the structured semantic information and the fine-grained cross-modal relationship learned from the old domain, a joint Banzhaf-based knowledge distillation is proposed. Specifically, the Banzhaf coalition evaluation values of image-text and text-image obtained on the kth and k-1th task models are calculated respectively using the data in the buffer: wherein respectively denote the image-text Banzhaf coalition evaluation values obtained by the data in the buffer on the k-1th and kth task models, and the value of the vth element of respectively denote the text-image Banzhaf coalition evaluation values obtained by the data in the buffer on the k-1th and kth task models, the value of the uth element of​​​ Based on formula (20), the Banzhaf coalition evaluation distillation loss is designed to depict the distribution approximation of the fine-grained cross-modal similarity between the new and old tasks based on the Banzhaf coalition evaluation:

3. The domain incremental image-text retrieval method based on Banzhaf coalition evaluation distillation according to claim 1, characterized in that: Pytorch 1.8.6 architecture is adopted, gradient descent is realized using Adam optimizer on a single NVIDIA A100 GPU; VIT-B-32 weight is used to initialize the image and text encoder; the image encoder is VIT containing 12 layers of self-attention layers, the number of multi-head attention is 12, the input image resolution is 224*224, and the patch size is 32*32; the text encoder is BERT containing 12 layers of self-attention layers, and the number of multi-head attention is 8; all experimental settings, the dynamic semantic prompt size is set to 5, the initial learning rate is 0.00001, the batch size is 32, and the iteration number is 30.

Images