Cross-modal retrieval method and device for online offset and medium

Abstract

The invention discloses a cross-modal retrieval method and device for online offset and a medium, and the method comprises the steps: obtaining an online query stream, and constructing a candidate subset of each query sample in the online query stream; respectively extracting representations of the query samples and candidate subsets of the query samples, and calculating query prediction of each query sample by adopting the two representations; respectively calculating scores between the query sample and the plurality of positive samples, and retaining a sample pair corresponding to the minimum score and a representation corresponding to the sample pair; calculating a modal difference between the sample pairs, and calculating an entropy threshold value of the online query stream; calculating adaptive loss based on the query prediction, the modal difference and the entropy threshold, and determining a gradient updating direction by adopting a gradient decoupling method according to the adaptive loss so as to update the query modal encoder; and extracting the representation of the online query by adopting the updated query modal encoder, calculating the similarity between the representation of the online query and the representation of the candidate sample, and taking the candidate sample with the highest similarity as a cross-modal retrieval result.

Description

Technical Field

[0001] This invention relates to the field of retrieval technology, and more specifically to a cross-modal retrieval method, device, and medium for online offset retrieval. Background Technology

[0002] Cross-modal retrieval aims to achieve semantic associations between data of different modalities (such as images and text, video and audio), and is an important research direction in multimodal representation learning, with significant application value in fields such as search engines and recommender systems. The core of cross-modal retrieval is to map data from different modalities to a common feature space, so that semantically related samples are close to each other in this space, while samples with different semantics are far apart. Through this semantic alignment mechanism, cross-modal retrieval systems can achieve retrieval functions such as "searching for images by text" or "searching for text by images."

[0003] In recent years, the emergence of Vision-Language Pretrained Models (VLMs) has significantly promoted the development of cross-modal retrieval technology. Typical representative models include CLIP and BLIP. The CLIP model is pre-trained on large-scale image-text pairing data using contrastive learning. It extracts multimodal features using image and text encoders respectively, and minimizes the distance between the image and its corresponding text in the embedding space through a contrastive loss function. This method can learn strong semantic alignment capabilities and possesses excellent zero-shot retrieval and classification performance. BLIP, building upon the CLIP framework, further enhances the model's semantic understanding and cross-modal generation capabilities by combining multi-task learning such as image-text matching and image caption generation. The BLIP model can better capture fine-grained semantic relationships in open-domain scenes, providing a more robust multimodal representation for cross-modal retrieval tasks. Summary of the Invention

[0004] In view of the above-mentioned shortcomings in the prior art, the present invention provides a cross-modal retrieval method, device and medium for online offset, which solves the problem that the existing cross-modal retrieval methods cannot effectively deal with online offset.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: Firstly, a cross-modal retrieval method for online offset is provided, comprising the following steps: S1. Obtain the online query stream and construct a candidate subset for each query sample in the online query stream based on the candidate samples in the candidate sample set; S2. The query modality encoder and candidate modality encoder, which are pre-trained neural networks, extract the representations of the query samples and their candidate subsets, respectively, and use the two representations to calculate the query prediction for each query sample. S3. Calculate the scores between the query sample and multiple positive samples respectively, retain the sample pair corresponding to the minimum score and the representation corresponding to the sample pair, and store the data of all query samples in the queue. S4. Calculate the modal differences between sample pairs based on their corresponding representations; and calculate the entropy threshold of the online query stream based on all query samples. S5. Calculate the adaptation loss based on query prediction, modality difference, and entropy threshold, and determine the gradient update direction based on the adaptation loss using the gradient decoupling method to update the query modality encoder. S6. Use an updated query modality encoder to extract the representation of the online query and calculate the similarity between it and the representation of the candidate sample. Use the candidate sample with the highest similarity as the cross-modal retrieval result.

[0006] Furthermore, methods for determining the gradient update direction using gradient decoupling include: Based on the adaptive loss, the optimization direction for general knowledge and domain-specific knowledge is calculated. and : in, For gradient; Let KL divergence be a metric. To query the modal encoder, the first... The query sample and candidate sample set of the first The probability of matching a candidate sample; The updated query modal encoder considers the first... The query sample and the first The probability of matching a candidate sample; To adapt to the loss; B is the total number of query samples in the online query stream; Optimization directions based on general knowledge and domain-specific knowledge and Calculate the gradient update direction to avoid conflicts between specific knowledge and general knowledge: in, This indicates the gradient update direction; To be orthogonal to direction, To be parallel to direction, ; For adaptive weights, It is an exponential function.

[0007] The beneficial effects of the above technical solution are: 1) Improved gradient update direction It will not be in line with the general knowledge optimization direction Conflict arises, that is This means that adapting to diverse queries will not encounter the problem of knowledge forgetting; 2) When the KL divergence increases, that is, when the neural network model is over-optimized, the adaptive weights will decrease accordingly to prevent the neural network model from overfitting to diverse query data.

[0008] Furthermore, the expression for calculating the adaptation loss is: , , in, To adapt to the loss; This represents the loss of modal uniformity. For modal difference loss; This represents the modal consistency loss. To minimize the loss for robust entropy; For robust hard sample mining loss; Let be the i-th query sample in the online query stream; B is the total number of query samples in the online query stream; It is 1 if and only if the condition is met; This is a query sample Weighting coefficients; For query samples The entropy corresponding to the query prediction; Take the larger of the two values; for Entropy; for The first in One sample, It is a subset of all candidate samples in the queue; For query samples The representation Compared with the query sample Corresponding positive samples The representation Cosine similarity between them; T is the transpose; For query samples The representation Compared with the query sample Corresponding negative samples The representation Cosine similarity between them.

[0009] The beneficial effects of the above technical solution are: robust entropy minimization loss. Aiming to maximize Minimize the similarity with positive samples. Similarity to negative samples; robust hard sample mining loss The aim is to increase the similarity between a given query and its corresponding positive sample, and decrease the similarity between a given query and its corresponding hard negative sample, thereby further improving cross-modal consistency.

[0010] In this scheme, although robust entropy minimization loss While this can initially improve cross-modal consistency, the real challenge lies in distinguishing between positive and difficult-to-negative samples. In other words, given a query sample, neural network models often perceive its corresponding positive and difficult-to-negative samples as highly similar and indistinguishable. In contrast, neural network models can effectively distinguish between positive samples and other simple negative samples. Therefore, simply using robust entropy minimization loss is insufficient. It fails to achieve better cross-modal consistency, while using robust hard sample mining loss. This is key to further improving cross-modal consistency.

[0011] Furthermore, modal uniformity loss and modal difference loss The expression: in, It serves as a representation center for online query flows; Euclidean distance; The center of the candidate subset corresponding to the online query stream; Central differences; This is due to modal differences.

[0012] The beneficial effects of the above technical solution are: loss The aim is to increase the uniformity within the query modality, making it easier for the retrieval model to distinguish between different queries; loss The aim is to recover modal differences in the common feature space of the source domain, thereby obtaining good cross-modal alignment relationships.

[0013] Furthermore, the expression for constructing the candidate subset in step S1 is: in, This represents the i-th query sample in the online query stream; for Candidate subset; To exclude from the online query stream Other query samples besides; According to In the candidate sample set The subset obtained by nearest neighbor filtering; Positive samples; For the candidate sample set The subset obtained by k-means clustering; and The multiple candidate samples covered are all negative samples.

[0014] Furthermore, the expression for calculating the query prediction for each query sample in step S2 is as follows: in, For query samples The representation, To query the modal encoder; This represents the i-th query sample in the online query stream; for Query prediction; for Candidate subset; for The representation, Candidate mode encoder; For normalization operators; Temperature number; Indicates transpose; i = 1, 2, ..., B, where B is the total number of query samples in the online query stream; Query prediction that constitutes the online query flow.

[0015] The beneficial effects of the above technical solution are as follows: query prediction can exclude irrelevant samples, effectively avoiding "needle in a haystack" retrieval and preventing model underfitting; in the process of calculating query prediction, valuable negative samples can be selected, which will promote subsequent consistency learning.

[0016] Furthermore, the expression for calculating the score between the query sample and the positive sample is as follows: in, For sample pairs ( , The ratings between ) For query samples The representation, To query the modal encoder; Positive sample The representation, Candidate mode encoder; B is the representation center of the online query stream, and B is the total number of query samples in the online query stream. Euclidean distance; The center of the candidate subset corresponding to the online query stream; The expression for calculating the modal difference between sample pairs is: in, For sample pairs ( , Modal differences between ) for The first in One characteristic, For all query samples in the queue A subset consisting of representations; for The first in One characteristic, For all in the queue A subset consisting of representations; Entropy threshold of online query stream The calculation formula is: in, for Entropy; for The first in One sample, It is a subset of all candidate samples in the queue.

[0017] Furthermore, cross-modal retrieval includes three retrieval tasks: image-text retrieval, video-audio retrieval, and combined retrieval. In image-text retrieval, the query sample is an image, and the candidate sample is text. In video-audio retrieval, the query sample is video, and the candidate sample is audio. In combined retrieval, the query sample is an image and its corresponding text description, and the candidate sample is an image.

[0018] Secondly, a cross-modal retrieval device for online offset is provided, comprising: The candidate subset construction module is used to obtain the online query stream and construct a candidate subset for each query sample in the online query stream based on the candidate samples in the candidate sample set; The query prediction calculation module is used to extract the representations of the query sample and its candidate subset using a pre-trained neural network query modality encoder and a candidate modality encoder, respectively, and to calculate the query prediction for each query sample using the two representations. The scoring calculation module is used to calculate the scores between the query sample and multiple positive samples respectively, retain the sample pair corresponding to the minimum score and the representation of the sample pair, and store the data of all query samples in the queue. The modal difference and entropy threshold calculation module is used to calculate the modal difference between sample pairs based on the corresponding representations of the sample pairs; and to calculate the entropy threshold of the online query stream based on all query samples. The update module is used to calculate the adaptation loss based on query prediction, modality difference and entropy threshold, and to determine the gradient update direction based on the adaptation loss using the gradient decoupling method, and update the query modality encoder. The cross-modal retrieval module is used to extract the representation of the online query using an updated query modality encoder, calculate the similarity between the representation and the representation of the candidate sample, and take the candidate sample with the highest similarity as the cross-modal retrieval result.

[0019] Thirdly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements a cross-modal retrieval method for online offsets.

[0020] The beneficial effects of this invention are as follows: The data retained by this scheme is stored in the source domain formed by the queue. Combined with the calculation of the adaptive loss, the gradient decoupling method is used to determine the gradient update direction, which can restore the common feature space of the source domain and prevent the forgetting of general knowledge. This effectively alleviates the negative impact of online offset on the cross-modal retrieval model and significantly improves the retrieval performance of cross-modal retrieval in various downstream applications. Attached Figure Description

[0021] Figure 1 This is a flowchart of a cross-modal retrieval method for online offsets.

[0022] Figure 2 This is a block diagram illustrating the principle of a cross-modal retrieval device for online offset. Detailed Implementation

[0023] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0024] refer to Figure 1 , Figure 1 A flowchart of a cross-modal retrieval method for online offsets is shown; as follows: Figure 1 As shown, the method S includes steps S1 to S6.

[0025] In step S1, the online query stream is obtained, and a candidate subset of each query sample in the online query stream is constructed based on the candidate samples in the candidate sample set; the expression for constructing the candidate subset in step S1 is: in, This represents the i-th query sample in the online query stream; for Candidate subset; To exclude from the online query stream Other query samples besides; According to In the candidate sample set The subset obtained by nearest neighbor filtering; Positive samples; For the candidate sample set The subset obtained by k-means clustering; and The multiple candidate samples covered are all negative samples.

[0026] In step S2, the query modality encoder and candidate modality encoder of the pre-trained neural network are used to extract the representations of the query samples and their candidate subsets, respectively, and the query prediction of each query sample is calculated using the two representations.

[0027] In implementation, the preferred expression for calculating the query prediction for each query sample in step S2 of this scheme is as follows: in, For query samples The representation, To query the modal encoder; This represents the i-th query sample in the online query stream; for Query prediction; for Candidate subset; for The representation, Candidate mode encoder; For normalization operators; Temperature number; Indicates transpose; i = 1, 2, ..., B, where B is the total number of query samples in the online query stream; Query prediction that constitutes the online query flow.

[0028] In step S3, the scores between the query sample and multiple positive samples are calculated respectively, the sample pair corresponding to the minimum score and the representation corresponding to the sample pair are retained, and the data of all query samples are stored in the queue.

[0029] In implementation, this scheme preferably uses the following expression to calculate the score between the query sample and the positive sample: in, For sample pairs ( , The ratings between ) For query samples The representation, To query the modal encoder; Positive sample The representation, Candidate mode encoder; B is the representation center of the online query stream, and B is the total number of query samples in the online query stream. Euclidean distance; It serves as the center of the candidate subset corresponding to the online query stream.

[0030] In step S4, the modal differences between sample pairs are calculated based on the representations corresponding to the sample pairs; and the entropy threshold of the online query stream is calculated based on all query samples.

[0031] The expression for calculating the modal difference between sample pairs is as follows: in, For sample pairs ( , Modal differences between ) for The first in One characteristic, For all query samples in the queue A subset consisting of representations; for The first in One characteristic, For all in the queue A subset consisting of representations; Entropy threshold of online query stream The calculation formula is: in, for Entropy; for The first in One sample, It is a subset of all candidate samples in the queue.

[0032] In step S5, the adaptation loss is calculated based on query prediction, modality difference, and entropy threshold. Based on the adaptation loss, the gradient decoupling method is used to determine the gradient update direction in order to update the query modality encoder. In one embodiment of the present invention, the expression for calculating the adaptation loss is: , in, To adapt to the loss; This represents the loss of modal uniformity. For modal difference loss; This represents the modal consistency loss. To minimize the loss for robust entropy; For robust hard sample mining loss; Let be the i-th query sample in the online query stream; B is the total number of query samples in the online query stream; It is 1 if and only if the condition is met; This is a query sample Weighting coefficients; For query samples The entropy corresponding to the query prediction; Take the larger of the two values; for Entropy; for The first in One sample, It is a subset of all candidate samples in the queue; For query samples The representation Compared with the query sample Corresponding positive samples The representation Cosine similarity between them; T is the transpose; For query samples The representation Compared with the query sample Corresponding negative samples The representation Cosine similarity between them.

[0033] Modal homogeneity loss and modal difference loss The expression: in, It serves as a representation center for online query flows; Euclidean distance; The center of the candidate subset corresponding to the online query stream; Central differences; This is due to modal differences.

[0034] In step S6, the updated query modality encoder is used to extract the representation of the online query and calculate the similarity between it and the representation of the candidate sample. The candidate sample with the highest similarity is taken as the cross-modal retrieval result.

[0035] During implementation, the retrieval results of the query sample in the preferred step S6 of this scheme are as follows: in, For query samples Search results; For query samples Through the updated query modal encoder The obtained representation; The aim is to find the candidate sample that is most likely to match.

[0036] In one embodiment of the present invention, the method for determining the gradient update direction using a gradient decoupling method includes: Based on the adaptive loss, the optimization direction for general knowledge and domain-specific knowledge is calculated. and : in, For gradient; Let KL divergence be a metric. To query the modal encoder, the first... The query sample and candidate sample set of the first The probability of matching a candidate sample; The updated query modal encoder considers the first... The query sample and the first The probability of matching a candidate sample; To adapt to the loss; B is the total number of query samples in the online query stream; Optimization directions based on general knowledge and domain-specific knowledge and Calculate the gradient update direction to avoid conflicts between specific knowledge and general knowledge: in, This indicates the gradient update direction; To be orthogonal to direction, To be parallel to direction, ; For adaptive weights, It is an exponential function.

[0037] In this scheme, cross-modal retrieval includes three retrieval tasks: image-text retrieval, video-audio retrieval, and combined retrieval. In image-text retrieval, the query sample is an image, and the candidate sample is text. In video-audio retrieval, the query sample is video, and the candidate sample is audio. In combined retrieval, the query sample is an image and its corresponding text description, and the candidate sample is an image.

[0038] like Figure 2 As shown, this solution also provides a cross-modal retrieval device for online offset, which includes: The candidate subset construction module is used to obtain the online query stream and construct a candidate subset for each query sample in the online query stream based on the candidate samples in the candidate sample set; The query prediction calculation module is used to extract the representations of the query sample and its candidate subset using a pre-trained neural network query modality encoder and a candidate modality encoder, respectively, and to calculate the query prediction for each query sample using the two representations. The scoring calculation module is used to calculate the scores between the query sample and multiple positive samples respectively, retain the sample pair corresponding to the minimum score and the representation of the sample pair, and store the data of all query samples in the queue. The modal difference and entropy threshold calculation module is used to calculate the modal difference between sample pairs based on the corresponding representations of the sample pairs; and to calculate the entropy threshold of the online query stream based on all query samples. The update module is used to calculate the adaptation loss based on query prediction, modality difference and entropy threshold, and to determine the gradient update direction based on the adaptation loss using the gradient decoupling method, and update the query modality encoder. The cross-modal retrieval module is used to extract the representation of the online query using an updated query modality encoder, calculate the similarity between the representation and the representation of the candidate sample, and take the candidate sample with the highest similarity as the cross-modal retrieval result.

[0039] This solution also includes a storage medium on which a computer program is stored, which, when executed by a processor, implements a cross-modal retrieval method for online offsets.

[0040] To further verify the effectiveness of the cross-modal retrieval method provided by this invention, experimental evaluations were conducted using various publicly available cross-modal retrieval datasets.

[0041] The experiments primarily included two test scenarios: Online Query Shift (OQS) and Diversified Query Shift (DQS). OQS refers to online query flows with a distribution shift, while DQS refers to online query flows from different domains. For comprehensive evaluation, this invention was compared with several existing time-adaptive methods, including Tent, PL, SHOT, EATA, SAR, READ, TSA, COME, and TCR.

[0042] To verify the effectiveness and generalization ability of the present invention under online distributed offset, the image and text retrieval datasets Flickr and COCO, the video and audio retrieval dataset VGG, and the combined image retrieval datasets CIRR and FIQ were selected for evaluation. The experimental comparison data are shown in Table 1.

[0043] To verify the effectiveness and generalization ability of this invention under diverse distribution shifts, the real-world cross-modal retrieval dataset Fashion-Gen was selected. This dataset contains 48 different product domains, which can fully reflect the diversity of multi-source heterogeneous queries. Furthermore, for image / text retrieval, video / audio retrieval, and combined retrieval tasks, publicly available datasets such as COCO, Flickr, AudioSet, CIRR, and FIQ were selected, and various perturbations were artificially injected to construct different types of distribution shift samples. For example, Gaussian noise and motion blur were added to the image modality; character-level or word-level noise was introduced to the text modality. By injecting these diverse perturbations, the robustness of the model to diverse distribution shifts can be effectively verified. The experimental data comparison is shown in Table 2.

[0044] In Tables 1 and 2, R@K (Recall@K) is defined as the percentage of correct samples among the top K returned samples in the search results, as shown in the tables. This metric measures the accuracy of the retrieval model; a higher value indicates more accurate search results and a better model performance. TR, AR, and CIR represent image-to-text retrieval, video-to-audio retrieval, and combined retrieval, respectively. The suffix "D" indicates a dataset with injected diverse perturbations.

[0045] Table 1. Cross-modal retrieval performance of different adaptation methods under online distribution offset.

[0046] Table 2. Cross-modal retrieval performance of different adaptation methods under diverse distributions (injected with multiple perturbations) during testing. As shown in Tables 1 and 2, the experimental results demonstrate the effectiveness of the proposed method on three cross-modal tasks. Compared to existing methods, the proposed method significantly improves retrieval accuracy when facing online distribution shifts and diverse distribution shifts, exhibiting better robustness and generalization ability.

[0047] Furthermore, this invention conducts ablation experiments on the Fashion-Gen dataset to verify the effectiveness of the robust hard sample mining loss and gradient decoupling method. The verification results are shown in Table 3.

[0048] Table 3 Ablation Experiment Results As shown in Table 3, the experimental results demonstrate that the robust hard sample mining loss and gradient decoupling method proposed in this invention are both effective and can independently improve the performance of the cross-modal retrieval model.

Claims

1. A cross-modal retrieval method for online offset, characterized in that, Including the following steps: S1. Obtain the online query stream and construct a candidate subset for each query sample in the online query stream based on the candidate samples in the candidate sample set; S2. The query modality encoder and candidate modality encoder, which are pre-trained neural networks, extract the representations of the query samples and their candidate subsets, respectively, and use the two representations to calculate the query prediction for each query sample. S3. Calculate the scores between the query sample and multiple positive samples respectively, retain the sample pair corresponding to the minimum score and the representation corresponding to the sample pair, and store the data of all query samples in the queue. S4. Calculate the modal differences between sample pairs based on their corresponding representations; And based on all query samples, calculate the entropy threshold of the online query stream; S5. Calculate the adaptation loss based on query prediction, modality difference, and entropy threshold, and determine the gradient update direction based on the adaptation loss using the gradient decoupling method to update the query modality encoder. S6. Use an updated query modality encoder to extract the representation of the online query and calculate the similarity between it and the representation of the candidate sample. Use the candidate sample with the highest similarity as the cross-modal retrieval result.

2. The cross-modal retrieval method according to claim 1, characterized in that, Methods for determining gradient update direction using gradient decoupling include: Based on the adaptive loss, the optimization direction for general knowledge and domain-specific knowledge is calculated. and : in, For gradient; Let KL divergence be a metric. To query the modal encoder, the first... The query sample and candidate sample set of the first The probability of matching a candidate sample; The updated query modal encoder considers the first... The query sample and the first The probability of matching a candidate sample; To adapt to the loss; B is the total number of query samples in the online query stream; Optimization directions based on general knowledge and domain-specific knowledge and Calculate the gradient update direction to avoid conflicts between specific knowledge and general knowledge: in, This indicates the gradient update direction; To be orthogonal to direction, To be parallel to direction, ; For adaptive weights, It is an exponential function.

3. The cross-modal retrieval method according to claim 1, characterized in that, The expression for calculating the adaptation loss is: ， ， in, To adapt to the loss; This represents the loss of modal uniformity. For modal difference loss; This represents the modal consistency loss. To minimize the loss for robust entropy; For robust hard sample mining loss; Let be the i-th query sample in the online query stream; B is the total number of query samples in the online query stream; It is 1 if and only if the condition is met; This is a query sample Weighting coefficients; For query samples The entropy corresponding to the query prediction; Take the larger of the two values; for Entropy; for The first in One sample, It is a subset of all candidate samples in the queue; For query samples The representation Compared with the query sample Corresponding positive samples The representation Cosine similarity between them; T is the transpose; For query samples The representation Compared with the query sample Corresponding negative samples The representation Cosine similarity between them.

4. The cross-modal retrieval method according to claim 3, characterized in that, Modal homogeneity loss and modal difference loss The expression: in, It serves as a representation center for online query flows; Euclidean distance; The center of the candidate subset corresponding to the online query stream; Central differences; This is due to modal differences.

5. The cross-modal retrieval method according to claim 1, characterized in that, The expression for constructing the candidate subset in step S1 is: in, This represents the i-th query sample in the online query stream; for Candidate subset; To exclude from the online query stream Other query samples besides; According to In the candidate sample set The subset obtained by nearest neighbor filtering; Positive samples; For the candidate sample set The subset obtained by k-means clustering; and The multiple candidate samples covered are all negative samples.

6. The cross-modal retrieval method according to claim 1, characterized in that, The expression for calculating the query prediction for each query sample in step S2 is as follows: in, For query samples The representation, To query the modal encoder; This represents the i-th query sample in the online query stream; for Query prediction; for Candidate subset; for The representation, Candidate mode encoder; For normalization operators; Temperature number; Indicates transpose; i = 1, 2, ..., B, where B is the total number of query samples in the online query stream; Query prediction that constitutes the online query flow.

7. The cross-modal retrieval method according to claim 5, characterized in that, The expression for calculating the score between the query sample and the positive sample is: in, For sample pairs ( , The ratings between ) For query samples The representation, To query the modal encoder; Positive samples The representation, Candidate mode encoder; B is the representation center of the online query stream, and B is the total number of query samples in the online query stream. Euclidean distance; The center of the candidate subset corresponding to the online query stream; The expression for calculating the modal difference between sample pairs is: in, For sample pairs ( , Modal differences between ) for The first in One characteristic, For all query samples in the queue A subset consisting of representations; for The first in One characteristic, For all in the queue A subset consisting of representations; Entropy threshold of online query stream The calculation formula is: in, for Entropy; for The first in One sample, It is a subset of all candidate samples in the queue.

8. The cross-modal retrieval method according to any one of claims 1-7, characterized in that, Cross-modal retrieval includes three retrieval tasks: image-text retrieval, video-audio retrieval, and combined retrieval. In image-text retrieval, the query sample is an image, and the candidate sample is text. In video-audio retrieval, the query sample is video, and the candidate sample is audio. In combined retrieval, the query sample is an image and its corresponding text description, and the candidate sample is an image.

9. A cross-modal retrieval device for online offset, characterized in that, include: The candidate subset construction module is used to obtain the online query stream and construct a candidate subset for each query sample in the online query stream based on the candidate samples in the candidate sample set; The query prediction calculation module is used to extract the representations of the query sample and its candidate subset using a pre-trained neural network query modality encoder and a candidate modality encoder, respectively, and to calculate the query prediction for each query sample using the two representations. The scoring calculation module is used to calculate the scores between the query sample and multiple positive samples respectively, retain the sample pair corresponding to the minimum score and the representation of the sample pair, and store the data of all query samples in the queue. The modal difference and entropy threshold calculation module is used to calculate the modal difference between sample pairs based on the corresponding representations of the sample pairs; And based on all query samples, calculate the entropy threshold of the online query stream; The update module is used to calculate the adaptation loss based on query prediction, modality difference and entropy threshold, and to determine the gradient update direction based on the adaptation loss using the gradient decoupling method, and update the query modality encoder. The cross-modal retrieval module is used to extract the representation of the online query using an updated query modality encoder, calculate the similarity between the representation and the representation of the candidate sample, and take the candidate sample with the highest similarity as the cross-modal retrieval result.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-modal retrieval method for online offset as described in any one of claims 1 to 8.

Images