A continuous training method of a cross-modal retrieval model based on deep evidence rectification

Abstract

The application provides a continuous training method of a cross-modal retrieval model based on deep evidence correction, relates to the technical field of artificial intelligence and multi-modal data processing, and comprises the following steps: extracting cross-modal features of cross-modal data samples, and projecting the cross-modal features to a shared embedding space; according to the cross-modal features projected to the shared embedding space, quantifying and screening the cross-modal data through a deep evidence network and a mixture distribution model to obtain first-class samples and second-class samples in the cross-modal data samples; performing consensus identification on the second-class samples through a near neighbor voting mechanism, and dividing the second-class samples into recovered correction samples and structure constraint samples; constructing a joint loss function according to the first-class samples, the recovered correction samples and the structure constraint samples, and optimizing the cross-modal retrieval model through the joint loss function. The application improves the precision and robustness of the cross-modal retrieval model in a complex dynamic environment.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal data processing technology, and in particular to a continuous training method for a cross-modal retrieval model based on deep evidence correction. Background Technology

[0002] With the explosive growth of multimedia content, cross-modal retrieval has become a key technology for connecting information in different forms. In practical applications, as cross-modal data accumulates continuously with the operation of the platform, retrieval systems need to continuously process a large amount of emerging new task data. However, with the continuous increase in data, semantic drift often occurs, causing the discrimination error of the original model to accumulate continuously, ultimately leading to a significant decrease in the accuracy of the retrieval model.

[0003] To address the challenges posed by the continuous accumulation of new data, existing cross-modal retrieval techniques typically incorporate continuous learning frameworks. Specifically, when processing new task samples, existing methods often forcibly align and map the features of these newly emerging multimodal data directly into a shared feature space. This aims to ensure that the model absorbs new knowledge without forgetting old knowledge, thereby maintaining long-term cross-modal retrieval and discrimination capabilities.

[0004] However, the existing methods mentioned above suffer from a lack of effective outlier handling mechanisms during multimodal alignment and parameter updates. Forced mapping can easily lead to blurred class boundaries, causing the model to misfit meaningless associations and significantly reducing retrieval accuracy. Therefore, to improve the accuracy and robustness of cross-modal retrieval models in complex dynamic environments, how to reasonably handle outliers during the training of cross-modal retrieval models has become an urgent problem to be solved in this field. Summary of the Invention

[0005] This invention provides a continuous training method for a cross-modal retrieval model based on deep evidence correction, in order to solve the technical problem of how to reasonably handle outlier samples in the training of a cross-modal retrieval model.

[0006] This invention provides a continuous training method for a cross-modal retrieval model based on deep evidence correction, comprising: Extract cross-modal features from cross-modal data samples and project the cross-modal features into a shared embedding space; Based on the cross-modal features projected into the shared embedding space, uncertainty quantification and filtering are performed on the cross-modal data using a deep evidence network and a hybrid distribution model to obtain a first type of sample and a second type of sample in the cross-modal data sample; the confidence level of the first type of sample is greater than the confidence level of the second type of sample; The second type of samples is identified through a nearest neighbor voting mechanism, and then divided into recycled correction samples and structural constraint samples. A joint loss function is constructed based on the first type of samples, the recycled and corrected samples, and the structural constraint samples, and the cross-modal retrieval model is optimized using the joint loss function.

[0007] According to the present invention, a continuous training method for a cross-modal retrieval model based on deep evidence correction extracts cross-modal features from cross-modal data samples and projects the cross-modal features into a shared embedding space, comprising: Inject task-specific adapters into the visual and text branches of the cross-modal retrieval model, respectively; The feature output is calculated based on the original weight matrix, input features, and parameter increments of the task-specific adapter; Based on the feature output, obtain the prediction output and global label output of the visual branch and the text branch; Based on the prediction output, the prediction uncertainty is quantified by the deep evidence network, and the target adapter corresponding to the cross-modal data sample is determined based on the prediction uncertainty. The global tag output corresponding to the target adapter is mapped to the shared embedding space through a linear projection layer to obtain the cross-modal features.

[0008] According to the present invention, a continuous training method for a cross-modal retrieval model based on deep evidence correction calculates feature output based on the original weight matrix, input features, and parameter increments of the task-specific adapter, including: Multiply the two parameter decomposition matrices to obtain the parameter increment of the task-specific adapter; Calculate the first product of the original weight matrix and the input features; Calculate the second product of the parameter increment and the input feature; The first product is added to the second product to obtain the feature output.

[0009] According to the present invention, a continuous training method for a cross-modal retrieval model based on deep evidence correction is provided. Based on the cross-modal features projected into the shared embedding space, uncertainty quantification and filtering of the cross-modal data are performed using a deep evidence network and a hybrid distribution model to obtain a first class of samples and a second class of samples in the cross-modal data samples, including: The deep evidence network transforms the prediction output of the cross-modal retrieval model into uncertainty indicators and evidence probability indicators. The loss value obtained based on the cross-modal features and the first combination of the uncertainty index, and the second combination of the loss value and the evidence probability index are respectively input into two parallel mixed distribution models for evaluation to obtain the output probability; The two output probabilities are weighted and fused to obtain a comprehensive compatibility score. Based on the comparison between the score threshold and the comprehensive compatibility score, the cross-modal data samples are divided into the first type of samples and the second type of samples.

[0010] According to the present invention, a continuous training method for a cross-modal retrieval model based on deep evidence correction transforms the prediction output of the cross-modal retrieval model into uncertainty indicators and evidence probability indicators through the deep evidence network, including: The predicted output is transformed into a non-negative evidence quantity through an activation function; Based on the Dirichlet distribution and the amount of non-negative evidence, a class probability distribution is constructed, and the strength parameter of the class probability distribution is extracted. The ratio of the total number of categories in the current task to the sum of evidence for each category is used as the uncertainty index; The ratio of the strength parameter to the sum of the evidence is used as the evidence probability index.

[0011] According to the present invention, a continuous training method for a cross-modal retrieval model based on deep evidence correction is provided, which divides the cross-modal data samples into a first class of samples and a second class of samples based on the comparison result of a score threshold and the comprehensive compatibility score, including: Calculate the statistical distribution of the overall compatibility score for the current batch of samples; The score threshold is adaptively adjusted based on the median of the statistical distribution; Samples whose comprehensive compatibility score is greater than the score threshold are classified as the first type of samples; Samples whose comprehensive compatibility score is less than or equal to the score threshold are classified as the second type of samples.

[0012] According to the present invention, a continuous training method for a cross-modal retrieval model based on deep evidence correction is provided, which uses a nearest neighbor voting mechanism to perform consensus identification on the second type of samples, dividing the second type of samples into retrieval correction samples and structural constraint samples, including: Retrieve multiple nearest neighbor samples of the second type of sample in the shared embedding space; The label distribution of multiple nearest neighbor samples is statistically analyzed and consensus voting is performed to obtain the frequency of occurrence of each label; The labels of the second type of samples whose candidate labels appear more frequently than a preset threshold are corrected to nearest neighbor consensus labels and used as the recycled corrected samples; The second type of samples whose occurrence frequency of each label is not greater than the preset threshold are used as the structural constraint samples.

[0013] According to the present invention, a continuous training method for a cross-modal retrieval model based on deep evidence correction is provided, which constructs a joint loss function based on the first type of samples, the recovered and corrected samples, and the structural constraint samples, including: Calculate the soft classification loss based on the first type of samples and the recovered and corrected samples; Dynamic weights are assigned to positive and negative sample pairs based on the comprehensive compatibility score in the cross-modal alignment process, and identity-level debiasing contrast loss is calculated based on the dynamic weights. The structural-level recovery contrast loss is calculated based on the distance between the structural constraint sample and its nearest neighbor center vector; The joint loss function is obtained by weighted summing of the soft classification loss, the identity-level debiasing contrastive loss, and the structure-level retrieval contrastive loss.

[0014] According to the present invention, a continuous training method for a cross-modal retrieval model based on deep evidence bias correction is provided, which assigns dynamic weights to positive and negative sample pairs based on the comprehensive compatibility score in the cross-modal alignment process, and calculates identity-level debiasing contrast loss based on the dynamic weights, including: Obtain the overall compatibility score output for each sample in the mixed distribution model evaluation; A negative correlation mapping relationship is established based on the comprehensive compatibility score to obtain the dynamic weight; The weighted contrast loss value is calculated based on the dynamic weights, the cosine similarity between the embedded features of positive and negative samples, and the temperature coefficient, to obtain the identity-level debiased contrast loss.

[0015] According to the present invention, a continuous training method for a cross-modal retrieval model based on deep evidence correction calculates a structural-level recovery contrastive loss based on the distance between the structural constraint sample and its nearest neighbor center vector, including: Calculate the nearest neighbor center vector for each of the structurally constrained samples in the shared embedding space; The structural-level reclamation contrastive loss is obtained based on the mean square error between the embedding features of the structural constraint sample and the nearest neighbor center vector.

[0016] The continuous training method for a cross-modal retrieval model based on deep evidence correction provided by this invention effectively distinguishes between normal data, potentially valuable outlier data (recovered and corrected samples), and truly anomalous interference data (structurally constrained samples) through preliminary screening using deep evidence networks and hybrid distribution models, and secondary consensus identification using a nearest neighbor voting mechanism. When constructing the joint loss function for optimization, different constraint strategies are adopted for these three types of samples, thereby avoiding forced mapping of anomalous data during cross-modal alignment and parameter updates. This solves the problem in existing methods where, due to the lack of an effective anomalous data processing mechanism, forced mapping easily leads to blurred inter-class boundaries, causing the model to misfit meaningless associations and significantly reducing retrieval accuracy during multimodal alignment and parameter updates. This method improves the accuracy and robustness of the cross-modal retrieval model in complex dynamic environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the continuous training method for a cross-modal retrieval model based on deep evidence correction provided by the present invention.

[0019] Figure 2 This is a schematic diagram illustrating the principle of the continuous training method for a cross-modal retrieval model based on deep evidence correction provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the structure of the dual-tower encoder and the task-specific LoRA adapter provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the out-of-distribution detection and sample adaptation correction process provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the structure of the continuous training device for the cross-modal retrieval model provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] In real-world applications, data accumulates continuously as the platform runs. To enable models to absorb new knowledge without forgetting old knowledge, continuous learning has been introduced into the field of cross-modal retrieval, allowing models to maintain more effective discrimination capabilities when dealing with emerging new tasks.

[0026] However, existing cross-modal continuous learning methods still face severe robustness challenges in practical deployments. First, as data accumulates, semantic drift often occurs, causing the original model's discrimination error to accumulate and leading to a significant decrease in model discrimination accuracy. Existing methods often forcibly map these new task data into a shared feature space, resulting in blurred inter-class boundaries. Second, traditional continuous learning frameworks generally lack effective outlier detection and handling mechanisms. During multimodal alignment and parameter updates, if these outlier samples cannot be identified and handled appropriately in a timely manner, the model will misfit meaningless associations during playback or adaptation, leading to catastrophic forgetting and significantly reducing cross-modal retrieval accuracy. This problem is particularly prominent in long-tailed distributions or low-quality multimedia streaming scenarios.

[0027] Therefore, there is a need in this field for a new cross-modal retrieval method that integrates out-of-distribution data detection mechanisms to effectively identify and process non-target distributed data during continuous learning, thereby improving the stability and accuracy of the retrieval model in complex dynamic environments.

[0028] The following is combined Figures 1 to 6 This invention describes a continuous training method for a cross-modal retrieval model based on deep evidence correction.

[0029] Figure 1 This is a flowchart illustrating the continuous training method for a cross-modal retrieval model based on deep evidence correction provided by the present invention. Figure 1 As shown, the method includes, but is not limited to, steps S1, S2, S3, and S4.

[0030] Step S1: Extract cross-modal features from cross-modal data samples and project the cross-modal features into a shared embedding space.

[0031] Cross-modal data samples are datasets containing information from two or more different modalities, specifically including multimodal data formats such as image-text pairs, video-audio pairs, text-audio pairs, and image-audio pairs. These data samples typically originate from multimedia content continuously accumulated in real-world application scenarios, such as internet social platforms, video websites, and news feeds.

[0032] Cross-modal features are high-dimensional continuous vectors that can represent the deep semantic information of data, extracted from raw cross-modal data through deep learning models. They are also called feature tensors or embedding representations, such as visual features of images, semantic features of text, or acoustic features of audio.

[0033] Combination Figure 2 Cross-modal features can be extracted using a pre-built dual-tower encoder. This encoder contains independent branches that process data from different modalities, such as an image coding branch responsible for extracting visual features from image data. Figure 2 The Vision Transformer (ViT, the backbone network for visual feature extraction) and the text encoding branch responsible for extracting deep semantic features from text data, such as... Figure 2 Bert (text feature extraction backbone network) in the text.

[0034] The shared embedding space is a unified, multidimensional (e.g., d-dimensional, where d can be 256, 512, 1024, etc.) mathematical vector space in which feature vectors of different modalities can be used for distance measurement (e.g., Euclidean distance) and similarity calculation (e.g., cosine similarity).

[0035] Projection is a process that uses linear or nonlinear transformations to map cross-modal features, originally existing in spaces with different dimensions or distributions, into a unified, shared embedding space. For example... Figure 2 In this process, images and text are processed by visual and text feature extractors, respectively, and then mapped to the feature alignment space (i.e., the shared embedding space) through a projection layer (such as a multilayer perceptron containing fully connected layers), thereby enabling the comparison and alignment of different cross-modal features under the same standard.

[0036] For example, when given a pair of images of "a cat running on the grass" and corresponding descriptive text, the image encoding branch extracts the visual feature vector of the image, and the text encoding branch extracts the semantic feature vector of the text. Then, through matrix multiplication, these two high-dimensional vectors are multiplied by their respective linear projection matrices and mapped into a 512-dimensional shared embedding space, so that their vector distance in this space is as close as possible.

[0037] Step S1 provides a unified and measurable feature representation foundation for subsequent cross-modal retrieval tasks. By extracting features separately through a dual-tower structure and projecting them into a shared space, optimal representation of specialized features within each modality is ensured, while semantic alignment between modalities is achieved.

[0038] Step S2: Based on the cross-modal features projected into the shared embedding space, uncertainty quantification and screening of the cross-modal data are performed using a deep evidence network and a hybrid distribution model to obtain the first type of samples and the second type of samples in the cross-modal data samples; the confidence level of the first type of samples is greater than that of the second type of samples.

[0039] Deep evidence networks are a deep learning probabilistic framework for quantifying models that predict cognitive uncertainty. For example... Figure 2 The evidence learning module in the model not only outputs traditional classification probabilities, but also calculates uncertainty index u and evidence probability index p, thereby effectively identifying samples that the model "does not know what it does not know".

[0040] Mixture distribution models are complex probabilistic models used to describe the overall distribution of data. They are typically composed of a combination of multiple simple distributions, such as... Figure 2 GMM (Gaussian Mixture Model) in the context of Gaussian Mixture Model.

[0041] Uncertainty quantification is used to measure how confident the model is in classifying a sample. Specifically, it can be obtained by converting the model output into Dirichlet distribution parameters using Evidential Deep Learning (EDL) theory.

[0042] Screening involves dividing a continuous data stream or sample set into independent subsets with different confidence levels based on the quantification results. A fixed hyperparameter threshold can be set, or a dynamic threshold can be set by calculating statistics from the batch data. The uncertainty index u and evidence probability index p calculated by the evidence learning module can be combined with the cross-entropy loss ce (e.g., through vector concatenation) and then fed into two parallel GMM modules to output an evaluation probability, which can be used for initial screening.

[0043] The first type of sample is the set of samples with high confidence, that is, normal data that the model considers to be very reliable, such as... Figure 2 The first category consists of high-confidence fit sets. The second category comprises a set of candidate anomaly samples with lower confidence, which may contain noisy or novel data distributions. Figure 2 The low-confidence candidate interference set in the middle.

[0044] For example, for a batch of newly collected image-text pairs, the deep evidence network calculates the uncertainty index of a certain sample to be 0.8 (high). The mixture distribution model, combining the uncertainty index and the loss value, gives the probability that the sample belongs to the normal distribution as 0.2. If the set threshold is 0.5, then the sample is classified as a low-confidence sample (Class II) because 0.2 < 0.5; conversely, if another sample has a probability of 0.9, it is classified as a Class I sample.

[0045] Step S2 can accurately identify the difference between the new sample and the model's current cognitive distribution, effectively capture out-of-distribution interference data that the model cannot provide sufficient evidence to support, and prevent outlier samples from being misfitted in subsequent training.

[0046] Step S3: Consensus identification of the second type of samples is performed through the nearest neighbor voting mechanism, and the second type of samples are divided into recycled correction samples and structural constraint samples.

[0047] The nearest neighbor voting mechanism is a sample classification method based on local structural consistency. It is used to find several nearest neighbors of a sample in the feature space, and then determine the category or attribute of the sample by voting based on the label distribution of these neighbors.

[0048] Consensus identification confirms whether a sample possesses explicit semantic information by examining whether a dominant common label exists among its local neighbors. Specifically, consensus identification can be performed through statistical frequency calculation, weighted voting, or node classification based on graph neural networks.

[0049] Reclaimed and corrected samples are those initially judged to have low confidence, but which have obvious consensus labels in their local neighborhoods, and can therefore be assigned or corrected to that consensus label and reintroduced into the normal training stream. Structural constraint samples are out-of-distribution interference data with extremely high uncertainty that cannot achieve label consensus in their local neighborhoods; these data are usually pure noise.

[0050] For example, for a blurry image in the second type of sample, the 10 nearest neighbors can be found in the feature space. If 8 of these 10 neighbors are labeled as cars, a consensus is reached, the label of the blurry image is corrected to car, and it is used as a sample for recycling and correction. If the labels of these 10 neighbors are varied, such as 2 cats, 3 dogs, and 5 trees, a consensus cannot be reached, and the image is used as a structural constraint sample.

[0051] Step S3 can perform a fine secondary verification on suspected outlier second-class samples, which can both prevent useful information with potential semantic value from being mistakenly isolated in order to salvage the information, and completely isolate the real pure outlier samples.

[0052] Step S4: Construct a joint loss function based on the first type of samples, the recovered and corrected samples, and the structural constraint samples, and optimize the cross-modal retrieval model using the joint loss function.

[0053] The joint loss function is a total loss function composed of a weighted combination of multiple sub-loss functions targeting different optimization objectives. Specifically, it can include classification loss, contrastive loss, mean squared error loss, etc.

[0054] Deep learning frameworks can be used to calculate the errors of each part after forward propagation, then the gradient can be calculated through backpropagation algorithm, and the network weights of the cross-modal retrieval model can be updated using an optimizer.

[0055] Combination Figure 2 The first type of samples (high-confidence fitting set) is used to calculate the identity-level biased contrastive loss; the recovered corrected samples are used to calculate the cross-entropy loss ce; and the structurally constrained samples are used to calculate the structural-level recovery contrastive loss. By combining these three losses proportionally, a total joint loss function is obtained, which is used to update the model parameters.

[0056] For example, in a training iteration, the classification loss on the first class of samples and the recovered correction samples is calculated to be 1.2, the biased contrast loss on the first class of samples is 0.8, and the structural loss on the structurally constrained samples is 0.5. The three are added together with preset weights (such as 1:1:0.5) to obtain a total loss of 2.25. The model is then optimized using this total loss.

[0057] Step S4 can actively weaken the impact of residual outliers on the feature space by jointly optimizing the multi-objective loss, thus avoiding feature collapse caused by forcibly aligning outliers in traditional contrastive learning.

[0058] As described above, the continuous training method for the cross-modal retrieval model based on deep evidence correction, as presented in this invention, effectively distinguishes between normal data, potentially valuable outlier data (recovered and corrected samples), and genuine anomalous interference data (structurally constrained samples) through preliminary screening using deep evidence networks and hybrid distribution models, and secondary consensus identification via nearest neighbor voting mechanisms. When constructing the joint loss function for optimization, different constraint strategies are adopted for these three types of samples, thereby avoiding forced mapping of anomalous data during cross-modal alignment and parameter updates. This solves the problem in existing methods where, due to the lack of an effective anomalous data processing mechanism, forced mapping easily leads to blurred inter-class boundaries, causing the model to misfit meaningless associations and significantly reducing retrieval accuracy during multimodal alignment and parameter updates. This method improves the accuracy and robustness of the cross-modal retrieval model in complex dynamic environments.

[0059] In one implementation, step S1 may further include: Inject task-specific adapters into the visual and text branches of the cross-modal retrieval model, respectively; The feature output is calculated based on the original weight matrix, input features, and parameter increments of the task-specific adapter; Based on the feature output, obtain the predicted output of the visual branch and the text branch, as well as the global label output; Based on the predicted output, the prediction uncertainty is quantified through a deep evidence network, and the target adapter corresponding to the cross-modal data sample is determined based on the prediction uncertainty. The global labeled output corresponding to the target adapter is mapped to the shared embedding space through a linear projection layer to obtain cross-modal features.

[0060] Task-specific adapters are injected into the visual and text branches of the cross-modal retrieval model, respectively. Combined with... Figure 3 The visual branch and the text branch are deep learning backbone networks that process image and text data, respectively. Examples of visual branches include ViT, and examples of text branches include Bert or RoBERTa.

[0061] In the vision branch, the input image first undergoes Patch Embedding linear projection processing, dividing the original image into fixed-size non-overlapping image patches. Each patch is flattened and projected into a vector through a linear layer, forming a serialized token sequence. Simultaneously, learnable positional encodings are appended to preserve the spatial structure information of the image. Subsequently, the token sequence is fed into ViT's multi-layer Transformer Block for global self-attention computation and feedforward network processing. During continuous learning, all pre-trained weights of ViT remain completely frozen, providing only stable low-level visual representations, such as edge detection, texture patterns, and basic geometric shapes.

[0062] In the text branch, the input text sequence is first processed by a tokenizer, which converts the text into a token ID sequence and maps it to an embedding vector. Then, a multi-layer bidirectional Transformer Block is used to extract context-related deep semantic representations. During continuous learning, all the backbone weights of the text branch remain locked to prevent catastrophic forgetting of existing language knowledge when training on new tasks.

[0063] Task-specific adapters are lightweight modules that incrementally update parameters for specific new tasks. Specifically, they can be low-rank adaptive adapters (LoRA).

[0064] Figure 3In the diagram, the Feedforward layer within the Transformer Block is a feedforward network, the Add&Norm Layer is a residual connection and layer normalization layer, and the Attention Layer is an attention layer. The red flame icon in the diagram represents the incremental update part that occurs during continuous learning, namely the task-independent low-rank decomposition matrices from LoRA 0 to LoRA N (corresponding to the low-rank adapters for each task).

[0065] For the vision branch, to adapt to the newly added tasks Task-independent LoRA adapters can be injected in parallel at specific layers of ViT (especially multi-head self-attention MSA layers). The LoRA adapters are inserted in parallel into the query matrix. sum numerical matrix At this point, incremental updates of low-rank parameters are implemented. Similarly, for text branches, parallel injection of parameters specific to the current task can be performed at the multi-head self-attention position of the Transformer layer. The independent LoRA adapter is injected and selected in the same way as the image branch. For example, when deploying a ViT-based vision branch, all original parameters of ViT are locked and do not participate in gradient calculation. Then, a new LoRA adapter module is instantiated and attached to the computation graph of the ViT multi-head attention layer. Only the parameters of this newly attached module participate in training. This step enables incremental learning of new knowledge, ensuring the model's flexibility in absorbing new task knowledge while effectively preventing catastrophic forgetting of old knowledge.

[0066] The feature output is calculated based on the original weight matrix, input features, and parameter increments from the task-specific adapter. The original weight matrix is ​​the pre-trained and frozen weight parameter matrix in the backbone network. The input features are the data vectors input to the current network layer. The parameter increments are additional adjustment matrices or tensors provided by the task-specific adapter.

[0067] The feature calculation output involves element-wise superposition or concatenation of the calculation results from the original backbone path and the incremental path of the adapter. For example, if the input feature is a 1024-dimensional vector X, the original weight matrix maps it to vector Y1 (without gradients throughout), while the adapter maps X to vector Y2 through parameter increments. The final output feature is Y1 + Y2. This step enables the adapter to adjust the model's feature representation, allowing the model to quickly adapt to the data distribution of new tasks without changing the original large-scale pre-trained weights.

[0068] Based on the feature output, obtain the predicted outputs and global label outputs for the visual and text branches. The predicted outputs are the unnormalized logical outputs used for classification at the network's end, such as... Figure 3The logits in the network are used to generate global label outputs, which are special feature vectors used to aggregate global information of the entire image or sentence. Specifically, they can include [CLS] labels, such as visual global features Ft and textual global features Fv. This step can extract efficient high-level feature representations from the deep network that can be used for subsequent uncertainty analysis and cross-modal alignment.

[0069] Based on the predicted output, prediction uncertainty is quantified using a deep evidence network, and a target adapter corresponding to the cross-modal data sample is determined based on this uncertainty. In a multi-task continuous learning scenario, the target adapter is selected from among multiple task-specific adapters currently deployed to process the current input sample. For example... Figure 3 As shown, the evidence learning module outputs the uncertainty values ​​u1, u2, ..., u based on the logits of each branch. N It is possible to compare u1 to u. N The algorithm calculates the uncertainty of a new sample and selects the task with the lowest uncertainty (task_t) as the target adapter to achieve target routing. For example, assuming there are three adapters corresponding to tasks 1, 2, and 3 respectively, after a new sample is input, its uncertainty is calculated to be 0.9 for task 1, 0.2 for task 2, and 0.8 for task 3. The adapter for task 2 is then selected as the target adapter, and this target adapter is used to generate the final features. This step enables intelligent routing and knowledge-selective activation in multi-task scenarios, ensuring strict isolation of different task parameters in physical storage and retrieval.

[0070] The global labeled output corresponding to the target adapter is mapped to the shared embedding space through a linear projection layer to obtain cross-modal features. The linear projection layer is a mapping layer composed of a fully connected neural network. For example... Figure 3 As shown, the routed visual global features Ft and textual global features Fv are fed into the projection layer. The feature vectors are multiplied by a weight matrix and a bias is added to map them to the feature alignment space, thus obtaining the final cross-modal features. and For example, assuming the visual global feature dimension is 768 and the text global feature dimension is 1024, both can be converted into 512-dimensional cross-modal features by multiplying two linear projection layer matrices with shapes of 768×512 and 1024×512 respectively. This step can eliminate the differences in dimensions and original distribution between different cross-modal features, achieving true inter-modal semantic alignment.

[0071] This invention achieves incremental learning and dynamic routing selection for new task knowledge by injecting task-specific adapters in parallel into branches and freezing the backbone network, while simultaneously utilizing the uncertainty of the evidence-based learning output to determine the target adapter route. This not only avoids the catastrophic forgetting of old knowledge caused by directly fine-tuning the entire model, but also effectively isolates the parameters of each task when calculating feature output. Furthermore, by mapping features to a shared space through a projection layer, it achieves efficient decoupling and accurate alignment of different cross-modal features. This allows features to retain their intra-modal expertise while possessing cross-modal comparability, providing a high-quality, decoupled feature foundation for subsequent outlier data processing, and further enhancing the model's representational ability and the stability of continuous learning.

[0072] In one implementation, the feature output is calculated based on the original weight matrix, input features, and parameter increments of the task-specific adapter, and may further include: Multiply the two parameter decomposition matrices to obtain the parameter increments of the task-specific adapter; Calculate the first product of the original weight matrix and the input features; Calculate the second product of the parameter increment and the input feature; Add the first product to the second product to obtain the feature output.

[0073] Multiplying the two parametric decomposition matrices yields the parameter increments for the task-specific adapter. The parametric decomposition matrices are two small-dimensional matrices used for low-rank approximation of a large matrix. The task-specific LoRA adapter uses the low-rank decomposition matrix (parametric decomposition matrix). and This is used to achieve incremental parameter representation. Here, d represents the feature dimension of the backbone network, r represents the low-rank dimension, and the low-rank dimension... Typical values or Then, the parameter increment BA is calculated. Since r is extremely small, the number of parameters in matrices A and B is much smaller than that of a complete d×d matrix. For example, assuming d=1024 and r=8, matrix A has 1024×8=8192 parameters, and matrix B has 8×1024=8192 parameters. Multiplying them yields a 1024×1024 parameter increment matrix, but only 16384 parameters actually need to be stored and updated, instead of the original 1048576 parameters. This step can significantly reduce the number of parameters that need to be updated and stored during continuous learning, improving training efficiency.

[0074] Calculate the first product of the original weight matrix and the input features. Original weight matrix This is the original weight matrix of the backbone network. The input features are... The first product is This step preserves the general stable representations already learned in the backbone network.

[0075] Calculate the second product of the parameter increment and the input features. The second product is BAx. This step can extract specific knowledge adjustments tailored to the current new task.

[0076] Add the first product to the second product to obtain the feature output. Final feature output. .

[0077] This invention employs a low-rank parameterization method, decomposing the massive parameter increment into the multiplication of two extremely small-dimensional matrices, significantly reducing the amount of parameters required for training and storage for each new task. Since the original weight matrix remains frozen and is calculated independently of the increment before being added, the parameter increments for different tasks are completely isolated in terms of physical storage and gradient updates, fundamentally preventing mutual interference during parameter updates. This achieves low-cost, extremely high stability, and scalability of the model.

[0078] In one implementation, step S2 may further include: The prediction output of the cross-modal retrieval model is transformed into uncertainty indicators and evidence probability indicators through deep evidence networks; The first combination of loss value and uncertainty index obtained based on cross-modal features, and the second combination of loss value and evidence probability index are respectively input into two parallel mixed distribution models for evaluation to obtain the output probability; The two output probabilities are weighted and fused to obtain a comprehensive compatibility score. Based on the comparison between the scoring threshold and the comprehensive compatibility score, the cross-modal data samples are divided into Class I samples and Class II samples.

[0079] A deep evidence network transforms the predicted output of a cross-modal retrieval model into uncertainty and evidence probability metrics. The predicted output is the model's Logits, the uncertainty metric *u* measures the model's degree of rejection, and the evidence probability metric *p* represents the probability that the predicted output corresponds to the target category based on existing evidence. A deep evidence network layer can be constructed, non-linearly mapping Logits to parameters using a pre-defined activation function and then calculating them using uncertainty theory formulas. For example, if the model outputs Logits of [5.0, 1.0], after transformation by the evidence network layer, the *u* value for this sample is 0.15, and the *p* value is 0.85. This step transforms unbounded logistic values ​​into statistically significant quantitative indicators, providing a reliable dimension for subsequent screening.

[0080] The first combination of loss value and uncertainty index obtained based on cross-modal features, and the second combination of loss value and evidence probability index, are respectively input into two parallel mixed distribution models for evaluation to obtain the output probability. Figure 4The figure shows the cross-entropy loss ce extracted from the feature alignment space, i.e., it should be... First group The vector concatenation of the normalized loss value and the normalized uncertainty index is given by the following formula: ; Second combination The vector concatenation of the normalized loss value and the normalized evidence probability index is given by the following formula: ; In the formula, As an indicator of uncertainty, For cross-entropy loss, It is an indicator of the probability of evidence.

[0081] Figure 4 The expression indicates that there are two GMM (Gaussian Mixture Model) evaluators. Feature vectors can be input into pre-trained GMM algorithm objects for evaluation. The core probability density calculation formula is: ); Where K is the number of Gaussian mixture components. The mixing weight of the k-th component. The mean Covariance The Gaussian distribution density function.

[0082] After inputting the data, the output probability is obtained by evaluating whether the sample belongs to the high-compatibility probability (i.e., data within the distribution). and For example, the sample vector X1=[0.2, 0.3] is input to the GMM. A The calculated probability is 0.8; X2 = [0.9, 0.3] Input GMM B The calculated probability is 0.82. This step utilizes multi-dimensional feature information for cross-validation evaluation, overcoming the limitations of single-index judgment in complex environments.

[0083] The two output probabilities are weighted and fused to obtain the overall compatibility score. The specific calculation formula is as follows: + ; in, and These are the high compatibility probabilities output by the two GMM evaluators, respectively. and For pre-set or dynamically adjusted fusion weights, such as =0.5、 =0.5. This step can comprehensively improve the robustness of the screening results by integrating the results from three dimensions: loss, uncertainty, and evidence probability.

[0084] Based on the comparison between the scoring threshold and the comprehensive compatibility score, the cross-modal data samples are divided into Class I and Class II samples. Figure 4 When the overall compatibility score exceeds the score threshold, the data flows to the upward path and is divided into a high-confidence adaptation set. This refers to the first type of sample; when the overall compatibility score is less than or equal to the score threshold, the data flows to the lower path and is divided into a low-confidence candidate interference set. This refers to the second type of sample. This step enables efficient physical isolation of mixed samples within a batch.

[0085] This invention constructs parallel GMM models and performs multi-dimensional vector combination and concatenation evaluation. This multimodal deep evidence and loss joint modeling mechanism completely overcomes the limitations of relying solely on loss values ​​or Softmax probabilities, which are prone to misjudgment. The weighted fusion strategy further smooths prediction noise, thereby significantly improving the accuracy and robustness of judging in-distribution and out-of-distribution data in streaming data scenarios with dense outlier samples, effectively preventing the model from misfitting abnormal samples.

[0086] In one implementation, the prediction output of the cross-modal retrieval model is transformed into uncertainty and evidence probability indicators through a deep evidence network, which may further include: The predicted output is transformed into non-negative evidence using an activation function; Based on the Dirichlet distribution and the amount of non-negative evidence, a class probability distribution is constructed, and the strength parameter of the class probability distribution is extracted. The ratio of the total number of categories in the current task to the sum of evidence for each category is used as an uncertainty indicator; The ratio of the strength parameter to the sum of evidence is used as an indicator of the probability of evidence.

[0087] The predicted output is transformed into non-negative evidence using an activation function. The activation function can be the Softplus activation function f(x) = ln(1 + e^(-x)). x The nonnegative evidence measure is a nonnegative real number, denoted as e, representing the strength of a feature's support for the existence of a specific category. The Softplus function can be applied element-wise to each Logit value in the model output to ensure that all output results are greater than 0. This step satisfies the mathematical prerequisite in evidence theory that the evidence measure supporting a hypothesis must be nonnegative.

[0088] A class probability distribution is constructed based on the Dirichlet distribution and the amount of nonnegative evidence, and the strength parameter of the class probability distribution is extracted. The Dirichlet distribution is a continuous multivariate probability distribution used for hyperparameterization of class multivariate distributions. Strength parameter Let be the concentration parameter of the Dirichlet distribution. =e+1. This step transforms the amount of deterministic nonnegative evidence e obtained from the network into the strength parameter of the class probability distribution. This transforms the probability point estimation determined by traditional classification networks into probability distribution estimation with cognitive uncertainty.

[0089] The ratio of the total number of categories in the current task to the sum of evidence for each category is used as the uncertainty index. The sum of evidence for each category is denoted as S, and the total number of categories in the current task is denoted as K. The formula for calculating the uncertainty index is: ; The smaller the sum of valid evidence S extracted by the model, that is, when the model has never seen this type of sample and cannot provide sufficient evidence, the higher the calculated uncertainty index, thus directly and keenly quantifying and capturing out-of-distribution (OOD) data.

[0090] The ratio of the strength parameter to the sum of the evidence is used as the evidence probability index. The calculation formula is: ; The strength parameter for a specific category is typically taken as the strength parameter of the true category or the predicted maximum category. This step can extract the maximum category probability based on the strength of true and valid evidence, eliminating the interference amplified by the traditional Softmax exponential model, and serving as the core input basis for another dimension of GMM evaluation.

[0091] This invention abandons the traditional Softmax confidence calculation method and instead directly models the uncertainty of the model's underlying predictive cognition. Through Dirichlet distribution parameterization, it can extremely sensitively identify the high uncertainty exhibited by the model due to knowledge blind spots at the feature level, thereby accurately identifying potential outliers that the model has never seen before. Compared to existing technologies, it has a higher uncertainty resolution capability, providing an extremely reliable and scientific quantitative input basis for subsequent screening of mixed distribution models.

[0092] In one implementation, based on the comparison between the score threshold and the comprehensive compatibility score, the cross-modal data samples are divided into Class I samples and Class II samples, which may further include: Calculate the statistical distribution of the overall compatibility score for the current batch of samples; The scoring threshold is adaptively adjusted based on the median of the statistical distribution; Samples with a comprehensive compatibility score greater than the score threshold are classified as the first type of samples; Samples with a comprehensive compatibility score less than or equal to the score threshold are classified as the second type of samples.

[0093] Calculate the statistical distribution of the overall compatibility score for the current batch of samples. The current batch of samples is a set of data samples that flow into the model in one forward propagation iteration, for example, a batch size of 256. Overall compatibility score The statistical distribution represents the overall situation of this sample. This step can obtain the overall macro-level compatibility of the current data stream segment, providing a global basis for subsequent dynamic adjustment of the threshold.

[0094] The scoring threshold is adaptively adjusted based on the median of the statistical distribution. Figure 4 The dynamic threshold indicated by the index is the median, which is the value in the middle after sorting the overall compatibility scores of the current batch. The adaptive adjustment is achieved by assigning the calculated median value of the current batch to the score threshold in each training step. For example, after sorting the 256 scores above, we take the average of the scores of the 128th and 129th ranked scores, assuming it's 0.65. Instead of using a preset fixed score threshold like 0.5, we use the score threshold for this batch. The value is adaptively updated to 0.65. This step can handle situations where the distribution patterns and the proportion of outliers in dynamic streaming data are constantly changing.

[0095] Samples with a comprehensive compatibility score greater than a score threshold are classified as Class I samples. This process iterates through all samples; if a sample's... > If the index is not found, it is placed into a high-confidence tensor set, which will then be used as a clean set for normal comparison and classification training. This step can safely separate out clean data that is highly consistent with the model's current understanding.

[0096] Samples with a comprehensive compatibility score less than or equal to the score threshold are classified into the second category. If the sample's... ≤ If a sample is found to be an outlier, it is then added to the corresponding tensor set and entered into the next step of the nearest neighbor correction graph network as a suspected outlier. This step can isolate suspicious samples and prevent them from contaminating the main training process.

[0097] The dynamic median threshold mechanism of this invention overcomes the limitation of traditional methods that use a single fixed threshold and are prone to failure when faced with fluctuations in the quality of different batches of data. By adaptively adjusting the threshold for each batch, the model can self-calibrate according to the actual distribution and overall quality of the incoming data. Even in extreme streaming scenarios with dense distribution of outliers, it can always maintain a reasonable coarse screening ratio, maintain extremely high screening robustness, and stably separate high-confidence and low-confidence samples.

[0098] In one implementation, step S3 may further include: Retrieve multiple nearest neighbor samples of the second type of sample in the shared embedding space; The label distribution of multiple nearest neighbor samples is statistically analyzed and consensus voting is performed to obtain the frequency of occurrence of each label; The labels of the second type of samples whose candidate labels appear more frequently than a preset threshold are corrected to nearest neighbor consensus labels and used as recycled correction samples. The second type of samples, in which the frequency of each label is no greater than a preset threshold, are used as structural constraint samples.

[0099] Retrieve multiple nearest neighbor samples of the second class of samples in the shared embedding space. Combined Figure 4 The nearest neighbor correction and retrieval module can retrieve nearest neighbor samples based on KNN (K-Nearest Neighbors) distance calculation algorithms (such as using the Faiss library to calculate Euclidean or cosine distance in the feature space). For example, for the second type of sample set... For each isolated sample in the training batch, calculate its K=5 nearest neighbor samples in the feature space composed of all samples in the current training batch. This step can utilize the local geometric gravity structure already formed in the feature space to help determine the true semantics of abnormal samples.

[0100] The distribution of labels among multiple nearest neighbor samples is statistically analyzed, and consensus voting is performed to obtain the frequency of each label. The true or predicted category labels of these K neighbors can be read, and the frequency and proportion of each candidate label can be recorded. For example, if the labels of 5 nearest neighbor samples are "A, A, A, A, B", after statistical voting, label A appears 4 times (80%), and label B appears 1 time (20%). This step can quantify the degree of consensus on semantic concepts among samples within a local region.

[0101] The second type of samples whose candidate labels appear more frequently than a preset threshold. The labels were corrected to nearest-neighbor consensus labels and used as the recycled correction samples. .like Figure 4As shown in the corrected sample branch, the preset threshold can be a set consensus ratio, such as 60%. For example, because the frequency of label A is 4 times, which is greater than the preset threshold of 60%, it indicates that although the sample was initially judged to have low confidence, it is still within the local consistency region of label A. Therefore, the label of the sample is corrected to A, and it is sent back to the training stream from the isolation state as a recovered corrected sample. This step can salvage useful samples with potential semantic value but whose representations have local morphological differences (such as changes in viewpoint or illumination).

[0102] The second type of samples, in which the frequency of each label is no greater than a preset threshold, are used as structural constraint samples. For example... Figure 4 As shown in the structural constraint sample branch, a special ignore flag can be marked on this sample to move it into... Set. For example, if the labels of the 5 nearest neighbor samples are "A, B, C, D, E", the highest frequency is only 1, which does not reach the threshold of 3. This indicates that the distribution of the nearest neighbor labels of this sample is extremely chaotic, with no obvious dominant category. It is formally determined to be meaningless out-of-distribution interference data and marked as a structural constraint sample. This step can strictly isolate the real pure noise and cut off its misleading path to the model classifier.

[0103] This invention does not adopt the strategy of simply and crudely eliminating or forcing hard matching of out-of-distribution data in traditional continuous learning methods. Instead, it utilizes the local structural consistency of samples in the feature space to perform fine secondary verification and representation reshaping. This not only accurately removes purely out-of-distribution interference data, but also prevents useful information with potential long-tail semantic value from being erroneously isolated to the greatest extent. This achieves efficient sample retrieval and greatly improves the model's comprehensive utilization rate of limited incremental training information.

[0104] In one implementation, step S4, which involves constructing a joint loss function based on the first type of samples, the recycled corrected samples, and the structural constraint samples, may further include: Calculate the soft classification loss based on the first type of samples and the recovered corrected samples; Dynamic weights are assigned to positive and negative sample pairs based on the comprehensive compatibility score in the cross-modal alignment process, and identity-level debiased contrast loss is calculated based on the dynamic weights. The structural-level recovery contrastive loss is calculated based on the distance between the structural constraint sample and its nearest neighbor center vector. The joint loss function is obtained by weighted summing of the soft classification loss, the identity-level debiasing contrastive loss, and the structure-level retrieval contrastive loss.

[0105] Calculate the soft classification loss based on the first type of samples and the recovered corrected samples. Combined with... Figure 4 The loss module on the right, for the first type of sample. and the recovery of corrected samples Collectively referred to as clean sets Soft classification loss / Represented as: ; in, Let be the total number of samples in the clean set, and y be the (original or corrected) label of the sample. For classifier header parameters, and For visual and textual cross-modal features in a shared space, N represents the number of cross-modal features. This step rigorously ensures the updating of the classifier head parameters. It is not affected by any abnormal data that does not reach a consensus, thus preventing the classification boundary from being skewed.

[0106] Dynamic weights are assigned to positive and negative sample pairs based on the comprehensive compatibility score during cross-modal alignment, and identity-level debiased contrastive loss is calculated based on these dynamic weights. Identity-level debiased contrastive loss / Used to automatically reduce the impact of high-loss pairs that are suspected outliers during modal alignment.

[0107] The structure-level recovery contrastive loss is calculated based on the distance between the structural constraint sample and its nearest neighbor center vector. / Discarded samples can be used as structural anchors to force a smooth distribution of features between modes.

[0108] The joint loss function is obtained by weighted summing of the soft classification loss, identity-level debiasing contrastive loss, and structure-level retrieval contrastive loss. , , , This is an adjustable balance coefficient, with typical values ​​dynamically adjusted according to the task.

[0109] This invention achieves multiple combined effects—classifier center purification, cross-modal alignment debiasing, and inter-modal feature distribution smoothing—by jointly optimizing three cooperative loss functions. Strictly limiting the classification loss to a clean set effectively prevents decision boundary drift; the debiased contrastive loss and structure recovery loss effectively weaken the adverse effects of residual outliers on the feature space, completely avoiding the drawbacks of forced alignment leading to feature collapse and blurred inter-class boundaries in traditional methods. This allows the model to learn a clean alignment space even in complex streaming scenarios with dense outliers.

[0110] In one implementation, dynamic weights are assigned to positive and negative sample pairs based on the comprehensive compatibility score during the cross-modal alignment process, and identity-level debiasing contrastive loss is calculated based on the dynamic weights. This may further include: Obtain the overall compatibility score for each sample in the mixed distribution model evaluation; A negative correlation mapping relationship is established based on the comprehensive compatibility score to obtain dynamic weights; The weighted contrast loss value is calculated based on the dynamic weights, the cosine similarity between the embedded features of positive and negative samples, and the temperature coefficient, to obtain the identity-level debiased contrast loss.

[0111] Obtain the overall compatibility score for each sample in the mixed distribution model evaluation. The overall compatibility score is... This can be denoted as follows in the weight allocation calculation process: .

[0112] A negative correlation mapping relationship is established based on the comprehensive compatibility score to obtain dynamic weights. If the sample's... The extremely low value indicates that it is likely an outlier residual that slipped through the initial screening. A negative correlation mapping is used to assign it a dynamic weight when calculating the loss. Extremely small (approaching 0), enabling automatic debiasing and interception of the contribution of comparative loss.

[0113] The weighted contrastive loss is calculated based on dynamic weights, the cosine similarity between the embedded features of positive and negative samples, and the temperature coefficient, resulting in the identity-level debiased contrastive loss. The specific calculation formula is as follows: ; in, Based on the above The dynamic weights are calculated and mapped. To match the cosine similarity between positive sample pairs (corresponding image-text features); Cosine similarity between mismatched negative sample pairs (incorrect image-text features); A temperature coefficient, such as 0.07, is used to control the smoothness; N represents the total number of sample pairs within the batch. This step introduces a weighted perspective into the contrastive learning framework, precisely reducing the effect of outliers stretching the feature space.

[0114] This invention introduces a debiased contrastive learning strategy with negatively correlated dynamic weights. During cross-modal feature alignment, the model automatically identifies and spontaneously reduces the update contribution of suspected outlier residual samples. This not only eliminates the need for tedious manual data cleaning but also ensures that even if a very small amount of incompletely filtered outlier data is mixed into the current high-volume training batches, these high-loss pairs will not significantly interfere with the alignment of the cross-modal shared embedding space. This significantly improves the model's robustness to feature retrieval in complex, low-quality data streams.

[0115] In one implementation, the structural-level recovery contrastive loss, calculated based on the distance between the structural constraint sample and its nearest neighbor center vector, may further include: Calculate the nearest neighbor center vector for each structurally constrained sample in the shared embedding space; The structural-level reclamation contrastive loss is obtained based on the mean square error between the embedding features of the structurally constrained samples and the nearest neighbor center vector.

[0116] For the samples previously identified as structurally constrained samples—that is, outliers with extremely chaotic and meaningless label distributions—their classification labels are invalidated and they are no longer considered. The calculation is performed, but since it is in the embedding space, the feature representations of its K nearest neighbor samples can be found. The average of these K nearest neighbor vectors is calculated element by element to obtain the nearest neighbor center vector, which finds a geometrically most reasonable structural gravitational support point in the feature space for isolated outliers that have completely lost their semantic labels.

[0117] Structural-level recovery comparison loss The specific calculation formula is as follows: ; in, This represents the total number of samples in the structural constraint sample set. The embedding features of the i-th structural constraint sample itself; Let be the nearest neighbor center vector of the i-th sample; To calculate the square of the L2 norm, i.e., the mean square error.

[0118] This invention, by imposing mean squared error constraints, avoids simply physically discarding extremely uncertain OOD samples that cannot be corrected. Instead, it mandates that their embedded features maintain geometric consistency with their nearest-neighbor center vectors. These seemingly useless discarded samples are transformed into structural anchors that maintain spatial structure. Under the strong guidance of structural loss, this helps the model effectively maintain the global smoothness of feature distribution across different modalities. It effectively avoids local spatial holes and feature collapse that may result from the direct removal of a large number of outliers, greatly improving the model's stability and structural preservation ability in streaming scenarios with dense outliers.

[0119] The following describes the continuous training device for the cross-modal retrieval model provided by the present invention. The continuous training device for the cross-modal retrieval model described below can be referred to in correspondence with the continuous training method for the cross-modal retrieval model based on deep evidence correction described above.

[0120] like Figure 5 As shown, the continuous training device for a cross-modal retrieval model based on deep evidence correction provided by the present invention includes: The feature extraction and projection module is used to extract cross-modal features from cross-modal data samples and project these cross-modal features into a shared embedding space. The data quantification and filtering module is used to quantify and filter the uncertainty of cross-modal data based on the cross-modal features projected onto the shared embedding space, through a deep evidence network and a hybrid distribution model, to obtain the first type of samples and the second type of samples in the cross-modal data samples; the confidence of the first type of samples is greater than the confidence of the second type of samples. The sample consensus identification module is used to identify the second type of samples through the nearest neighbor voting mechanism, and divide the second type of samples into recycling correction samples and structural constraint samples. The retrieval model optimization module is used to construct a joint loss function based on the first type of samples, the recovered and corrected samples, and the structural constraint samples, and to optimize the cross-modal retrieval model through the joint loss function.

[0121] Figure 6 A schematic diagram of the physical structure of an electronic device is provided. This device may include a processor, a communications interface, memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions from the memory to execute a continuous training method for a cross-modal retrieval model based on deep evidence correction.

[0122] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the continuous training method of the cross-modal retrieval model based on deep evidence correction provided by the above methods.

[0124] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a continuous training method for a cross-modal retrieval model based on deep evidence correction provided by the methods described above.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A continuous training method for a cross-modal retrieval model based on deep evidence correction, characterized in that, include: Extract cross-modal features from cross-modal data samples and project the cross-modal features into a shared embedding space; Based on the cross-modal features projected into the shared embedding space, uncertainty quantification and filtering are performed on the cross-modal data using a deep evidence network and a hybrid distribution model to obtain a first type of sample and a second type of sample in the cross-modal data sample; the confidence level of the first type of sample is greater than the confidence level of the second type of sample; The second type of samples is identified through a nearest neighbor voting mechanism, and then divided into recycled correction samples and structural constraint samples. A joint loss function is constructed based on the first type of samples, the recycled and corrected samples, and the structural constraint samples, and the cross-modal retrieval model is optimized using the joint loss function.

2. The continuous training method for a cross-modal retrieval model based on deep evidence correction according to claim 1, characterized in that, Extracting cross-modal features from cross-modal data samples and projecting these cross-modal features into a shared embedding space includes: Inject task-specific adapters into the visual and text branches of the cross-modal retrieval model, respectively; The feature output is calculated based on the original weight matrix, input features, and parameter increments of the task-specific adapter; Based on the feature output, obtain the prediction output and global label output of the visual branch and the text branch; Based on the prediction output, the prediction uncertainty is quantified by the deep evidence network, and the target adapter corresponding to the cross-modal data sample is determined based on the prediction uncertainty. The global tag output corresponding to the target adapter is mapped to the shared embedding space through a linear projection layer to obtain the cross-modal features.

3. The continuous training method for a cross-modal retrieval model based on deep evidence correction according to claim 2, characterized in that, The feature output is calculated based on the original weight matrix, input features, and parameter increments of the task-specific adapter, including: Multiply the two parameter decomposition matrices to obtain the parameter increment of the task-specific adapter; Calculate the first product of the original weight matrix and the input features; Calculate the second product of the parameter increment and the input feature; The first product is added to the second product to obtain the feature output.

4. The continuous training method for a cross-modal retrieval model based on deep evidence correction according to claim 1, characterized in that, Based on the cross-modal features projected into the shared embedding space, uncertainty quantification and filtering are performed on the cross-modal data using a deep evidence network and a hybrid distribution model to obtain a first class of samples and a second class of samples in the cross-modal data samples, including: The deep evidence network transforms the prediction output of the cross-modal retrieval model into uncertainty indicators and evidence probability indicators. The loss value obtained based on the cross-modal features and the first combination of the uncertainty index, and the second combination of the loss value and the evidence probability index are respectively input into two parallel mixed distribution models for evaluation to obtain the output probability; The two output probabilities are weighted and fused to obtain a comprehensive compatibility score. Based on the comparison between the score threshold and the comprehensive compatibility score, the cross-modal data samples are divided into the first type of samples and the second type of samples.

5. The continuous training method for a cross-modal retrieval model based on deep evidence correction according to claim 4, characterized in that, The deep evidence network transforms the prediction output of the cross-modal retrieval model into uncertainty indicators and evidence probability indicators, including: The predicted output is transformed into a non-negative evidence quantity through an activation function; Based on the Dirichlet distribution and the amount of non-negative evidence, a class probability distribution is constructed, and the strength parameter of the class probability distribution is extracted. The ratio of the total number of categories in the current task to the sum of evidence for each category is used as the uncertainty index; The ratio of the strength parameter to the sum of the evidence is used as the evidence probability index.

6. The continuous training method for a cross-modal retrieval model based on deep evidence correction according to claim 4, characterized in that, Based on the comparison result between the scoring threshold and the comprehensive compatibility score, the cross-modal data samples are divided into a first type of sample and a second type of sample, including: Calculate the statistical distribution of the overall compatibility score for the current batch of samples; The score threshold is adaptively adjusted based on the median of the statistical distribution; Samples whose comprehensive compatibility score is greater than the score threshold are classified as the first type of samples; Samples whose comprehensive compatibility score is less than or equal to the score threshold are classified as the second type of samples.

7. The continuous training method for a cross-modal retrieval model based on deep evidence correction according to claim 1, characterized in that, The second type of samples is identified through a nearest neighbor voting mechanism, and then divided into recycled and corrected samples and structurally constrained samples, including: Retrieve multiple nearest neighbor samples of the second type of sample in the shared embedding space; The label distribution of multiple nearest neighbor samples is statistically analyzed and consensus voting is performed to obtain the frequency of occurrence of each label; The labels of the second type of samples whose candidate labels appear more frequently than a preset threshold are corrected to nearest neighbor consensus labels and used as the recycled corrected samples; The second type of samples whose occurrence frequency of each label is not greater than the preset threshold are used as the structural constraint samples.

8. The continuous training method for a cross-modal retrieval model based on deep evidence correction according to claim 1, characterized in that, A joint loss function is constructed based on the first type of samples, the recycled and corrected samples, and the structural constraint samples, including: Calculate the soft classification loss based on the first type of samples and the recovered and corrected samples; Dynamic weights are assigned to positive and negative sample pairs based on the comprehensive compatibility score in the cross-modal alignment process, and identity-level debiasing contrast loss is calculated based on the dynamic weights. The structural-level recovery contrast loss is calculated based on the distance between the structural constraint sample and its nearest neighbor center vector; The joint loss function is obtained by weighted summing of the soft classification loss, the identity-level debiasing contrastive loss, and the structure-level retrieval contrastive loss.

9. The continuous training method for a cross-modal retrieval model based on deep evidence correction according to claim 8, characterized in that, Dynamic weights are assigned to positive and negative sample pairs based on the comprehensive compatibility score during the cross-modal alignment process, and identity-level debiasing contrast loss is calculated based on the dynamic weights, including: Obtain the overall compatibility score output for each sample in the mixed distribution model evaluation; A negative correlation mapping relationship is established based on the comprehensive compatibility score to obtain the dynamic weight; The weighted contrast loss value is calculated based on the dynamic weights, the cosine similarity between the embedded features of positive and negative samples, and the temperature coefficient, to obtain the identity-level debiased contrast loss.

10. The continuous training method for a cross-modal retrieval model based on deep evidence correction according to claim 8, characterized in that, The structural-level recovery contrastive loss is calculated based on the distance between the structural constraint sample and its nearest neighbor center vector, including: Calculate the nearest neighbor center vector for each of the structurally constrained samples in the shared embedding space; The structural-level reclamation contrastive loss is obtained based on the mean square error between the embedding features of the structural constraint sample and the nearest neighbor center vector.

Images