Cross-modal retrieval method for text video

Abstract

The invention relates to the field of information retrieval, and discloses a cross-modal retrieval method for a text video. In the training stage, candidate pairs are input in batches, and the similarity of the candidate pairs is calculated respectively; then, taking the similarity of each candidate pair as a classification probability, and based on Dirichlet distribution, carrying out modeling on the uncertainty of the classification probability; and taking the probability of the classification probability of the candidate pair corresponding to the sample pair as 100% and the probability of other candidate pairs as 0%, constructing a label, and completing the training of the model. In the retrieval stage, the candidate pairs are input in batches, uncertainty modeling is aligned while the similarity is obtained, the similarity is corrected through uncertainty, and matching is conducted through the corrected similarity. Therefore, the method provided by the invention can quantify the uncertainty in the text video pair so as to carry out more reliable similarity evaluation, effectively improve the retrieval performance, and solve the reliability problem caused by only dependence on similarity in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval, and in particular to a cross-modal retrieval method for text and video. Background Art

[0002] With the rise of multimedia platforms like TikTok, YouTube, and Netflix, users are increasingly demanding more accurate text-based video search results. Traditional text-based video retrieval methods primarily rely on keyword searches with video tags, calculating the similarity between the user's query text and the tagged text. This is a single-modal retrieval approach. However, among the vast amount of videos, many lack tags.

[0003] To solve this problem, in recent years, methods based on CLIP (Contrastive Language-Image Pre-training) have made significant progress in text-video multimodal retrieval. The CLIP model maps text and video frames to the same high-dimensional vector space through contrastive learning. By pre-training on large-scale datasets, CLIP learns rich semantic information, enabling text and video content to be accurately matched in the same space. In text-video retrieval, CLIP can directly vectorize text and video frames, and retrieve them by calculating the cosine similarity of the vectors. Due to the pre-training of CLIP on large-scale datasets, its feature expression and generalization capabilities are very strong, and it has achieved remarkable results in multimodal retrieval tasks. However, relying solely on similarity is not enough to support effective cross-modal interaction and reliable same-modal distinction. Taking the given text query "a man is playing baseball" as an example, the top 3 retrieval results returned by the X-Pool model based on feature similarity are as follows: Figure 1 As shown, while #Rank 1 and #Rank 2 are related to baseball, the query does not accurately match the actual description. In contrast, #Rank 3 highlights a man playing baseball and being interviewed. However, this is a failed retrieval task because #Rank 3 includes an irrelevant interview scene, resulting in a larger misalignment in feature interactions. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to propose a cross-modal retrieval method for text and video to solve the reliability problem caused by relying solely on similarity in the existing technology.

[0005] The present invention solves the above technical problems by adopting a technical solution: a cross-modal retrieval method for text and video, comprising:

[0006] A. Model training, including:

[0007] A1. Input B sample pairs for this round of training according to the set hyperparameter B, where B is an integer greater than 1. The sample pairs include image sequences and text, corresponding to the original video-text pairs. The text is the text contained in the corresponding video-text pair, and the image sequence is composed of a set of image frames sampled from the video contained in the corresponding video-text pair.

[0008] Q image sequences are used as queries and C texts are used as candidates; and / or Q texts are used as queries and C image sequences are used as candidates, where at least one of Q and C is an integer greater than 1; for each query, a candidate pair is constructed with the C candidates, each candidate pair including a positive sample and a negative sample of the query, wherein the positive sample is a candidate pair for which a corresponding sample pair exists, and the negative sample is a candidate pair for which a corresponding sample pair does not exist;

[0009] A2. For each candidate pair of each query, extract the video features of its image sequence and the text features of its text, and calculate their similarity based on the extracted video features and text features;

[0010] A3. For each candidate pair of each query, respectively, use the non-negative activation function f(·) to use its similarity Derive the corresponding evidence And calculate the corresponding parameters according to the following formula The evidence Indicates from The support collected will be Q query and the i-th C The measure of support for matching candidates:

[0011]

[0012] A4. Calculate the total loss of this round of training and update the model parameters in reverse order.

[0013] The total loss includes a matching loss based on similarity and an evidence loss of uncertainty in the matching probability from the query to the candidate. The calculation of the evidence loss of uncertainty in the matching probability from the query to the candidate includes:

[0014] For each query, use the parameters obtained in step A3 to construct its Dirichlet distribution And based on its Dirichlet distribution Obtain the Dirichlet distribution probability p of matching with each candidate iQiC ; Among them, Dirichlet distribution Parameters Subscript i Q is the index number queried during the training phase, i Cis the index number of the candidate in the training phase, and the subscript C is the set number of candidates;

[0015] For each query, the Dirichlet distribution probability of each candidate is As the predicted value; with the corresponding label As the true value, where the label corresponding to the positive sample The probability of matching is 100%, corresponding to the label of the negative sample The probability of the matching probability is 0%. The loss of evidence is calculated with the goal of minimizing the difference between the predicted value and the true value.

[0016] Using the evidence loss of each query, we construct the uncertainty of the matching probability from the query to the candidate.

[0017] A5. Repeat steps A1 to A4 until the training end condition is met, and a trained model is obtained.

[0018] B. Enter N Q queries and N C candidates, and N Q and N C At least one of them is an integer greater than 1; based on each query and each candidate input, two of them are combined to form a candidate pair, and a query task containing multiple candidate pairs is constructed;

[0019] Use the trained model to perform retrieval based on query tasks, including:

[0020] B1. For each candidate pair included in the query task, obtain its similarity according to the method in step A2.

[0021] B2. For each query included in the query task Respectively:

[0022] B21. Follow the method in step A3 to obtain the parameters

[0023] B22, according to the parameters The overall uncertainty of matching with each candidate is calculated as follows:

[0024]

[0025] Among them, the subscript j1 is the index number of the query in the actual retrieval stage, and j2 is the index number of the candidate in the actual retrieval stage;

[0026] B3. For each query included in the query task The similarity between it and each candidate obtained in step B1 Use the corresponding uncertainty Make corrections to obtain the final similarity between it and each candidate

[0027] B4. Based on the final similarity between each query and each candidate Get the final matching result.

[0028] In step A2, the similarity is calculated according to the following formula:

[0029]

[0030] in, Indicates the i Q The query characteristics, Indicates the i C candidate features, ||·|| represents the norm;

[0031] In step A3, the corresponding evidence is derived according to the following non-negative activation function

[0032] or

[0033] In step A4, the Dirichlet distribution based on each query According to the following formula, the Dirichlet distribution probability of matching with each candidate is obtained

[0034]

[0035] in, Dirichlet distribution Parameters, Dirichlet distribution Strength;

[0036] In step B3, for each query included in the query task The similarity between it and each candidate obtained in step B1 Use the corresponding uncertainty Correct it according to the following formula to obtain the final similarity between it and each candidate

[0037]

[0038] Among them, γ is the weight hyperparameter.

[0039] Furthermore, in step A4, the contrast loss function is used to calculate the similarity matching loss:

[0040]

[0041] in, Indicates the i Q The similarity c of the positive samples of the query, Indicates the i Q query and the i-th C The similarity of candidates; λ represents the temperature hyperparameter, and e is a natural constant;

[0042] In step A4, the uncertainty evidence loss of the matching probability from the query to the candidate is calculated according to the following formula:

[0043]

[0044] in, Dirichlet distribution Parameters, Dirichlet distribution Strength; To put the i Q query and the i-th C The Dirichlet distribution probability of matching candidates; For the i Q query and the i-th C candidate probability labels, where the label corresponding to the positive sample Label corresponding to negative samples Q is the number of queries and C is the number of candidates.

[0045] Furthermore, the similarity in step A2 is at least one of global similarity, local similarity and fusion similarity;

[0046] The calculation of the global similarity includes: using a visual encoder to obtain a frame embedding of each image frame included in the image sequence; fusing the frame embeddings of each image frame included in the image sequence to obtain a global visual feature of the candidate pair; using a text encoder to use the embedding of the start mark output by the text encoder as the global text feature of the candidate pair; and calculating the global similarity between the image sequence and the text based on the obtained global video feature and global text feature of the candidate pair;

[0047] The calculation of the local similarity includes: using a visual encoder to obtain a frame embedding of each image frame included in the image sequence; using the frame embedding of each image frame included in the image sequence to generate local features of each image frame respectively; fusing the local features of each image frame included in the image sequence to obtain local visual features of the candidate pair; using a text encoder to obtain a word embedding of each word included in the text; using the word embedding of each word included in the text to generate local features of each word respectively; fusing the local features of each word included in the text to obtain local text features of the candidate pair; and calculating the local similarity between the image sequence and the text based on the obtained local video features and local text features of the candidate pair.

[0048] The calculation of the fused similarity includes: fusing the global video features and local video features of the candidate pair image sequence to obtain the fused video features; fusing the global text features and local text features of the candidate pair text to obtain the fused text features; and calculating the fused similarity between the image sequence and the text based on the obtained fused video features and fused text features of the candidate pair;

[0049] If the similarity in step A2 includes at least two similarities among global similarity, local similarity and fused similarity, then: in step A3, for each similarity adopted, its corresponding parameters are calculated respectively; in step A4, for each similarity adopted, the matching loss based on it and the evidence loss using it as the uncertainty of the matching probability are calculated respectively; then, based on the loss of each similarity adopted, the total loss of this round of training is constructed; in step B1, each similarity adopted is calculated respectively; in step B2, for each similarity adopted, its corresponding parameters are calculated respectively, and the corresponding uncertainty is calculated based on the corresponding parameters: in step B3, for each similarity adopted, its uncertainty is used to revise it; finally, the revised similarities are fused to obtain the final similarity.

[0050] Furthermore, the similarity in step A2 includes a global similarity; wherein, the frame embeddings of each image frame included in the image sequence are fused to obtain the global visual features of the image sequence of the candidate pair, including:

[0051] Based on the cross-attention mechanism, the attention weight of the frame visual features of each image frame is obtained according to the following formula, and the sum of the attention weights of the frame visual features of each image frame contained in the image sequence is used as the global visual feature of the image sequence of the candidate pair:

[0052]

[0053] Q T =W Q FT ,K f =W K F f ,V f =W V F f

[0054] Among them, W Q 、W K and W V The query matrix Q T , key matrix K f Sum matrix V f The conversion parameters, T represents the matrix transpose, D is the feature dimension, F T is the global text feature, F f is the frame visual feature of the image frame.

[0055] Furthermore, the similarity in step A2 includes at least one of local similarity and fused similarity; the local visual features and local text features are obtained by independent local feature extraction modules respectively;

[0056] The local feature extraction module includes a linear layer, a self-attention layer and a fusion layer;

[0057] The linear layer extracts the local features of each embedded input according to the following formula:

[0058] F2=ω1F1

[0059] Among them, F1 represents the input feature matrix with a dimension of M×D composed of the input embeddings, and ω1 represents the dimension of the linear layer. The parameter matrix of F2 indicates the output dimension is The local feature matrix of the input; M is the number of embeddings of the input, and D is the feature dimension of the embedding of the input;

[0060] The self-attention layer calculates the self-attention weight of each input local feature according to the following formula:

[0061] β=softmax(ω2tanh(F2))

[0062] Among them, β represents the self-attention weight of dimension M×1, and ω2 represents the dimension of the self-attention layer. Parameters;

[0063] The fusion layer fuses the local features of each input according to the following formula:

[0064] F l =sigmoid(ω3(βF2))

[0065] Among them, ω3 represents the dimension of the fusion layer. The parameter matrix, F 1 The local features of dimension 1×D are obtained by fusing the local features of each input.

[0066] Furthermore, the similarity in step A2 includes fused similarity;

[0067] According to the following formula, the global video features and local video features of the candidate image sequence are fused to obtain the fused video features;

[0068]

[0069] According to the following formula, the global text features and local text features of the candidate text are fused to obtain the fused text features;

[0070]

[0071] in, Represents the global video features, Represents local video features, represents the fused video features, Represents global text features, Represents local text features, Represents the fused text features.

[0072] Further, based on the obtained video features and text features of the candidate pair, the similarity between the image sequence and the text is calculated, including a feature-based feature embedding method and / or a feature-based probability embedding method;

[0073] The feature-based feature embedding method directly uses the video features and text features of the candidate pair to calculate the similarity between its image sequence and text;

[0074] The feature-based probabilistic embedding method calculates the similarity between the image sequence and the text, including:

[0075] Using the visual feature conversion module, the visual features of the candidate pair's image sequence are converted from feature embedding to probabilistic embedding to obtain the mean and variance of the feature distribution of the visual features; using the mean and variance of the feature distribution of the visual features, based on the Gaussian distribution, the probabilistic embedding-based visual features of the candidate pair are generated;

[0076] Using the text feature conversion module, the text features of the candidate pair are converted from feature embedding to probabilistic embedding to obtain the mean and variance of the feature distribution of the text features; using the mean and variance of the feature distribution of the text features, based on the Gaussian distribution, the text features of the candidate pair based on probabilistic embedding are generated;

[0077] Based on the probabilistic embedding-based visual features and probabilistic embedding-based text features of the candidate pairs, the similarity is calculated as the similarity between the candidate pair image sequence and the text.

[0078] Furthermore, based on the obtained video features and text features of the candidate pair, the similarity between the image sequence and the text is calculated, including a feature-based probabilistic embedding method; the feature-based probabilistic embedding method includes:

[0079] The visual features and text features are passed through independent probabilistic embedding modules to generate K probabilistically embedded visual features and K probabilistically embedded text features of candidate pairs;

[0080] In step A, the number of candidates C=B, the number of queries Q=B, and the B image sequences and B texts contained in the input B sample pairs are combined in pairs to form B×B candidate pairs; based on the probabilistically embedded visual features and the probabilistically embedded text features of the candidate pairs, similarity is calculated, including: if the candidate pair is a positive sample, the maximum similarity between its K probabilistically embedded visual features and the K probabilistically embedded text features is used as its similarity; otherwise, the minimum similarity between its K probabilistically embedded visual features and the K probabilistically embedded text features is used as its similarity;

[0081] In step B, the query task includes N×N candidate pairs, where N is an integer greater than 1; similarity is calculated based on the probabilistically embedded visual features and the probabilistically embedded text features of the candidate pairs, including: if the query and candidate index numbers are the same, then the maximum similarity between their K probabilistically embedded visual features and their K probabilistically embedded text features is used as their similarity; otherwise, the minimum similarity between their K probabilistically embedded visual features and their K probabilistically embedded text features is used as their similarity;

[0082] The probabilistic embedding module includes a first linear layer, a second linear layer and a generator network;

[0083] The first linear layer performs linear mapping on the input to obtain the mean of the input feature distribution The second linear layer performs linear mapping on the input to obtain the variance of the input feature distribution The generative network uses the mean of the input feature distribution according to the following formula and variance Generate K outputs;

[0084]

[0085] Where k represents the output index number; ∈ kRepresents a Gaussian distribution The kth noise is obtained by sampling, and e is a natural constant.

[0086] Furthermore, in step B, based on the input N Q queries and their N C Candidates, construct the query task as follows:

[0087] If N Q =N C , then construct a system including N Q ×N C candidate pairs of query tasks; if N Q ≠N C And N Q <N C , then construct N Q query tasks, and query tasks correspond one to one with queries and include N C ×N C candidate pairs; if N Q ≠N C And N Q >N C , then construct N C query tasks, and the query tasks correspond to the candidates one by one and include N Q ×N Q candidate pairs.

[0088] Furthermore, in step A, the image sequence and text are used as queries respectively;

[0089] The similarity in step A2 includes global similarity and fused similarity; wherein the global similarity is calculated by using the obtained global video features and global text features of the candidate pair, using a feature-based feature embedding method to obtain the global similarity between the image sequence and the text; the fused similarity is calculated by using the obtained fused video features and fused text features of the candidate pair, using a feature-based probability embedding method to obtain the fused similarity between the image sequence and the text;

[0090] In step A4, the total loss is constructed according to the following formula

[0091]

[0092] Among them, α L and β L are weights, is the contrast loss for matching based on global similarity, is the contrast loss for matching based on fused similarity, is the evidence loss for uncertainty in matching from query to candidate based on global similarity, is the evidence loss for uncertainty in matching from query to candidate based on the fused similarity, is the divergence loss of probabilistic embedding;

[0093] Contrastive loss and contrast loss Among them, the superscript g represents the similarity based on global similarity, and the superscript m represents the similarity based on fusion. It means that they are all contrast losses as shown in the following formula:

[0094]

[0095] Among them, subscript i1 is the index number of the sample pair to which the image sequence belongs, subscript i2 is the index number of the sample pair to which the text belongs, subscript i is the index number of the sample pair, and s ii represents the similarity between the i-th image sequence and its text, Represents the similarity between the image sequence of the i-th sample and the i2-th text, represents the similarity between the text of the i-th sample and the i1-th image sequence; λ represents the temperature hyperparameter, and e is a natural constant;

[0096] Loss of evidence and loss of evidence Among them, the superscript g represents the similarity based on global similarity, and the superscript m represents the similarity based on fusion. It means that they are all evidence losses as shown in the following formula:

[0097]

[0098] in, Dirichlet distribution Parameters, Dirichlet distribution Strength; To put the i Q query and the i-th C The Dirichlet distribution probability of matching candidates; For the i Q query and the i-th C candidate probability labels, where the label corresponding to the positive sample Label corresponding to negative samples Indicates that when the image sequence is used as the query Indicates that the query is based on text

[0099] Divergence loss for probabilistic embeddings for:

[0100]

[0101] in, Represents the i1th image sequence Fusion of visual features The mean of the characteristic distribution of and variance The distribution represented Indicates the i2th text Fusion of text features The mean of the characteristic distribution of and variance The distribution represented represents the Gaussian distribution, and KL represents the divergence between two probability distributions.

[0102] Furthermore, the similarity in step A2 includes global similarity and fused similarity; the visual encoder of the CLIP model is used to obtain the frame embedding of each image frame contained in the image sequence; the text encoder of the CLIP model is used to obtain the word embedding of each word contained in the text, and the embedding of the start mark output by the text encoder is used as the global text feature of the candidate pair;

[0103] In step B3, use the following formula to use uncertainty Revise the global similarity Obtain the final global similarity

[0104]

[0105] According to the following formula, using uncertainty Revised fusion similarity Get the final fusion similarity

[0106]

[0107] According to the following formula, the final global similarity is integrated Similarity to the final fusion Get the final similarity

[0108]

[0109] Among them, γ1 and γ2 are weight hyperparameters respectively.

[0110] The beneficial effects of the present invention are:

[0111] The inventors believe that the problems existing in the prior art are caused by low-quality data and / or high-similarity data. Among them, low-quality data, such as multi-scene videos, repeated images, and inadequate descriptions, are inherently detrimental to cross-modal interaction and inevitably lead to unreliable retrieval results; while high-similarity data, such as same-scene videos, similar images, and consistent descriptions, can cause confusion, thereby interfering with the reliability of retrieval. Therefore, the inventors believe that to solve the problems of the prior art, it is necessary to introduce a measure of uncertainty in similarity while evaluating the similarity of text-video pairs to determine its reliability.

[0112] Based on the above analysis, the inventors proposed the present method, which treats retrieval queries as classification tasks and constructs labels for training. This method explicitly models the uncertainty of similarity. Given labels and a Dirichlet distribution, the supervised model learns how to generate similarity uncertainty that approximates the labels. This method can quantify uncertainty in text-video pairs, enabling more reliable similarity assessments and effectively improving retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 Display of search results obtained based on a given text query using the X-Pool model;

[0114] Figure 2 Schematic diagram of evidence loss constructed for the present invention. DETAILED DESCRIPTION

[0115] This paper proposes a cross-modal text and video retrieval method. Compared to existing methods that rely solely on feature-level similarity matching, this method also models the uncertainty of similarity. By measuring this uncertainty, it improves retrieval reliability. The model training process includes: A1, inputting candidate pairs in batches; A2, calculating the similarity of each candidate pair; A3, modeling the uncertainty of the similarity; and A4, calculating the loss and updating the results. The actual retrieval process includes: B1, inputting candidate pairs in batches and calculating their similarities; B2, modeling the uncertainty of the similarity; B3, using the uncertainty of the similarity to correct it; and B4, outputting the retrieval results based on the corrected similarity.

[0116] In both the training phase and the actual retrieval phase, candidate pairs must be input in batches. This is so that multiple queries and / or multiple candidates are included in each batch of processing to meet the uncertainty requirement of calculating between multiple queries and candidates. After obtaining multiple queries and / or multiple candidates, the query task can be regarded as a classification task that classifies each query into each candidate, and the similarity is regarded as the classification probability to model its uncertainty. Specifically, taking the training phase as an example, using the query and the similarity between each candidate, the category probability can be expressed as The credible representation of the class probability is Class probability The overall uncertainty is expressed as And follow the following equation:

[0117]

[0118] Setting evidence Indicates from The support collected will be Q query and the i-th C The measure of the support for matching candidates can be obtained by calculating any non-negative activation function f(·), such as: or Higher in classification and lower Indicates greater ambiguity and lower confidence in the results.

[0119] Class probability Credibility uncertainty in, represents the parameters of the Dirichlet distribution, represents the strength of the Dirichlet distribution.

[0120] During the training phase, the loss comes from two parts: similarity and its uncertainty. That is, the total loss includes the matching loss based on similarity and the evidence loss of the uncertainty of the matching probability from the query to the candidate. Among them, the matching loss based on similarity can adopt the loss function used in existing methods. For example, the contrast loss function is used to calculate the matching loss of similarity:

[0121]

[0122] in, Indicates the i Q The similarity of the positive samples of the query, Indicates the i Q query and the i-th C The similarity of candidates; λ represents the temperature hyperparameter, and e is a natural constant.

[0123] The method of the present invention is essentially still a retrieval method, and the candidate pairs do not have corresponding classification labels. Therefore, in order to calculate the evidence loss of uncertainty in the matching probability from the query to the candidate and guide the model to learn the uncertainty from the interaction of text and video modalities, the inventors define: if the query and the candidate belong to the same sample pair, that is, the positive sample, then the label The probability of matching is 100%, for example: If the query and the candidate do not belong to the same sample pair, that is, a negative sample, then the label The probability of a match is 0%, for example:

[0124] At this time, if Figure 2 As shown, given the label and Dirichlet distribution, we can query the Dirichlet distribution probability of the candidate As the predicted value; with the corresponding label As the true value; with the goal of minimizing the difference between the predicted value and the true value, calculate its evidence loss. The evidence loss function can use existing functions under the Bayesian framework, such as: using the generalized mean square error loss as shown below to calculate the evidence loss:

[0125]

[0126] According to the above generalized mean square error loss, for the i-th Q The query can be deduced:

[0127]

[0128] Combining the basic conclusions of Dirichlet distribution, we can obtain:

[0129]

[0130] In the actual retrieval stage, it is not necessary to obtain Only need to obtain Therefore, during the training phase, The calculation of can be done in any existing way, for example: assuming So:

[0131]

[0132] Therefore, the evidence loss of uncertainty in the matching probability from query to candidate can be calculated as follows:

[0133]

[0134] in, Dirichlet distribution Parameters, Dirichlet distribution Strength; To put the i Q query and the i-th C The Dirichlet distribution probability of matching candidates; For the i Q query and the i-th Ccandidate probability labels, where the label corresponding to the positive sample Label corresponding to negative samples Q is the number of queries and C is the number of candidates.

[0135] The method of the present invention can be used to retrieve from video to text, and also from document to video. Therefore, during training, one of the image sequence and text can be used as the query and the other as the candidate, or the image sequence and text can be used as the query respectively. If the image sequence and text are used as the query respectively, the contrast loss is It can be expressed as:

[0136]

[0137] Among them, subscript i1 is the index number of the sample pair to which the image sequence belongs, subscript i2 is the index number of the sample pair to which the text belongs, subscript i is the index number of the sample pair, and s ii represents the similarity between the i-th image sequence and its text, Represents the similarity between the image sequence of the i-th sample and the i2-th text, represents the similarity between the text of the i-th sample and the i1-th image sequence; λ represents the temperature hyperparameter, and e is a natural constant.

[0138] If image sequences and text are used as queries respectively, the evidence loss is It can be expressed as:

[0139]

[0140] in, Dirichlet distribution Parameters, Dirichlet distribution Strength; To put the i Q query and the i-th C The Dirichlet distribution probability of matching candidates; For the i Q query and the i-th C candidate probability labels, where the label corresponding to the positive sample Label corresponding to negative samples Indicates that when the image sequence is used as the query Indicates that the query is based on text

[0141] In summary, the method of the present invention combines prior knowledge and data to complete the modeling of the uncertainty of similarity; in actual retrieval, by using the posterior distribution, the predicted uncertainty can also be obtained, and the matching probability, that is, the similarity, can be corrected accordingly to perform a more reliable similarity assessment, thereby improving the reliability of retrieval.

[0142] However, it should be noted that the core of the present invention lies in how to achieve modeling of uncertainty, and is irrelevant to the method for calculating similarity, the method for obtaining features for calculating similarity, etc.

[0143] For the correction of similarity, we can use the Bayesian method to modify the classification probability using various types of uncertainty, such as: for each query contained in the query task The similarity between it and each candidate obtained in step B1 Use the corresponding uncertainty Correct it according to the following formula to obtain the final similarity between it and each candidate

[0144]

[0145] Among them, γ is the weight hyperparameter.

[0146] Similarity can be calculated using various existing algorithms such as cosine similarity. For example, similarity can be calculated using the following formula:

[0147]

[0148] in, Indicates the i Q The query characteristics, Indicates the i C candidate features, and ||·|| represents the norm.

[0149] Generally, it is considered that the distance d=1-s. Therefore, the method of the present invention can also use distance to measure similarity, and it only needs to be substituted as s=1-d.

[0150] The method for obtaining features for calculating similarity can be any existing method. For example: based on the global video features and global text features of the obtained candidate pair, the global similarity between the image sequence and the text is obtained; based on the local video features and local text features of the obtained candidate pair, the local similarity between the image sequence and the text is obtained; based on the fused video features and fused text features of the obtained candidate pair, the fused similarity between the image sequence and the text is obtained. Therefore, the similarity can be one or more of global similarity, local similarity, and fused similarity. Depending on the actual data distribution characteristics and needs, one or a combination of these can be selected.

[0151] The above-mentioned global similarity is calculated based on the obtained global video features and global text features. Global features refer to features that describe overall information. Among them, for image sequences, the global visual features are obtained by fusing the frame embeddings of each image frame contained in the image sequence, which can reflect the overall structure and content of the video, but are insensitive to local changes. For text, its global text features refer to the features of the entire text, usually using the embedding of the start mark output by the text encoder as its global text features, which helps to understand the overall content and theme of the text. Therefore, modeling the uncertainty of global similarity can ensure the effectiveness of cross-modal interaction and avoid the impact of low-quality data on interaction.

[0152] Local similarity is calculated based on the obtained local video features and local text features. Local features refer to the features of a specific area or local location. For image sequences, first, the frame embeddings of each image frame contained in the image sequence are used to generate local features of each image frame respectively. Then, the local features of each image frame contained in the image sequence are fused to obtain the local visual features of the candidate pairs. Local visual features are sensitive to local changes but lack overall information. For text, first, the word embeddings of each word contained in the text are used to generate local features of each word respectively. Then, the local features of each word contained in the text are fused to obtain the local text features of the candidate pairs. Local text features help understand the local semantics and grammatical structure of the text. Therefore, modeling the uncertainty of local similarity can reliably distinguish homomodal samples and avoid the impact of high similarity data on reliability.

[0153] The fused similarity is calculated based on the obtained fused video features and fused text features. The fused features are achieved by fusing local features with global features, taking advantage of both and compensating for their respective shortcomings.

[0154] When the similarity includes at least two similarities among global similarity, local similarity, and fused similarity, correspondingly, in step A3, for each similarity adopted, its corresponding parameters are calculated respectively; in step A4, for each similarity adopted, its loss is obtained based on its matching loss and the evidence loss for deducing its uncertainty; then, based on the loss of each similarity adopted, the total loss of this round of training is constructed. In step B1, each similarity adopted is calculated respectively; in step B2, for each similarity adopted, its corresponding parameters are calculated respectively, and the corresponding uncertainty is calculated based on the corresponding parameters; in step B3, for each similarity adopted, its uncertainty is revised respectively; finally, the revised similarities are fused to obtain the final similarity.

[0155] In order to better capture the uncertainty, diversity and inherent structural relationships of the data, the present invention also introduces probabilistic embedding. By shifting feature embedding to probabilistic embedding, more semantic information can be captured, thereby achieving a more comprehensive alignment method. Specifically, based on the obtained video features and text features of the candidate pairs, the similarity between their image sequence and text is calculated, including a feature-based feature embedding method and / or a feature-based probabilistic embedding method; the feature-based feature embedding method directly uses the video features and text features of the candidate pairs to calculate the similarity between their image sequence and text; the feature-based probabilistic embedding method calculates the similarity between their image sequence and text, including:

[0156] The visual feature conversion module is used to convert the visual features of the candidate pair's image sequence from feature embedding to probabilistic embedding, and the mean and variance of the feature distribution of the visual features are obtained; the mean and variance of the feature distribution of the visual features are used to generate the visual features of the candidate pair based on probabilistic embedding based on Gaussian distribution; the text feature conversion module is used to convert the text features of the candidate pair's text from feature embedding to probabilistic embedding, and the mean and variance of the feature distribution of the text features are obtained; the mean and variance of the feature distribution of the text features are used to generate the text features of the candidate pair based on probabilistic embedding based on Gaussian distribution; based on the visual features based on probabilistic embedding and the text features based on probabilistic embedding of the candidate pair, the similarity is calculated as the similarity between the candidate pair's image sequence and the text.

[0157] Each of the above-mentioned features is a feature embedding in itself, and can be directly embedded using a feature-based feature embedding method; at the same time, its feature embedding can also be switched to probabilistic embedding, using a feature-based probabilistic embedding method. As an optional solution, it uses a feature from the global, local, or fusion, and uses a feature-based probabilistic embedding method to calculate its similarity; as an optional solution, it uses a feature from the global, local, or fusion, and uses a feature-based feature embedding method to obtain the similarity of the feature embedding of the feature, and at the same time, uses a feature-based probabilistic embedding method to obtain the similarity of the probabilistic embedding of the feature; as an optional solution, it uses a feature, and uses a feature-based feature embedding method to obtain the similarity of the feature embedding of the feature; at the same time, it uses another feature, and uses a feature-based probabilistic embedding method to obtain the similarity of the probabilistic embedding of the feature. The latter two can achieve diversified embedding, reduce semantic interference within the same modality, and overcome the limitations of a single deterministic probabilistic modeling feature.

[0158] In order to achieve effective contrast exclusion, a further feature-based probabilistic embedding method includes: passing visual features and text features through independent probabilistic embedding modules to generate K probabilistically embedded visual features and K probabilistically embedded text features of candidate pairs.

[0159] When calculating similarity using this method, a square matrix needs to be constructed and the values are taken according to the index number. During the training phase, the square matrix can be easily constructed by inputting pairs. It is only necessary to set the number of candidates C = B and the number of queries Q = B. Therefore, during the training phase, the B image sequences and B texts contained in the B input sample pairs are combined into B×B candidate pairs. It is defined that if the candidate pair is a positive sample, the maximum similarity between its K probabilistically embedded visual features and its K probabilistically embedded text features is used as its similarity; otherwise, the minimum similarity between its K probabilistically embedded visual features and its K probabilistically embedded text features is used as its similarity.

[0160] In the actual retrieval stage, it is necessary to construct a query task including N×N candidate pairs, where N is an integer greater than 1, and define: if the query and candidate index numbers are the same, the maximum similarity between the K probabilistically embedded visual features and the K probabilistically embedded text features is used as their similarity; otherwise, the minimum similarity between the K probabilistically embedded visual features and the K probabilistically embedded text features is used as their similarity. However, the actual situation is more complicated. Specifically, based on the input N Q queries and their N C candidates, if N Q =N C , then N Q =N C = N, so we only need to construct a Q ×N C candidate pairs of query tasks; and if N Q ≠N C , it needs to be decomposed into multiple query tasks, such as: N Q <N C , then N C Build N for N Q query tasks, and query tasks correspond one to one with queries and include N C ×N C candidate pairs; N Q >N C , then N Q Build N for N C query tasks, and the query tasks correspond to the candidates one by one and include N Q ×N Q candidate pairs.

[0161] Preferably, the similarity in step A2 includes global similarity and fused similarity. The global similarity uses the obtained global video features and global text features of the candidate pair, adopting a feature-based feature embedding approach to calculate the global similarity between the image sequence and the text, thus overcoming the cross-modal interaction problem. The fused similarity uses the obtained fused video features and fused text features of the candidate pair, adopting a feature-based probabilistic embedding approach to calculate the fused similarity between the image sequence and the text, thus overcoming the same-model differentiation problem.

[0162] The following is further described with reference to the embodiments.

[0163] Example

[0164] In this embodiment, a cross-modal text and video retrieval method is provided, and model training thereof includes:

[0165] A1. Data input

[0166] In this step, according to the set hyperparameter B, B sample pairs for this round of training are input, where B is an integer greater than 1.

[0167] The sample pairs consist of image sequences and text, corresponding to the original video-text pairs. The text is the text contained in the corresponding video-text pair, and the image sequence is composed of a set of image frames sampled from the video contained in the corresponding video-text pair. In this embodiment, uniform sampling is used. The B image sequences and B text contained in the B input sample pairs are combined in pairs to form B×B candidate pairs.

[0168] To facilitate subsequent calculations, in this embodiment, the uncertainty matrix of the global similarity, the uncertainty matrix of the fusion similarity, and the label matrix are also constructed with dimensions of B×B, such as Figure 2 As shown, the rows of each matrix correspond to the text T, the columns correspond to the image sequence V, and the main diagonal element i Q =i C =i corresponds to sample pair i. The label matrix is evaluated according to the following indicator function:

[0169]

[0170] A2. Similarity Calculation

[0171] In this step, for each candidate pair, the video features of its image sequence and the text features of its text are extracted respectively, and based on the extracted video features and text features, the similarity between its image sequence and text is calculated.

[0172] In this embodiment, similarity includes global similarity and fused similarity. Therefore, each feature needs to be extracted separately, and any existing algorithm can be used. In this embodiment, the CLIP model's visual encoder is used to obtain the frame embeddings of each image frame in the image sequence; the CLIP model's text encoder is used to obtain the word embeddings of each word in the text.

[0173] In this embodiment, the acquisition of global features includes: using the embedding of the start sign [EOS] output by the text encoder as the global text feature of the candidate pair; based on the cross-attention mechanism, obtaining the attention weight of the frame visual feature of each image frame according to the following formula, and using the sum of the attention weights of the frame visual features of each image frame contained in the image sequence as the global visual feature of the image sequence of the candidate pair:

[0174]

[0175] Q T =W Q F T ,K f =W K F f ,V f =W V F f

[0176] Among them, W Q 、W K and W V The query matrix Q T , key matrix K f Sum matrix V f The conversion parameters, T represents the matrix transpose, D is the feature dimension, F T is the global text feature, F f is the frame visual feature of the image frame.

[0177] In this embodiment, local features are obtained by using independent local feature extraction modules to obtain local visual features and local text features. The local feature extraction module includes a linear layer, a self-attention layer, and a fusion layer.

[0178] The linear layer extracts the local features of each embedded input according to the following formula:

[0179] F2=ω1F1

[0180] Among them, F1 represents the input feature matrix with a dimension of M×D composed of the input embeddings, and ω1 represents the dimension of the linear layer. The parameter matrix of F2 indicates the output dimension is The local feature matrix of ; M is the number of input embeddings, and D is the feature dimension of the input embedding.

[0181] The self-attention layer calculates the self-attention weight of each input local feature according to the following formula:

[0182] β=softmax(ω2tanh(F2))

[0183] Among them, β represents the self-attention weight of dimension M×1, and ω2 represents the dimension of the self-attention layer. Parameters.

[0184] The fusion layer fuses the local features of each input according to the following formula:

[0185] F l =sigmoid(ω3(βF2))

[0186] Among them, ω3 represents the dimension of the fusion layer. The parameter matrix, F l The local features of dimension 1×D are obtained by fusing the local features of each input.

[0187] In this embodiment, the fusion features are obtained according to the following formulas:

[0188]

[0189] in, Represents the global video features, Represents local video features, represents the fused video features, Represents global text features, Represents local text features, Represents the fused text features.

[0190] In this embodiment, the global similarity is calculated by using the global video features and global text features of the candidate pairs, adopting a feature-based feature embedding method, and using the following formula to calculate the global similarity between the image sequence and the text:

[0191]

[0192] in, represents the global video features of the i1-th image sequence, represents the global text feature of the i2th text, and ||·|| represents the norm.

[0193] In this embodiment, the fused similarity is calculated by using the obtained fused video features and fused text features of the candidate pair and adopting a feature-based probabilistic embedding method to obtain the fused similarity between the image sequence and the text, including:

[0194] First, the fused visual features and the fused text features are respectively passed through independent probabilistic embedding modules to generate K probabilistically embedded visual features and K probabilistically embedded text features of candidate pairs.

[0195] Then, calculate K×K similarities according to the following formula. If the image sequence and the text belong to the same sample pair, the maximum similarity between the K probabilistically embedded visual features and the K probabilistically embedded text features is used as the fused similarity; otherwise, the minimum similarity between the K probabilistically embedded visual features and the K probabilistically embedded text features is used as the fused similarity:

[0196]

[0197] in, represents the probability embedding video feature of the kth image sequence i1, represents the k-th probability embedding text feature of the i2-th text, and ||·|| represents the norm.

[0198] In this embodiment, the probabilistic embedding module includes a first linear layer, a second linear layer and a generator network;

[0199] The first linear layer performs linear mapping on the input to obtain the mean of the input feature distribution The second linear layer performs linear mapping on the input to obtain the variance of the input feature distribution The generative network uses the mean of the input feature distribution according to the following formula and variance Generate K outputs;

[0200]

[0201] Where k represents the output index number; ∈ k Represents a Gaussian distribution The kth noise is obtained by sampling, and e is a natural constant.

[0202] A3. Uncertainty Modeling

[0203] In this step, for each candidate pair, the similarity between its image sequence and text is used through the non-negative activation function f(·) Derive the corresponding evidence And calculate the corresponding parameters according to the following formula The evidence Indicates from The support collected in matches the i1th image sequence with the i2th text:

[0204]

[0205] In this embodiment, the corresponding evidence is derived according to the following non-negative activation function:

[0206]

[0207] In this embodiment, the similarity includes global similarity and fused similarity. Therefore, it is necessary to calculate and obtain corresponding parameters for each similarity used.

[0208] Specifically, for the global similarity Calculate the corresponding parameters Similarity for fusion Calculate the corresponding parameters And the parameters and Assign values to the uncertainty matrix of global similarity and the uncertainty matrix of fusion similarity respectively.

[0209] A4. Parameter Update

[0210] In this step, the total loss of this round of training is calculated and the model parameters are updated in reverse.

[0211] In this embodiment, similarity includes global and fused similarity, and uses image sequences and text as queries. Therefore, for each similarity, it is necessary to calculate the matching loss based on it and the evidence loss using it as the uncertainty of the matching probability. Then, based on the losses for each similarity, the total loss for this round of training is constructed. Furthermore, to prevent convergence to zero, this embodiment also introduces a divergence loss between probabilistic embeddings.

[0212] Therefore, the total loss of this embodiment is for:

[0213]

[0214] Among them, α L and β L are weights, is the contrast loss for matching based on global similarity, is the contrast loss for matching based on fused similarity, is the evidence loss for uncertainty in matching from query to candidate based on global similarity, is the evidence loss for uncertainty in matching from query to candidate based on the fused similarity, is the divergence loss of the probabilistic embedding.

[0215] Contrastive loss and contrast loss Among them, the superscript g represents the similarity based on global similarity, and the superscript m represents the similarity based on fusion. It means that they are all contrast losses as shown in the following formula:

[0216]

[0217] Among them, subscript i1 is the index number of the sample pair to which the image sequence belongs, subscript i2 is the index number of the sample pair to which the text belongs, subscript i is the index number of the sample pair, and s ii represents the similarity between the i-th image sequence and its text, Represents the similarity between the image sequence of the i-th sample and the i2-th text, Represents the similarity between the text of the i-th sample and the i1-th image sequence; λ represents the temperature hyperparameter, and e is a natural constant. The temperature hyperparameter λ1 and contrastive loss in The temperature hyperparameter λ2 in can be the same or different.

[0218] Loss of evidence and loss of evidence Among them, the superscript g represents the similarity based on global similarity, and the superscript m represents the similarity based on fusion. When calculating, first, the uncertainty matrix of global similarity and the uncertainty matrix of fusion similarity obtained in step A3 are used to construct parameters by row and column respectively:

[0219]

[0220] Then, based on the parameters and Constructing Dirichlet distribution and And according to the formula Calculate the strength of each Dirichlet distribution.

[0221] Then, based on Dirichlet distribution and According to the following formula:

[0222]

[0223] Obtain Dirichlet distribution probability and The superscript V indicates that an image sequence is used as a query, and the superscript T indicates that a text is used as a query.

[0224] Afterwards, the label matrix is used to obtain the corresponding true value, combined with the Dirichlet distribution probability and According to the following evidence losses The calculation formula is used to calculate and obtain the evidence loss of each part respectively. and

[0225]

[0226] Finally, according to the following formula, and Loss of evidence pass and Loss of evidence

[0227]

[0228] Divergence loss for probabilistic embeddings for:

[0229]

[0230] in, Represents the i1th image sequence V i1 Fusion of visual features The mean of the characteristic distribution of and variance The distribution represented Represents the i2th text T i2 Fusion of text features The mean of the characteristic distribution of and variance The distribution represented represents the Gaussian distribution, and KL represents the divergence between two probability distributions.

[0231] A5. Iteration

[0232] Steps A1 to A4 are executed repeatedly until the training end condition is met, and a trained model is obtained.

[0233] After training is complete, enter N Q queries and N C candidates, if N Q =N C , then construct a system including N Q ×N C candidate pairs of query tasks; if NQ ≠N C And N Q <N C , then construct N Q query tasks, and query tasks correspond one to one with queries and include N C ×N C candidate pairs; if N Q ≠N C And N Q >N C , then construct N C query tasks, and the query tasks correspond to the candidates one by one and include N Q ×N Q Candidate pairs. According to the query task, the trained model is used for retrieval, including:

[0234] B1. Similarity calculation

[0235] In this step, for each candidate pair included in the query task, the similarity s is obtained according to the method of step A2. j1j2 In this embodiment, the similarity includes global similarity and fusion similarity. Therefore, in this step, the global similarity is calculated separately. Similarity to fusion

[0236] B2. Uncertainty Modeling

[0237] In this step, for each query included in the query task Respectively:

[0238] B21. Follow the method in step A3 to obtain the parameters

[0239] B22, according to the parameters The following formula is used to calculate the difference between the candidate The overall uncertainty of the matching

[0240]

[0241] Among them, the subscript j1 is the index number of the query in the actual retrieval stage, and j2 is the index number of the candidate in the actual retrieval stage.

[0242] Similarly, in this embodiment, the similarity includes global similarity and fusion similarity. This step is for global similarity. Similarity to fusion Calculate the corresponding parameters respectively and And according to the corresponding parameters, calculate the corresponding uncertainty and

[0243] B3. Similarity Correction

[0244] In this step, for each query included in the query task The obtained in step B1 is compared with each candidate Similarities between Use the corresponding uncertainty According to the following formula, make corrections and obtain the difference between it and each candidate The final similarity between

[0245]

[0246] Among them, γ is the weight hyperparameter.

[0247] Similarly, similarity includes global similarity and fused similarity. It is necessary to revise each adopted similarity by using its uncertainty. Finally, the revised similarities are fused to obtain the final similarity. Specifically, in this embodiment, according to the following formula, using uncertainty Revise the global similarity Obtain the final global similarity

[0248]

[0249] According to the following formula, using uncertainty Revised fusion similarity Get the final fusion similarity

[0250]

[0251] According to the following formula, the final global similarity is integrated Similarity to the final fusion Get the final similarity

[0252]

[0253] Among them, γ1 and γ2 are weight hyperparameters respectively.

[0254] B4. Output

[0255] This step is based on the final similarity between each query and each candidate Get the final matching result.

[0256] Experimental verification:

[0257] To verify the effectiveness of the present invention, the inventors compared the solution of the above embodiment with existing methods on five datasets: MSRVTT, DiDeMo, MSVD, Charades, and VATEX. The test metrics included R@k, MdR, and MnR.

[0258] R@k is an evaluation metric used in retrieval systems. R@1 represents the proportion of correct targets found in the top 1 search result, R@5 represents the proportion of correct targets found in the top 5 search results, and R@10 represents the proportion of correct targets found in the top 10 search results. MdR represents the median rank of correct results in the search result list. MnR represents the average rank of correct results across all queries.

[0259] The model training cycle is 5; the image sequence consists of 12 frames uniformly sampled from the video and resized to 224×224 pixels. The hyperparameters are set as follows: α L =1×10 -1 , β L =1×10 -4 , B=32, λ1=λ2=1×10 -1 , K=7.

[0260] To evaluate the impact of each component on the model, the inventors conducted an ablation study, and the results are shown in Table 1.

[0261] Table 1. Test results of ablation study

[0262]

[0263] Model #1, as the baseline model, uses feature-based feature embedding to calculate global similarity; Model #2 uses feature-based feature embedding to calculate global similarity and model its uncertainty; Model #3 uses feature-based feature embedding to calculate global similarity; and uses feature-based probabilistic embedding to calculate fused similarity; Model #4 uses feature-based feature embedding to calculate global similarity and model its uncertainty; and uses feature-based probabilistic embedding to calculate fused similarity; Model #5 uses feature-based feature embedding to calculate global feature embedding similarity; and uses feature-based probabilistic embedding to calculate global probabilistic embedding similarity; Model #6 uses feature-based feature embedding to calculate global similarity and model its uncertainty; and uses feature-based probabilistic embedding to calculate fused similarity and model its uncertainty.

[0264] As shown in Table 1, by increasing uncertainty, solution #2 improves retrieval performance by 1.3% (R@1) compared to solution #1, validating the effectiveness of modeling uncertainty. Compared to solution #3 (#1), and solution #4 (#2), by using diverse probabilistic embedding to address the limitations of single feature representation and increase inter-modal differentiation, this improves retrieval performance by 2.8% (R@1). Solution #6, also described in the examples, further improves the model's retrieval performance by 2.9% (R@1) compared to solution #4.

[0265] The retrieval results of the embodiment model and the comparison model on the MSRVTT dataset are shown in Table 2. The model of the embodiment surpasses the comparison model in both text-to-video and video-to-text retrieval tasks, improving R@1 by 2.0%.

[0266] Table 2. Retrieval performance on the MSRVTT dataset

[0267]

[0268] The text-to-video retrieval results on the DiDeMo, MSVD, Charades, and VATEX datasets are shown in Table 3. The model of the embodiment shows consistent improvements on multiple datasets, including: the multi-text MSVD dataset, with an R@1 improvement of 3.0%; the long video DiDeMo dataset, with an R@1 improvement of 10.9%.

[0269] To evaluate the model's generalization ability on unseen data, we compared it to out-of-domain text-to-video retrieval. First, we trained the model on the "source" dataset, MSRVTT, and then tested it on the "target" dataset, DiDeMo. The results are shown in Table 4. We can observe that while some models perform strongly in their domain, their performance often degrades significantly when generalized to out-of-domain data. Compared to other methods, the model in this example demonstrates superior retrieval performance for both in-domain and out-of-domain scenarios.

[0270] In Table 4, R@Sum=R@1+R@5+R@10.

[0271] Table 3. Retrieval performance on DiDeMo, MSVD, Charades, and VATEX datasets

[0272]

[0273] Table 4. Out-of-domain text-to-video retrieval performance

[0274]

[0275] Finally, it should be noted that the above embodiments are merely preferred implementations and are not intended to limit the present invention. It should be noted that those skilled in the art will be able to make modifications, equivalent substitutions, and improvements without departing from the spirit and scope of the present invention and the claims, all of which should be included within the scope of protection of the present invention.

Claims

1. A cross-modal retrieval method for text and video, characterized by: A. Model training, including: A1. Input B sample pairs for this round of training according to the set hyperparameter B, where B is an integer greater than 1. The sample pairs include image sequences and text, corresponding to the original video-text pairs. The text is the text contained in the corresponding video-text pair, and the image sequence is composed of a set of image frames sampled from the video contained in the corresponding video-text pair. Q image sequences are used as queries and C texts are used as candidates; and / or Q texts are used as queries and C image sequences are used as candidates, where at least one of Q and C is an integer greater than 1; for each query, a candidate pair is constructed with the C candidates, each candidate pair including a positive sample and a negative sample of the query, wherein the positive sample is a candidate pair for which a corresponding sample pair exists, and the negative sample is a candidate pair for which a corresponding sample pair does not exist; A2. For each candidate pair of each query, extract the video features of its image sequence and the text features of its text, and calculate their similarity based on the extracted video features and text features; A3. For each candidate pair of each query, respectively, use the non-negative activation function f(·) to use its similarity Derive the corresponding evidence And calculate the corresponding parameters according to the following formula The evidence Indicates from The support collected will be Q query and the i-th C The measure of support for matching candidates: A4. Calculate the total loss of this round of training and update the model parameters in reverse order. The total loss includes a matching loss based on similarity and an evidence loss of uncertainty in the matching probability from the query to the candidate. The calculation of the evidence loss of uncertainty in the matching probability from the query to the candidate includes: For each query, use the parameters obtained in step A3 to construct its Dirichlet distribution And based on its Dirichlet distribution Obtain the Dirichlet distribution probability p of matching with each candidate iQiQ ; Among them, Dirichlet distribution Parameters Subscript i Q is the index number queried during the training phase, i C is the index number of the candidate in the training phase, and the subscript C is the set number of candidates; For each query, the Dirichlet distribution probability of each candidate is As the predicted value; with the corresponding label As the true value, where the label corresponding to the positive sample The probability of matching is 100%, corresponding to the label of the negative sample The probability of the matching probability is 0%. The loss of evidence is calculated with the goal of minimizing the difference between the predicted value and the true value. Using the evidence loss of each query, we construct the uncertainty of the matching probability from the query to the candidate. A5. Repeat steps A1 to A4 until the training end condition is met, and a trained model is obtained. B. Enter N Q queries and N C candidates, and N Q and N C At least one of them is an integer greater than 1; based on each query and each candidate input, two of them are combined to form a candidate pair, and a query task containing multiple candidate pairs is constructed; Use the trained model to perform retrieval based on query tasks, including: B1. For each candidate pair included in the query task, obtain its similarity according to the method in step A2. B2. For each query included in the query task Respectively: B21. Follow the method in step A3 to obtain the parameters B22, according to the parameters The overall uncertainty of matching with each candidate is calculated as follows: Among them, the subscript j1 is the index number of the query in the actual retrieval stage, and j2 is the index number of the candidate in the actual retrieval stage; B3. For each query included in the query task The similarity between it and each candidate obtained in step B1 Use the corresponding uncertainty Make corrections to obtain the final similarity between it and each candidate B4. Based on the final similarity between each query and each candidate Get the final matching result.

2. The cross-modal text and video retrieval method according to claim 1, wherein: In step A2, the similarity is calculated according to the following formula: in, Indicates the i q The query characteristics, Indicates the i C candidate features, ||·|| represents the norm; In step A3, the corresponding evidence is derived according to the following non-negative activation function or In step A4, the Dirichlet distribution based on each query According to the following formula, the Dirichlet distribution probability of matching with each candidate is obtained in, Dirichlet distribution Parameters, Dirichlet distribution Strength; In step B3, for each query included in the query task The similarity between it and each candidate obtained in step B1 Use the corresponding uncertainty Correct it according to the following formula to obtain the final similarity between it and each candidate Among them, γ is the weight hyperparameter.

3. The cross-modal text and video retrieval method according to claim 1, wherein: In step A4, the contrast loss function is used to calculate the similarity matching loss: in, Indicates the i Q The similarity of the positive samples of the query, Indicates the i Q query and the i-th c The similarity of candidates; λ represents the temperature hyperparameter, and e is a natural constant; In step A4, the uncertainty evidence loss of the matching probability from the query to the candidate is calculated according to the following formula: in, Dirichlet distribution Parameters, Dirichlet distribution Strength; To put the i Q query and the i-th C The Dirichlet distribution probability of matching candidates; For the i Q query and the i-th C candidate probability labels, where the label corresponding to the positive sample Label corresponding to negative samples Q is the number of queries and C is the number of candidates.

4. The cross-modal text and video retrieval method according to claim 1, wherein: The similarity in step A2 is at least one of global similarity, local similarity, and fused similarity; The calculation of the global similarity includes: using a visual encoder to obtain a frame embedding of each image frame included in the image sequence; fusing the frame embeddings of each image frame included in the image sequence to obtain a global visual feature of the candidate pair; using a text encoder to use the embedding of the start mark output by the text encoder as the global text feature of the candidate pair; and calculating the global similarity between the image sequence and the text based on the obtained global video feature and global text feature of the candidate pair; The calculation of the local similarity includes: using a visual encoder to obtain a frame embedding of each image frame included in the image sequence; using the frame embedding of each image frame included in the image sequence to generate local features of each image frame respectively; fusing the local features of each image frame included in the image sequence to obtain local visual features of the candidate pair; using a text encoder to obtain a word embedding of each word included in the text; using the word embedding of each word included in the text to generate local features of each word respectively; fusing the local features of each word included in the text to obtain local text features of the candidate pair; and calculating the local similarity between the image sequence and the text based on the obtained local video features and local text features of the candidate pair. The calculation of the fused similarity includes: fusing the global video features and local video features of the candidate pair image sequence to obtain the fused video features; fusing the global text features and local text features of the candidate pair text to obtain the fused text features; and calculating the fused similarity between the image sequence and the text based on the obtained fused video features and fused text features of the candidate pair; If the similarity in step A2 includes at least two similarities among global similarity, local similarity and fused similarity, then: in step A3, for each similarity adopted, its corresponding parameters are calculated respectively; in step A4, for each similarity adopted, the matching loss based on it and the evidence loss using it as the uncertainty of the matching probability are calculated respectively; then, based on the loss of each similarity adopted, the total loss of this round of training is constructed; in step B1, each similarity adopted is calculated respectively; in step B2, for each similarity adopted, its corresponding parameters are calculated respectively, and the corresponding uncertainty is calculated based on the corresponding parameters: in step B3, for each similarity adopted, its uncertainty is used to revise it; finally, the revised similarities are fused to obtain the final similarity.

5. The cross-modal text and video retrieval method according to claim 4, wherein: The similarity in step A2 includes global similarity; wherein, the frame embeddings of each image frame included in the image sequence are fused to obtain the global visual features of the image sequence of the candidate pair, including: Based on the cross-attention mechanism, the attention weight of the frame visual features of each image frame is obtained according to the following formula, and the sum of the attention weights of the frame visual features of each image frame contained in the image sequence is used as the global visual feature of the image sequence of the candidate pair: Q T =W Q F T ,K f =W K F f ,V f =W V F f Among them, W Q 、W K and W V The query matrix Q T , key matrix K f Sum matrix V f The conversion parameters, T represents the matrix transpose, D is the feature dimension, F T is the global text feature, F f is the frame visual feature of the image frame.

6. The cross-modal text and video retrieval method according to claim 4, wherein: The similarity in step A2 includes at least one of local similarity and fusion similarity; The local visual features and local text features are respectively obtained by independent local feature extraction modules; The local feature extraction module includes a linear layer, a self-attention layer and a fusion layer; The linear layer extracts the local features of each embedded input according to the following formula: F2=ω1F1 Among them, F1 represents the input feature matrix with a dimension of M×D composed of the input embeddings, and ω1 represents the dimension of the linear layer. The parameter matrix of F2 indicates the output dimension is The local feature matrix of the input; M is the number of embeddings of the input, and D is the feature dimension of the embedding of the input; The self-attention layer calculates the self-attention weight of each input local feature according to the following formula: β=softmax(ω2tanh(F2)) Among them, β represents the self-attention weight of dimension M×1, and ω2 represents the dimension of the self-attention layer. Parameters; The fusion layer fuses the local features of each input according to the following formula: F l =sigmoid(ω3(βF2)) Among them, ω3 represents the dimension of the fusion layer. The parameter matrix, F l The local features of dimension 1×D are obtained by fusing the local features of each input.

7. The cross-modal text and video retrieval method according to claim 4, wherein: The similarity in step A2 includes the fusion similarity; According to the following formula, the global video features and local video features of the candidate image sequence are fused to obtain the fused video features; According to the following formula, the global text features and local text features of the candidate text are fused to obtain the fused text features; in, Represents the global video features, Represents local video features, represents the fused video features, Represents global text features, Represents local text features, Represents the fused text features.

8. The cross-modal text and video retrieval method according to any one of claims 4 to 7, characterized in that: Based on the obtained video features and text features of the candidate pair, calculating the similarity between the image sequence and the text, including a feature-based feature embedding method and / or a feature-based probability embedding method; The feature-based feature embedding method directly uses the video features and text features of the candidate pair to calculate the similarity between its image sequence and text; The feature-based probabilistic embedding method calculates the similarity between the image sequence and the text, including: Using the visual feature conversion module, the visual features of the candidate pair's image sequence are converted from feature embedding to probabilistic embedding to obtain the mean and variance of the feature distribution of the visual features; using the mean and variance of the feature distribution of the visual features, based on the Gaussian distribution, the probabilistic embedding-based visual features of the candidate pair are generated; Using the text feature conversion module, the text features of the candidate pair are converted from feature embedding to probabilistic embedding to obtain the mean and variance of the feature distribution of the text features; using the mean and variance of the feature distribution of the text features, based on the Gaussian distribution, the text features of the candidate pair based on probabilistic embedding are generated; Based on the probabilistic embedding-based visual features and probabilistic embedding-based text features of the candidate pairs, the similarity is calculated as the similarity between the candidate pair image sequence and the text.

9. The cross-modal text and video retrieval method according to claim 8, wherein: Based on the obtained video features and text features of the candidate pairs, the similarity between the image sequence and the text is calculated, including a feature-based probabilistic embedding method; The feature-based probabilistic embedding method includes: The visual features and text features are passed through independent probabilistic embedding modules to generate K probabilistically embedded visual features and K probabilistically embedded text features of candidate pairs; In step A, the number of candidates C=B, the number of queries Q=B, and the B image sequences and B texts contained in the input B sample pairs are combined in pairs to form B×B candidate pairs; based on the probabilistically embedded visual features and the probabilistically embedded text features of the candidate pairs, similarity is calculated, including: if the candidate pair is a positive sample, the maximum similarity between its K probabilistically embedded visual features and the K probabilistically embedded text features is used as its similarity; otherwise, the minimum similarity between its K probabilistically embedded visual features and the K probabilistically embedded text features is used as its similarity; In step B, the query task includes N×N candidate pairs, where N is an integer greater than 1; similarity is calculated based on the probabilistically embedded visual features and the probabilistically embedded text features of the candidate pairs, including: if the query and candidate index numbers are the same, then the maximum similarity between their K probabilistically embedded visual features and their K probabilistically embedded text features is used as their similarity; otherwise, the minimum similarity between their K probabilistically embedded visual features and their K probabilistically embedded text features is used as their similarity; The probabilistic embedding module includes a first linear layer, a second linear layer and a generator network; The first linear layer performs linear mapping on the input to obtain the mean of the input feature distribution The second linear layer performs linear mapping on the input to obtain the variance of the input feature distribution The generative network uses the mean of the input feature distribution according to the following formula and variance Generate K outputs; Where k represents the output index number; ∈ k Represents a Gaussian distribution The kth noise is obtained by sampling, and e is a natural constant.

10. The cross-modal text and video retrieval method according to claim 9, wherein: In step B, based on the input N Q queries and their N C Candidates, construct the query task as follows: If N Q =N C , then construct a system including N Q ×N C candidate pairs of query tasks; if N Q ≠N C And N Q <N C , then construct N Q query tasks, and query tasks correspond one to one with queries and include N C ×N C candidate pairs; if N Q ≠N C And N Q >N C , then construct N C query tasks, and the query tasks correspond to the candidates one by one and include N Q ×N Q candidate pairs.

11. The cross-modal text and video retrieval method according to claim 9, wherein: In step A, image sequences and text are used as queries respectively; The similarity in step A2 includes global similarity and fused similarity; wherein the global similarity is calculated by using the obtained global video features and global text features of the candidate pair, using a feature-based feature embedding method to obtain the global similarity between the image sequence and the text; the fused similarity is calculated by using the obtained fused video features and fused text features of the candidate pair, using a feature-based probability embedding method to obtain the fused similarity between the image sequence and the text; In step A4, the total loss is constructed according to the following formula Among them, α L and β L are weights, is the contrast loss for matching based on global similarity, is the contrast loss for matching based on fused similarity, is the evidence loss for uncertainty in matching from query to candidate based on global similarity, is the evidence loss for uncertainty in matching from query to candidate based on the fused similarity, is the divergence loss of probabilistic embedding; Contrastive loss and contrast loss Among them, the superscript g represents the similarity based on global similarity, and the superscript m represents the similarity based on fusion. It means that they are all contrast losses as shown in the following formula: Among them, subscript i1 is the index number of the sample pair to which the image sequence belongs, subscript i2 is the index number of the sample pair to which the text belongs, subscript i is the index number of the sample pair, and s ii represents the similarity between the i-th image sequence and its text, Represents the similarity between the image sequence of the i-th sample and the i2-th text, represents the similarity between the text of the i-th sample and the i1-th image sequence; λ represents the temperature hyperparameter, and e is a natural constant; Loss of evidence and loss of evidence Among them, the superscript g represents the similarity based on global similarity, and the superscript m represents the similarity based on fusion. It means that they are all evidence losses as shown in the following formula: in, Dirichlet distribution Parameters, Dirichlet distribution Strength; To put the i Q query and the i-th C The Dirichlet distribution probability of matching candidates; For the i Q query and the i-th C candidate probability labels, where the label corresponding to the positive sample Label corresponding to negative samples Indicates that when the image sequence is used as the query Indicates that the query is based on text Divergence loss for probabilistic embeddings for: in, Represents the i1th image sequence Fusion of visual features The mean of the characteristic distribution of and variance The distribution represented Indicates the i2th text Fusion of text features The mean of the characteristic distribution of and variance The distribution represented represents the Gaussian distribution, and KL represents the divergence between two probability distributions.

12. The cross-modal text and video retrieval method according to claim 9, wherein: The similarity in step A2 includes global similarity and fused similarity. The visual encoder of the CLIP model is used to obtain the frame embedding of each image frame contained in the image sequence. The text encoder of the CLIP model is used to obtain the word embedding of each word contained in the text, and the embedding of the start mark output by the text encoder is used as the global text feature of the candidate pair. In step B3, use the following formula to use uncertainty Revise the global similarity Obtain the final global similarity According to the following formula, using uncertainty Revised fusion similarity Get the final fusion similarity According to the following formula, the final global similarity is integrated Similarity to the final fusion Get the final similarity Among them, γ1 and γ2 are weight hyperparameters respectively.

Images