Text video pair similarity evaluation model construction method

Abstract

The invention relates to the field of information retrieval, and discloses a text video pair similarity evaluation model construction method, which comprises the following steps of: firstly, inputting a positive sample pair and constructing a negative sample pair; then, visual features of the sample pairs are obtained through a visual encoder, and text features of the sample pairs are obtained through a text encoder; then, respectively calculating the similarity of coarse, medium and fine granularity of the sample pair; wherein the coarse-grained similarity is used for carrying out overall similarity calculation on the video and the text; the medium-granularity similarity is used for carrying out frame-level similarity calculation on image frames and texts contained in the video; and the fine-grained similarity is used for performing factor-level similarity calculation on visual entities contained in the video and words contained in the text. In the retrieval process, more features can be introduced, the similarity of the text and the similarity of the video are compared from multiple granularities, the influence caused by semantic misalignment of the text and the vision can be reduced, and the retrieval performance is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval, and in particular to a method for constructing a text-video pair similarity assessment model. Background Art

[0002] In recent years, with the rise of multimedia platforms such as Tik Tok, YouTube, and Netflix, users have increasingly demanded more accuracy in finding video content based on text. Currently, there are roughly three types of text-video retrieval methods:

[0003] 1. Modal retrieval based on traditional manual design

[0004] Based on traditional text-video multimodal retrieval methods, such as keyword matching, retrieval is performed through video metadata tags; feature extraction, extracting low-level features such as color histograms and TF-IDF from videos and texts, and matching through feature vector similarity; multimodal fusion, using statistical techniques such as canonical correlation analysis, mapping text and video features to a common representation space.

[0005] However, based on the traditional text-video multimodal retrieval method, which relies on manual annotation, incomplete or inaccurate labels will affect the effect, and traditional feature extraction is difficult to capture complex semantic information, and fixed rules and features lack flexibility and generalization ability.

[0006] Therefore, although it has certain practicality in simple and resource-constrained scenarios, it is gradually replaced by deep learning and CLIP-based methods under the needs of complex semantic matching.

[0007] 2. Modal Retrieval Based on Deep Learning

[0008] Modal retrieval methods based on deep learning have significantly improved the retrieval effect. Such methods usually use an encoder with a two-stream network architecture to extract features from text and video respectively, and then match them in the fusion layer. For example, recurrent neural networks and convolutional neural networks are good at processing sequence and spatial features respectively. When used in combination, they can effectively capture the temporal dependencies and spatial features in videos and texts. Cross-modal attention mechanisms such as the Transformer model capture the contextual relationship between videos and texts through self-attention mechanisms. However, these methods require a large amount of data for training and have high requirements for computing resources.

[0009] Nevertheless, deep learning methods perform well in handling complex semantic matching tasks, significantly outperforming traditional methods.

[0010] 3. Modality Retrieval Based on Pre-trained CLIP

[0011] The CLIP-based method has made significant progress in text-video multimodal retrieval. The full name of the CLIP model is Contrastive Language-Image Pre-training, which is usually translated into Chinese as Contrastive Language-Image Pre-training Model. The CLIP model maps text and video frames to the same high-dimensional vector space through contrastive learning. By pre-training on large-scale datasets, CLIP learns rich semantic information, so that text and video content can be accurately matched in the same space. In text-video retrieval, CLIP can directly vectorize text and video frames and retrieve them by calculating the cosine similarity of the vectors. Due to the pre-training of CLIP on large-scale datasets, its feature expression ability and generalization ability are very strong, and it has achieved remarkable results in multimodal retrieval tasks.

[0012] Although CLIP requires a lot of computing resources during training, it performs well in practical applications, especially when dealing with complex semantic matching tasks, and its results are significantly better than traditional methods and some other deep learning methods.

[0013] As mentioned above, the current mainstream text-video retrieval technology, represented by CLIP, has also made significant progress and achievements in further research. Further research on CLIP can be roughly divided into two categories: one is to enhance the representation learning of text and visual features extracted by CLIP, and the other is the interactive alignment of text features and visual features.

[0014] The core of the first type of work is to learn the representation of features. For example, EM-Net uses the expectation maximization algorithm to compactly represent visual and text features, enhancing the semantic representation ability of text and visual features; T-Mass uses random text modeling and text regularization methods to extract valid frames and text, enhancing the semantic similarity between text and video.

[0015] The core of the second type of work is the interaction between text features and visual features. For example, MSIA introduces a multi-level semantic interaction model, uses adaptive frames and text-guided attention mechanisms, reduces video redundancy and enhances modal interaction. HBI uses a new type of multi-cooperative game theory for coarse-grained and fine-grained interactions involving actions, scenes, entities and other information in visual information.

[0016] In summary, the current mainstream text-video retrieval technology, represented by CLIP, focuses on the research of enhanced feature representation learning and granular alignment strategies; however, these methods often overlook an important issue: due to the significant semantic differences between text and visual modalities, there are difficulties in enhanced representation learning and interactive alignment. Specifically, images have richer details and content, while manually labeled text information is usually relatively brief. This semantic asymmetry leads to difficulties in content matching between text features and visual features. This core difference directly affects feature-enhanced representation learning and subsequent granular alignment, making it difficult to accurately match features of different modalities in the same space, thus affecting the retrieval effect. Summary of the invention

[0017] The technical problem to be solved by the present invention is to propose a method for constructing a text-video pair similarity assessment model, which can reduce the impact caused by the semantic asymmetry between text features and visual features and significantly improve the retrieval performance.

[0018] The technical solution adopted by the present invention to solve the above technical problems is:

[0019] A method for constructing a text-video pair similarity assessment model, wherein the text-video alignment model comprises a visual encoder, a text encoder and an alignment model, wherein the visual encoder and the text encoder are pre-trained models; the training of the text-video alignment model comprises:

[0020] A1. Extracting text-video pairs from the original data set, wherein the text-video pairs include a video and its corresponding text; obtaining an image sequence from the video of each text-video pair, wherein the image sequence is composed of a group of image frames; the image sequence obtained by sampling and the corresponding text of the video constitute a training sample pair of the text-video pair; and the training sample pairs of each text-video pair in the original data set constitute a training set;

[0021] A2. Input training sample pairs as positive sample pairs for this round of training; construct negative sample pairs of the positive sample pairs, wherein the negative sample pairs include text negative sample pairs and / or video negative sample pairs. The text negative sample pairs are sample pairs consisting of the text contained in the positive sample pairs and the image sequences contained in other training sample pairs other than the positive sample pairs. The video negative sample pairs are sample pairs consisting of the image sequences contained in the positive sample pairs and the text contained in other training sample pairs other than the positive sample pairs.

[0022] For the image sequence contained in each sample pair, the visual encoder is input respectively to obtain the visual features of each image frame contained therein; for the text contained in each sample pair, the text encoder is input respectively to obtain its text features;

[0023] A3. For each sample pair, input its text features and visual features into the alignment model to calculate its coarse-grained similarity, medium-grained similarity, and fine-grained similarity.

[0024] The coarse-grained similarity is obtained by calculating the overall similarity between the image sequence and the text of the sample pair based on the input text features and the visual features;

[0025] The medium-grained similarity is obtained by performing frame-level similarity calculation on the image frames and texts contained in the image sequence of the sample pair based on the input text features and visual features;

[0026] The fine-grained similarity is obtained by calculating the factor-level similarity between the visual entities contained in the image sequence and the words contained in the text based on the input text features and visual features;

[0027] A4. Using the coarse-grained similarity, medium-grained similarity, and fine-grained similarity of the positive sample pair and its negative sample pair, calculate the coarse-grained feature alignment loss, medium-grained feature alignment loss, and fine-grained feature alignment loss, and use this to calculate the total loss of this round of training; based on the total loss, update the parameters of the alignment model and fine-tune the parameters of the visual encoder and text encoder;

[0028] A5. Loop through steps A2 to A4 until the training end condition is met, and obtain a trained alignment model.

[0029] Furthermore, the visual encoder and the text encoder are respectively the visual encoder and the text encoder of the pre-trained CLIP model;

[0030] In step A1, an image sequence is obtained from the video of each text-video pair by random sampling or average sampling;

[0031] In step A2, at least two training sample pairs are input as positive sample pairs for this round of training; the negative sample pairs include text negative sample pairs and video negative sample pairs; for each positive sample pair, the image sequence contained in it and the text of other positive sample pairs input are used to form the video negative sample of the positive sample pair, and the text contained in it and the image sequence of other positive sample pairs input are used to form the text negative sample pair of the positive sample pair.

[0032] Furthermore, the alignment model further includes a feature compression module; for the text features and visual features input to the alignment model, the alignment model first compresses the input visual features using the feature compression module, and then calculates the coarse-grained similarity, the medium-grained similarity and the fine-grained similarity; the compression of the input visual features includes:

[0033] The visual features of the input alignment model are used as the original visual features v o , calculate and obtain the redundant part it contains as the redundant visual feature v r ; Remove the original visual feature v according to the following formula o Redundant visual features v in r , get the compressed visual feature v c , and use it as input to calculate coarse-grained similarity, medium-grained similarity, and fine-grained similarity;

[0034] v c =v o -v r

[0035] The redundant visual features v obtained by calculation r , should meet the following requirements:

[0036]

[0037] Among them, s(·) represents the similarity function, v o represents the original visual features of the input alignment model, t is the text features of the input alignment model, and v r To represent the original visual feature v o The redundant visual features of the redundant part in v c To remove the visual feature v o Visual features after the redundant part in the image; Min means minimization and Max means maximization.

[0038] Furthermore, we define the similarity-aware compression factor C and use v r =C or v r =C·ε, calculate and obtain redundant visual features v r , where ε is a random factor; the similarity-aware compression factor C is calculated in any of the following ways:

[0039] Method 1: C = Expand (S)

[0040] Method 2: C = Expand(exp(γS))

[0041] Method 3:

[0042] Method 4: C = Expand(exp(MLP(S)))

[0043] Where S = [s 1 ,s 2 ,…,s i …,s F ],s i=s(t,fi), MLP(S)=LN(Rule(LN(S))); s(·) represents the similarity function, fi represents the visual features of the i-th image frame in the image sequence, s i represents the similarity between the i-th image frame in the image sequence and the text feature t, F is the number of image frames contained in the image sequence; C is the similarity perception compression factor; MLP is a fully connected network, LN is a linear layer in the fully connected network, Rule is a Rule activation function, exp is an exponential function, and Expand represents the similarity between v r The dimension of is expanded to make it equal to v o The expansion function is consistent with the dimension of .

[0044] Furthermore, the coarse-grained similarity is obtained by performing overall similarity calculation on the image sequence and text of the sample pair based on the input text features and visual features, including:

[0045] Firstly, based on the similarity between the visual features of each image frame in the image sequence of the sample pair and its text features, the weights of each image frame in the image sequence of the sample pair are constructed; then, the visual features of each image frame in the sample pair are aggregated in the weighted sum manner to obtain the video coarse-grained representation of the sample pair; then, the similarity between the video coarse-grained representation of the sample pair and its text features is calculated as the coarse-grained similarity of the sample pair;

[0046] The medium-grained similarity is obtained by performing frame-level similarity calculation on the image frames and texts contained in the image sequence of the sample pair based on the input text features and visual features, including:

[0047] First, the query is constructed with the text features of the sample pair, and the key and value are constructed with the visual features of the sample pair. The cross-attention mechanism is used to obtain the embedded representation of each image frame contained in the image sequence of the sample pair. Then, based on the embedded representation of each image frame, the video granular representation of the sample pair is obtained. After that, the similarity between the video granular representation of the sample pair and its text features is calculated as the medium-granular similarity of the sample pair.

[0048] The fine-grained similarity is obtained by calculating the factor-level similarity between the visual entities contained in the image sequence and the words contained in the text based on the input text features and visual features, including:

[0049] Firstly, the text features of the sample pair are decomposed into K text factors, and the coarse-grained video representation of the input sample pair is decomposed into K video factors, and K video-text factor pairs are formed, where K is the number of words contained in the text of the input sample pair; then, the similarity between the text factors and the video factors contained in each video-text factor pair is calculated respectively; then, based on the similarity between the text factors and the video factors of each video-text factor, the fine-grained similarity of the sample pair is obtained.

[0050] Furthermore, the calculation of the coarse-grained representation of the video includes:

[0051] First, according to the following formula, based on the similarity between the visual features of each image frame contained in the image sequence of the sample pair and its text features, the weight a of each image frame contained in the image sequence of the sample pair is constructed: i :

[0052]

[0053] Then, according to the following formula, the visual features of each image frame contained in the sample pair are aggregated in a weighted sum manner to obtain the video coarse-grained representation v of the sample pair: cg :

[0054]

[0055] Afterwards, the video coarse-grained representation v of the sample pair is calculated according to the following formula: cg The similarity between its text feature t is taken as the coarse-grained similarity s of the sample pair cg :

[0056]

[0057] Among them, f i and f j They represent the visual features of the ith and jth image frames contained in the image sequence respectively, τ is a hyperparameter, F represents the number of image frames contained in the image sequence; the superscript T represents matrix transpose, exp is the exponential function, and |·| represents the modulo operation.

[0058] Furthermore, in the calculation of the medium-grained similarity, the query is constructed with the text features of the sample pair, the key and value are constructed with the visual features of the sample pair, and the cross-attention mechanism is used to obtain the embedded representation of each image frame contained in the image sequence of the sample pair:

[0059]

[0060] Q t =LN(t T )W Q

[0061] K v =LN(v)W K

[0062] V v =LN(v c )

[0063] Among them, W Q , W K and W V Construct query representation Q t , Key Characterization K v The sum value represents V v The transformation matrix of p K v The feature dimension of ; softmax represents the softmax function, the superscript T represents the matrix transpose, and LN represents the linear layer;

[0064] According to the following formula, the granularity representation v in the video of the sample pair is calculated mg The similarity between its text feature t is taken as the medium-grained similarity s of the sample pair mg :

[0065]

[0066] Among them, |·| represents the modulo operation;

[0067] In the calculation of the medium-grained similarity, the video medium-grained representation v of the sample pair is obtained based on the embedded representation of each image frame according to the following formula: mg :

[0068] v mg =LN(r(v|t))+FC(r(v|t))) T

[0069] r(v|t)=LN(Attention(Q t ,K v ,V v )W 1 )

[0070] in, represents the embedded representation of each image frame contained in the image sequence, W 1 is the learnable weight, LN is the linear layer, and FC is the fully connected layer.

[0071] Furthermore, the calculation of the fine-grained similarity includes:

[0072] First, the text features of the sample pair are decomposed into K text factors fw according to the following formula: k , decompose the video coarse-grained representation of the input sample pair into K video factors fek , and form K video-text factor pairs:

[0073] f k =W k t

[0074] f k =W k v cg

[0075] Among them, v cg is the coarse-grained representation of the video, t is the text feature, W k is the kth learnable decomposition factor, subscript k is the sequence number and k = 1, 2, ..., K, K is the number of words contained in the text of the sample pair;

[0076] Then, the similarity between the text factor and the video factor contained in each video-text factor pair is calculated according to the following formula;

[0077] s=(FW) T FE

[0078] Where superscript T represents matrix transpose, FE = [fe 1 ,fe 2 ,…,fe k …,fe K ] is the video factor matrix, FW=[fw 1 ,fw 2 ,…,fw k …,fw K ] is the text factor matrix.

[0079] Furthermore, in the calculation of the fine-grained similarity, the fine-grained similarity s of the sample pair is obtained based on the similarity between the text factor and the video factor of each video text factor according to the following formula: fg :

[0080] s fg =s·g

[0081] g = MLP(cat[FW,FE])

[0082] MLP(cat[FW,FE])=LN(Rule(LN(cat[FW,FE])))

[0083] Among them, g is the similarity confidence, MLP is the fully connected network, LN is the linear layer in the fully connected network, Rule is the Rule activation function, and cat is the concatenation function.

[0084] Furthermore, the negative sample pairs include text negative sample pairs and video negative sample pairs;

[0085] Calculate the text-to-video alignment loss as follows:

[0086]

[0087] Calculate the video-to-text alignment loss as follows:

[0088]

[0089] The alignment loss is calculated as follows:

[0090]

[0091] Where M represents the number of video negative samples plus one, where plus one represents the positive sample corresponding to the video negative sample; N represents the number of text negative samples plus one, where plus one represents the positive sample corresponding to the text negative sample; B is the number of positive sample pairs in this round of training, e represents the natural exponent, λ is a hyperparameter representing the scaling factor, and s(·) represents the similarity function;

[0092] The total loss for:

[0093]

[0094] Among them, α and β are weight hyperparameters, represents the coarse-grained feature alignment loss, which is to represent the coarse-grained representation of the video v cg and text feature t as input, obtained by calculating the alignment loss; represents the mid-granularity feature alignment loss, which is to represent the mid-granularity representation v in the video mg and text feature t as input, obtained by calculating the alignment loss; represents the fine-grained feature alignment loss, which is the video factor matrix FE = [fe 1 ,fe 2 ,…,fe k …,fe K ] and the text factor matrix FW = [fw 1 ,fw 2 ,…,fw k …,fw K ] as input, obtained by calculating the alignment loss; fe k is the kth video factor, fw k is the kth text factor.

[0095] The beneficial effects of the present invention are as follows: the model of the present invention firstly calculates the overall similarity between the image sequence and the text of the sample pair; then, calculates the frame-level similarity between the image frame and the text contained in the image sequence of the sample pair; and then, calculates the factor-level similarity between the visual entities contained in the image sequence and the words contained in the text. Therefore, during retrieval, more feature information can be introduced through multi-level similarity, and the similarity between text and video can be compared from multiple granularities, which can reduce the impact caused by the semantic asymmetry between text features and visual features, and significantly improve the retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 A schematic diagram showing the interference of video semantic information on modality retrieval;

[0097] Figure 2 A schematic diagram of a retrieval process based on the text-video alignment model of the present invention;

[0098] Figure 3 This is a visualization result of the retrieval based on the embodiment of the present invention. DETAILED DESCRIPTION

[0099] The present invention aims to propose a method for constructing a text-video pair similarity assessment model. The constructed text-video alignment model is used to compare the similarity between video and text, and can be used for text-based video retrieval, as well as video-based text retrieval. Firstly, based on the interactive alignment of the entire video and the entire sentence, the interaction of coarse-grained information is realized; secondly, the cross-attention mechanism is used to directly obtain the video frame with the most representative information, and the interaction of medium-grained information is realized by aligning the frame with the entire sentence, as a supplement to the coarse-grained information; thirdly, the coarse-grained information is refined to obtain the alignment of visual entities and text words, and the interaction of fine-grained information is realized. The present invention introduces more feature information through hierarchical feature alignment, and compares the similarity of text and video from multiple dimensions, which can reduce the impact caused by the semantic inequality between text features and visual features, and significantly improve the retrieval performance.

[0100] Secondly, the modality retrieval task is based on the similarity between modalities, but the richness and diversity of video semantic information interferes with modality retrieval. Figure 1As shown, the words "tutor", "microphone" and so on in the visual information fail to match the manually labeled text information. This weak matching relationship is often because the text information cannot fully cover the complete visual information. At the same time, the visual information itself is diverse, and this diversity often presents redundancy in modal retrieval. Therefore, the present invention further introduces a feature compression module, which removes redundant visual information through a visual semantic compression method, allowing the text to interact with the effective visual information, greatly enhancing the representation and flexibility of the visual information, and providing support for hierarchical granular alignment.

[0101] Example

[0102] The present embodiment provides a method for constructing a text-video pair similarity assessment model, wherein the text-video alignment model includes a visual encoder, a text encoder and an alignment model, wherein the visual encoder and the text encoder are pre-trained models, which can adopt any existing video encoding model and text encoding model. In the present embodiment, the visual encoder and the text encoder are respectively the visual encoder and the text encoder of the pre-trained CLIP model.

[0103] The training of the text-video alignment model includes:

[0104] S1. Data preparation

[0105] The purpose of this step is to construct training sample pairs that meet the subsequent model input requirements based on the original text-video pairs. Specifically, the text-video pairs of the original data set are extracted, and the text-video pairs include videos and their corresponding texts; image sequences are obtained from the videos of each text-video pair, and the image sequences are composed of a group of image frames; the image sequences obtained by sampling and the corresponding text of the video constitute the training sample pairs of the text-video pair; and the training sample pairs of each text-video pair in the original data set constitute the training set.

[0106] It is difficult for existing models to process videos directly. Therefore, it is necessary to obtain an image sequence from the video of each text-video pair. The image sequence may include all image frames in the video, or may be obtained from the video of each text-video pair by random sampling or average sampling. In this embodiment, average sampling is adopted.

[0107] In an exemplary scheme, the MSRVTT dataset is used. First, obtain text-video pairs from the MSRVTT dataset, such as the video video9770.mp4 and the corresponding text "A person is connecting something to system". From the video, sample by average sampling to obtain F image frames, and record the video ID at the same time, so that paired training sample pairs are obtained, such as data{'video_ID':video_ID,'video':imgs,'text':caption}. For the convenience of description, the image sequence of image frames imgs containing F frames is denoted as V, and the text description caption related to the image is denoted as T.

[0108] S2. Construct positive and negative sample pairs and extract features

[0109] The ultimate goal of text and video retrieval is to compare the similarity between several texts and visuals to obtain a similarity matrix. The goal is to make the values ​​of the main diagonal elements of the similarity matrix larger and the values ​​of the other elements smaller. Therefore, the commonly used training loss function is the symmetric cross entropy function, which requires the construction of positive and negative sample pairs of text and video.

[0110] Therefore, in this step, the training sample pairs are input as the positive sample pairs for this round of training; negative sample pairs of the positive sample pairs are constructed, and the negative sample pairs include text negative sample pairs and / or video negative sample pairs. The text negative sample pairs are sample pairs consisting of the text contained in the positive sample pairs and the image sequences contained in other training sample pairs other than the positive sample pairs; the video negative sample pairs are sample pairs consisting of the image sequences contained in the positive sample pairs and the text contained in other training sample pairs other than the positive sample pairs.

[0111] The above text negative sample pairs and video negative sample pairs can respectively calculate the loss of text-to-video retrieval and video-to-text retrieval. Therefore, one can be selected according to actual needs and computing resources. For example, if only text-to-video retrieval is performed, only text negative samples can be used. In this embodiment, in order to obtain better results and realize two-way retrieval, the negative sample pairs include text negative sample pairs and video negative sample pairs.

[0112] To construct negative sample pairs, a text library and a video library can be constructed from the text and image sequences of the training set respectively. Then, a video negative sample pair can be constructed based on the video of the positive sample and the text randomly obtained from the text library, and a text negative sample can be constructed based on the text of the positive sample and the image sequence randomly obtained from the video library.

[0113] However, for the convenience of calculation, it is recommended that, in this step, at least two training sample pairs be input as positive sample pairs for this round of training; the negative sample pairs include text negative sample pairs and video negative sample pairs; for each positive sample pair, the image sequence contained therein and the text of other positive sample pairs inputted are used to form the video negative sample of the positive sample pair, and the text contained therein and the image sequence of other positive sample pairs inputted are used to form the text negative sample pair of the positive sample pair.

[0114] At this time, based on the text and image sequences of each positive sample, negative samples are cross-constructed, without introducing sample resources other than the text and image sequences contained in the positive sample. Therefore, feature extraction can be performed before or after the negative sample is constructed. In this embodiment, Batch_size=32.

[0115] Feature extraction is to extract rich text-visual semantic information from text and image sequences as input for subsequent feature alignment. For the image sequence contained in each sample pair, the visual encoder is input separately to obtain the visual features of each image frame it contains; for the text contained in each sample pair, the text encoder is used separately to obtain its text features.

[0116] In an exemplary scheme, based on the input image sequence V and text T, V is injected into the visual encoder φ of CLIP v Get the visual feature v in Similarly, the text T is injected into the text encoder φ of CLIP t Get text features Among them, B is the batch size of the input training sample pairs in this step, F is the number of image frames contained in the image sequence, and D is the feature dimension. is the field of real numbers. The formula is as follows:

[0117] t=φ t (T),v=φ v (V)

[0118] S3. Hierarchical feature alignment

[0119] In this step, for each sample pair, its text features and visual features are respectively input into the alignment model to calculate its coarse-grained similarity, medium-grained similarity and fine-grained similarity. The coarse-grained similarity is obtained by calculating the overall similarity between the image sequence and the text of the sample pair based on the input text features and visual features. The medium-grained similarity is obtained by calculating the frame-level similarity between the image frames and the text contained in the image sequence of the sample pair based on the input text features and visual features. The fine-grained similarity is obtained by calculating the factor-level similarity between the visual entities contained in the image sequence and the words contained in the text based on the input text features and visual features.

[0120] However, considering that the modal retrieval task is based on the similarity between modalities, the richness and diversity of video semantic information interferes with the retrieval of modalities. Therefore, in this step, before performing hierarchical alignment, a visual semantic compression method is introduced to remove redundant visual information, so that the text interacts with effective visual information. This method greatly enhances the representation and flexibility of visual information, and provides support for subsequent coarse-grained and fine-grained hierarchical granularity alignment. Further, the alignment model also includes a feature compression module; for the text features and visual features input to the alignment model, the alignment model first uses the feature compression module to compress the input visual features, and then calculates the coarse-grained similarity, medium-grained similarity and fine-grained similarity.

[0121] In summary, this step includes:

[0122] S31. Compressed visual features

[0123] Compress the input visual features, including:

[0124] The visual feature of the input alignment model, that is, the visual feature v calculated in step S2, is used as the original visual feature v o , calculate and obtain the redundant part it contains as the redundant visual feature v r ; Remove the original visual feature v according to the following formula o Redundant visual features v in r , get the compressed visual feature v c , and use it as input to calculate coarse-grained similarity, medium-grained similarity, and fine-grained similarity;

[0125] v c =v o -v r

[0126] The redundant visual features v obtained by calculation r , should meet the following requirements:

[0127]

[0128] Among them, s(·) represents the similarity function, v o represents the original visual features of the input alignment model, t is the text features of the input alignment model, and v r To represent the original visual feature v o The redundant visual features of the redundant part in v c To remove the visual feature v o Visual features after the redundant part in the image; Min means minimization and Max means maximization.

[0129] The above calculation indicates that for the redundant visual feature v r The goal of representation learning is to reduce the redundant visual features v r Similarity with text feature t, while maintaining the original visual feature v o And the compressed visual features v c has a greater similarity with the text feature t, that is, s(v r ,t) is as small as possible, s(v o ,t) and s(v c ,t) as large as possible, and s(v c ,t) is greater than s(v o ,t). Redundant visual features v r The calculation of , can adopt any scheme based on the above conditions, for example: by defining the similarity-aware compression factor C, and taking v r =C or v r =C·ε, calculate and obtain redundant visual features v r , where ε is a random factor.

[0130] The similarity-aware compression factor C may be calculated in any of the following ways:

[0131] Method 1: C = Expand (S)

[0132] Method 2: C = Expand(exp(γS))

[0133] Method 3:

[0134] Method 4: C = Expand(exp(MLP(S)))

[0135] Where S = [s 1 ,s 2 ,…,s i …,s F ],s i =s(t,f i ), MLP(S)=LN(Rule(LN(S))); s(·) represents the similarity function, f i represents the visual features of the i-th image frame in the image sequence, s i represents the similarity between the i-th image frame in the image sequence and the text feature t, F is the number of image frames contained in the image sequence; C is the similarity perception compression factor; MLP is a fully connected network, LN is a linear layer in the fully connected network, Rule is a Rule activation function, exp is an exponential function, and Expand represents the similarity between v r The dimension of is expanded to make it equal to v o The expansion function is consistent with the dimension of .

[0136] In this embodiment, the redundant visual feature v is calculated according to the following formula: r :

[0137] v r =C·ε

[0138] C = Expand(exp(MLP(S)))

[0139] MLP(S)=LN(Rule(LN(S)))

[0140] S=[s 1 ,s 2 ,…,s i …,s F ]

[0141] s i =s(t,f i )

[0142] Among them, s(·) represents the cosine similarity function, f i represents the visual features of the i-th image frame in the image sequence, s i represents the similarity between the i-th image frame in the image sequence and the text feature t, F is the number of image frames contained in the image sequence; C is the similarity perception compression factor, ε is a random factor that satisfies the normal distribution; MLP is a fully connected network, LN is a linear layer in the fully connected network, Rule is the Rule activation function, exp is an exponential function, and Expand represents the activation of v r The dimension of is expanded to make it equal to v o The expansion function is consistent with the dimension of .

[0143] In this embodiment, first, the cosine similarity between the text features and the visual features is calculated to obtain the similarity matrix Next, the fully connected network is used to map the redundant visual features of the similarity matrix to obtain a similarity perception compression factor C. At the same time, a random distribution that satisfies the normal distribution is introduced. The redundant visual features v r Characterized by C·ε.

[0144] S32. Calculate coarse-grained similarity

[0145] The coarse-grained similarity is obtained by coarse-grained alignment between the entire video and the sentence, that is, based on the input text features and visual features, the overall similarity of the image sequence and the text of the sample pair is calculated. Therefore, it is necessary to first aggregate the image frames contained in the image sequence to obtain the overall visual features of the video. The aggregation method can adopt any existing method, such as summation, averaging, weighted sum, etc.

[0146] Among them, the recommended method is to use weighted sum to aggregate based on text conditions, including:

[0147] Firstly, based on the similarity between the visual features of each image frame contained in the image sequence of the sample pair and its text features, the weights of each image frame contained in the image sequence of the sample pair are constructed; then, the visual features of each image frame contained in the sample pair are aggregated by weighted sum to obtain the coarse-grained video representation of the sample pair; then, the similarity between the coarse-grained video representation of the sample pair and its text features is calculated as the coarse-grained similarity of the sample pair.

[0148] Specifically, in this embodiment, the calculation of the coarse-grained representation of the video includes:

[0149] First, according to the following formula, based on the similarity between the visual features of each image frame contained in the image sequence of the sample pair and its text features, the weight a of each image frame contained in the image sequence of the sample pair is constructed: i :

[0150]

[0151] Then, according to the following formula, the visual features of each image frame contained in the sample pair are aggregated in a weighted sum manner to obtain the coarse-grained representation of the video of the sample pair:

[0152]

[0153] Afterwards, the video coarse-grained representation v of the sample pair is calculated according to the following formula: cg The similarity between its text feature t is taken as the coarse-grained similarity sc of the sample pair g :

[0154]

[0155] Among them, f i and f j They represent the visual features of the ith and jth image frames contained in the image sequence respectively, τ is a hyperparameter, F represents the number of image frames contained in the image sequence; the superscript T represents matrix transpose, exp is the exponential function, and |·| represents the modulo operation.

[0156] S33, Calculation of medium-sized similarity

[0157] Coarse-grained alignment usually lacks learnable parameters, and will include invalid frames and miss valid frames. Therefore, medium-grained similarity is introduced as a supplement. The medium-grained similarity is obtained by calculating the frame-level similarity of the image frames and texts contained in the image sequence of the sample pair based on the input text features and visual features. Its essence is to obtain the video frames with the most representative information through frame-level similarity comparison, reduce the impact of invalid frames, and increase the contribution of valid frames. It can use the attention of each frame and text as a weight or screening indicator, or directly use the key frames of the selected video.

[0158] Among them, the recommended one is to use the cross-attention mechanism, including:

[0159] Firstly, the query is constructed with the text features of the sample pair, and the key and value are constructed with the visual features of the sample pair. The cross-attention mechanism is used to obtain the embedded representation of each image frame contained in the image sequence of the sample pair. Then, based on the embedded representation of each image frame, the medium-granularity representation of the video of the sample pair is obtained. After that, the similarity between the medium-granularity representation of the video of the sample pair and its text features is calculated as the medium-granularity similarity of the sample pair.

[0160] Specifically, in this embodiment, in the calculation of the medium-granularity similarity, the query is constructed with the text features of the sample pair, the key and value are constructed with the visual features of the sample pair, and the cross-attention mechanism is used to obtain the embedded representation of each image frame contained in the image sequence of the sample pair:

[0161]

[0162] Q t =LN(t T )W Q

[0163] K v =LN(v)W K

[0164] V v =LN(v c )W V

[0165] Among them, W Q , W K and W V Construct query representation Q t , Key Characterization K v The sum value represents V v The transformation matrix of p K v The characteristic dimension of is the scaling factor of attention; softmax represents the softmax function, superscript T represents matrix transpose, and LN represents the linear layer.

[0166] Based on the embedded representation of each image frame, we can simply use the summation method to obtain the granularity representation v of the sample pair in the video mg In this embodiment, in order to obtain the final valid frame to achieve alignment with the text, first, the embedding representation of the image frame is represented by the following formula, and the weight is applied The aggregated video embedding r(v|t) based on the text feature t is obtained, and the embedded representation of the image frame is embedded into the joint space related to the text feature:

[0167] r(v|t)=LN(Attention(Q t ,K v ,V v )W 1 )

[0168] in, represents the embedded representation of each image frame contained in the image sequence, W 1 is the learnable weight and LN is the linear layer.

[0169] Then, we further use the full connection layer and residual connection method according to the following formula to obtain the final video granularity representation v mg :

[0170] v mg =LN(r(v|t))+FC(r(v|t))) T

[0171] Among them, LN is the linear layer and FC is the fully connected layer.

[0172] Finally, the granularity representation v in the video of the sample pair is calculated according to the following formula: mg The similarity between its text feature t is taken as the medium-grained similarity s of the sample pair mg :

[0173]

[0174] Here, |·| represents a modulo operation.

[0175] S34. Calculate fine-grained similarity

[0176] In order to fully mine richer semantic information, such as detailed actions of characters, scene information, etc., fine-grained similarity is further introduced. The fine-grained similarity is obtained by calculating the factor-level similarity between the visual entities contained in the image sequence and the words contained in the text based on the input text features and visual features.

[0177] The features of words contained in a text can be obtained by decomposing the text features, or by using words through a pre-trained language model. The features of visual entities contained in an image sequence can also be extracted through existing visual models, for example, by obtaining fine-grained features from image patches, and then obtaining the features of visual entities through clustering.

[0178] In order to simplify the complexity of the calculation and obtain the advantage of processing time while ensuring the granularity, the recommended methods include:

[0179] Firstly, the text features of the sample pair are decomposed into K text factors, and the coarse-grained video representation of the input sample pair is decomposed into K video factors, and K video-text factor pairs are formed, where K is the number of words contained in the text of the input sample pair; then, the similarity between the text factors and the video factors contained in each video-text factor pair is calculated respectively; then, based on the similarity between the text factors and the video factors of each video-text factor, the fine-grained similarity of the sample pair is obtained.

[0180] Specifically, in this embodiment, the calculation of the fine-grained similarity includes:

[0181] First, the text features of the sample pair are decomposed into K text factors fw according to the following formula: k , decompose the video coarse-grained representation of the input sample pair into K video factors fe k , and form K video-text factor pairs:

[0182] f k =W k t

[0183] f k =W k v cg

[0184] Among them, v cg is the coarse-grained representation of the video, t is the text feature, W k is the kth learnable decomposition factor, subscript k is the sequence number and k = 1, 2, ..., K, K is the number of words contained in the text of the sample pair;

[0185] Then, the similarity between the text factor and the video factor contained in each video-text factor pair is calculated according to the following formula;

[0186] s=(FW) T FE

[0187] Where superscript T represents matrix transpose, FE = [fe 1 ,fe 2 ,…,fe k…,fe K ] is the video factor matrix, FW=[fw 1 ,fw 2 ,…,fw k …,fw K ] is the text factor matrix.

[0188] Based on the similarity between the text factor and the video factor of each video text factor, the fine-grained similarity of the sample pair can be obtained by averaging, summing, etc. In this embodiment, the confidence g is introduced, and the similarity is adopted according to the following formula: With confidence Through an adaptive pooling operation, the final fine-grained similarity is obtained based on the similarity between the text factor and the video factor of each video text factor.

[0189] s fg =s·g

[0190] g = MLP(cat[FW,FE])

[0191] MLP(cat[FW,FE])=LN(Rule(LN(cat[FW,FE])))

[0192] Among them, g is the similarity confidence, MLP is the fully connected network, LN is the linear layer in the fully connected network, Rule is the Rule activation function, and cat is the concatenation function.

[0193] S4. Parameter update

[0194] For the design of training objectives, the three-granularity alignment losses can be directly combined. That is, the coarse-grained similarity, medium-grained similarity, and fine-grained similarity of the positive sample pair and its negative sample pair are used to calculate the coarse-grained feature alignment loss, medium-grained feature alignment loss, and fine-grained feature alignment loss, and the total loss of this round of training is calculated based on this; based on the total loss, the parameters of the alignment model are updated, and the parameters of the visual encoder and text encoder are fine-tuned.

[0195] Specifically, the negative sample pairs include text negative sample pairs and video negative sample pairs;

[0196] Calculate the text-to-video alignment loss as follows:

[0197]

[0198] Calculate the video-to-text alignment loss as follows:

[0199]

[0200] The alignment loss is calculated as follows:

[0201]

[0202] Wherein, M represents the number of video negative samples plus one, where plus one represents the positive sample corresponding to the video negative sample; N represents the number of text negative samples plus one, where plus one represents the positive sample corresponding to the text negative sample; B is the number of positive sample pairs in this round of training, e represents the natural exponent, λ is a hyperparameter representing the scaling factor, and s(·) represents the similarity function. In this implementation example, the similarity function s(·) is cosine similarity.

[0203] The total loss for:

[0204]

[0205] Among them, α and β are weight hyperparameters, represents the coarse-grained feature alignment loss, which is to represent the coarse-grained representation of the video v cg and text feature t as input, obtained by calculating the alignment loss; represents the mid-granularity feature alignment loss, which is to represent the mid-granularity representation v in the video mg and text feature t as input, obtained by calculating the alignment loss; represents the fine-grained feature alignment loss, which is the video factor matrix FE = [fe 1 ,fe 2 ,…,fe k …,fe K ] and the text factor matrix FW = [fw 1 ,fw 2 ,…,fw k …,fw K ] as input, obtained by alignment loss calculation; fe k is the kth video factor, fw k is the kth text factor.

[0206] In this embodiment, negative samples are constructed by cross-constructing the text and image sequences of each positive sample, and therefore, M=N=B.

[0207] S5. Loop through steps S2 to S4 until the training end condition is met, and obtain a trained alignment model.

[0208] After the text-video alignment model is trained, it can be used to perform retrieval tasks, such as Figure 1 As shown, the text-to-video retrieval method comprises the following steps:

[0209] B1. Input the search text, and construct search sample pairs based on the search text and the videos contained in the video library;

[0210] B2. Use the visual encoder and text encoder of the CLIP model to obtain the visual features and text features of each retrieval sample pair;

[0211] B3. Using the trained alignment model, obtain the coarse-grained similarity, medium-grained similarity, and fine-grained similarity of each retrieval sample pair;

[0212] B4. Sorting the similarity based on the sum of the coarse-grained similarity, medium-grained similarity and fine-grained similarity of each retrieval sample pair, and outputting the L videos with the greatest similarity as the retrieval results of the retrieval text.

[0213] The video to text retrieval method includes the following steps:

[0214] B1. Input the search video and construct search sample pairs based on the search video and the texts contained in the text library;

[0215] B2. Use the visual encoder and text encoder of the CLIP model to obtain the visual features and text features of each retrieval sample pair;

[0216] B3. Using the trained alignment model, obtain the coarse-grained similarity, medium-grained similarity, and fine-grained similarity of each retrieval sample pair;

[0217] B4. Sorting the similarity based on the sum of the coarse-grained similarity, medium-grained similarity and fine-grained similarity of each retrieval sample pair, and outputting the L texts with the greatest similarity as the retrieval results of the retrieval video.

[0218] Experimental verification:

[0219] In order to verify the effect of the present invention, the inventors compared the scheme of the above embodiment with the existing methods on six benchmark text-video retrieval datasets, including MSRVTT, MSVD, ActivityNet, Charades, DiDeMo and VATEX.

[0220] Specifically, the MSRVTT dataset contains 10,000 videos, each with 20 text descriptions, and the video length is approximately 10 to 30 seconds. Following the MSRVTT-test-1k division, 9,000 videos are used for training and 1,000 videos are used for testing. The MSVD dataset contains 1,970 videos, each with approximately 40 text descriptions, and the video length is approximately 10 to 25 seconds. Following the official division, 1,200 videos are used for training and 670 videos are used for testing. The ActivityNet dataset contains 20,000 videos, with a video length of approximately 180 seconds. All sentence descriptions of the video are connected into a query. In terms of division, 10,009 videos are used for training and 4,917 videos are used for testing. The Charades dataset contains 9,848 video clips, totaling approximately 38.8 hours, with an average length of 30 seconds per clip and corresponding text descriptions. The DiDeMo dataset contains 10,000 randomly selected 25 to 30 second unedited videos from YFCC100M, annotated with 40,000 text descriptions. The VATEX dataset is a new large-scale multilingual video description dataset containing more than 41,250 videos and 825,000 subtitles in English and Chinese.

[0221] The backbone models ViT-B / 32 and ViT-B / 16 of CLIP are used for feature extraction, with a latent space dimension of D=512, a weight decay of 0.2, and a dropout rate of 0.3. The batch size is set to Batch_size=32, and the number of training rounds is Epoch=5. The initial learning rate is kept at 1e-5, and the module for feature alignment fine-tunes the CLIP model at a rate of 1e-6. The training process uses the AdamW optimizer with a cosine schedule and a coefficient τ of 0.1. 12 frames are uniformly sampled from the video and resized to 224×224 for all datasets. The experiments are conducted on an A800 GPU with parameters set to α=1e-3, β=1e-1, and K=8 for decomposing the number of visual entities and text words.

[0222] The test indicators include R@k, MdR, and MnR.

[0223] R@k is an evaluation metric in the retrieval system, which is used to measure the proportion of correct targets contained in the first k retrieval results. It reflects the retrieval ability of the model to different degrees. Specifically, R@1 represents the proportion of correct targets found in the first retrieval result, R@5 represents the proportion of correct targets found in the first 5 retrieval results, and R@10 represents the proportion of correct targets found in the first 10 retrieval results. A higher R@k value indicates that the model has better precision and recall within the corresponding k retrieval results.

[0224] MdR is one of the indicators for evaluating the performance of a retrieval system, which indicates the median ranking of the correct results in the retrieval result list. Specifically, it measures where the correct results usually appear in the retrieval results in a set of queries. The lower the MdR, the better the retrieval effect of the model, because it means that the correct results generally appear in a higher position. Compared with the average ranking MnR, MdR is less sensitive to extreme values ​​of abnormally high or low rankings, and can more accurately reflect the typical performance of the retrieval system. It is an effective indicator for evaluating the stability and reliability of the system.

[0225] MnR is an indicator to measure the performance of the retrieval system, which represents the average ranking position of the correct results in all queries. It reflects the overall retrieval effect of the system by calculating the ranking of the correct results in each query and taking the average. A lower MnR value indicates a better system performance, because it means that the correct results usually appear in a higher position. MnR averages all rankings, so it can fully reflect the performance of the system under various queries, but it is also susceptible to extreme ranking values. Overall, MnR provides a comprehensive assessment of the accuracy and effectiveness of the retrieval system.

[0226] The experimental results are shown as follows, where Table 1 shows the performance of various solutions on the MSRVTT dataset, Table 2 shows the performance of various solutions on the DiDeMo dataset, Table 3 shows the performance of various solutions on the ActivityNet and MSVD datasets, and Table 4 shows the performance of various solutions on the Charades and VATEX datasets, where Ours is the solution of an embodiment of the present invention, and Publications is the journal or conference where the comparison solution model is published.

[0227] Table 1 Performance of each scheme on the MSRVTT dataset

[0228]

[0229] Table 2 Performance of various solutions on the DiDeMo dataset

[0230]

[0231] Table 3 Performance of various solutions on ActivityNet and MSVD datasets

[0232]

[0233] Table 4 Performance of various schemes on Charades and VATEX datasets

[0234]

[0235] Table 5 shows the performance of the scheme of the embodiment of the present invention using different compression factors C on the MSRVTT dataset, where:

[0236] The first group is without compression, directly using v o The performance of subsequent operations, where w / o C means without C;

[0237] The second group directly adopts the S method, that is: v r =C,C=Expand(S);

[0238] The third group uses S plus coefficient γ, that is: v r = C, C = Expand(exp(γS));

[0239] The fourth group is the average method, that is:

[0240] The fifth and sixth groups are both MLP methods, that is: v r =C·ε,C=Expand(exp(MLP(S))); wherein, the ε of the fifth group obeys the normal distribution and is denoted as N, and the ε of the sixth group obeys the uniform distribution and is denoted as U.

[0241] Table 5 shows the performance of the scheme of the embodiment of the present invention using different compression factors C on the MSRVTT dataset

[0242]

[0243] Figure 3 This is a visualization of the retrieval results. In the figure, Query is the given retrieval text, the image is the top five query results ranked by similarity, and the number is the similarity value. From the results, we can see that Rank 1, that is, the query result with the greatest similarity, is the correct retrieval result.

[0244] Finally, it should be noted that the above embodiments are only preferred implementations and are not intended to limit the present invention. It should be pointed out that for those skilled in the art, several modifications, equivalent replacements, improvements, etc. can be made without departing from the scope of the present invention and the scope of protection of the claims, and all of these should be included in the protection scope of the present invention.

Claims

1. A method for constructing a text-video pair similarity assessment model, characterized in that: The text-video alignment model includes a visual encoder, a text encoder and an alignment model, wherein the visual encoder and the text encoder are pre-trained models; The training of the text-video alignment model includes: A1. Extracting text-video pairs from the original data set, wherein the text-video pairs include a video and its corresponding text; obtaining an image sequence from the video of each text-video pair, wherein the image sequence is composed of a group of image frames; the image sequence obtained by sampling and the corresponding text of the video constitute a training sample pair of the text-video pair; and the training sample pairs of each text-video pair in the original data set constitute a training set; A2. Input training sample pairs as positive sample pairs for this round of training; construct negative sample pairs of the positive sample pairs, wherein the negative sample pairs include text negative sample pairs and / or video negative sample pairs. The text negative sample pairs are sample pairs consisting of the text contained in the positive sample pairs and the image sequences contained in other training sample pairs other than the positive sample pairs. The video negative sample pairs are sample pairs consisting of the image sequences contained in the positive sample pairs and the text contained in other training sample pairs other than the positive sample pairs. For the image sequence contained in each sample pair, the visual encoder is input respectively to obtain the visual features of each image frame contained therein; for the text contained in each sample pair, the text encoder is input respectively to obtain its text features; A3. For each sample pair, input its text features and visual features into the alignment model to calculate its coarse-grained similarity, medium-grained similarity, and fine-grained similarity. The coarse-grained similarity is obtained by calculating the overall similarity between the image sequence and the text of the sample pair based on the input text features and the visual features; The medium-grained similarity is obtained by performing frame-level similarity calculation on the image frames and texts contained in the image sequence of the sample pair based on the input text features and visual features; The fine-grained similarity is obtained by calculating the factor-level similarity between the visual entities contained in the image sequence and the words contained in the text based on the input text features and visual features; A4. Using the coarse-grained similarity, medium-grained similarity, and fine-grained similarity of the positive sample pair and its negative sample pair, calculate the coarse-grained feature alignment loss, medium-grained feature alignment loss, and fine-grained feature alignment loss, and use this to calculate the total loss of this round of training; based on the total loss, update the parameters of the alignment model and fine-tune the parameters of the visual encoder and text encoder; A5. Loop through steps A2 to A4 until the training end condition is met, and obtain a trained alignment model.

2. A method for constructing a text-video pair similarity assessment model as claimed in claim 1, characterized in that: The visual encoder and text encoder are respectively the visual encoder and text encoder of the pre-trained CLIP model; In step A1, an image sequence is obtained from the video of each text-video pair by random sampling or average sampling; In step A2, at least two training sample pairs are input as positive sample pairs for this round of training; the negative sample pairs include text negative sample pairs and video negative sample pairs; for each positive sample pair, the image sequence contained in it and the text of other positive sample pairs input are used to form the video negative sample of the positive sample pair, and the text contained in it and the image sequence of other positive sample pairs input are used to form the text negative sample pair of the positive sample pair.

3. The method for constructing a text-video pair similarity assessment model according to claim 1, characterized in that: The alignment model further includes a feature compression module; for the text features and visual features input into the alignment model, the alignment model first compresses the input visual features using the feature compression module, and then calculates the coarse-grained similarity, the medium-grained similarity and the fine-grained similarity; the compression of the input visual features includes: The visual features of the input alignment model are used as the original visual features v o , calculate and obtain the redundant part it contains as the redundant visual feature v r ; Remove the original visual feature v according to the following formula o Redundant visual features v in r , get the compressed visual feature v c , and use it as input to calculate coarse-grained similarity, medium-grained similarity, and fine-grained similarity; v c =v o -v r The redundant visual features v obtained by calculation r , should meet the following requirements: Among them, s(·) represents the similarity function, v o represents the original visual features of the input alignment model, t is the text features of the input alignment model, and v r To represent the original visual feature v o The redundant visual features of the redundant part in v c To remove the visual feature v o Visual features after the redundant part in the image; Min means minimization and Max means maximization.

4. A method for constructing a text-video pair similarity assessment model as claimed in claim 3, characterized in that: Define the similarity-aware compression factor C and use v r =C or v r =C·ε, calculate and obtain redundant visual features v r , where ε is a random factor; the similarity-aware compression factor C is calculated in any of the following ways: Method 1: C = Expand (S) Method 2: C = Expand(exp(γS)) Method 3: Method 4: C = Expand(exp(MLP(S))) Where S = [s1, s2, …, s i …,s i ],s i =s(t,f i ), MLP(S)=LN(Rule(LN(S))); s(·) represents the similarity function, f i represents the visual features of the i-th image frame in the image sequence, s i represents the similarity between the i-th image frame in the image sequence and the text feature t, F is the number of image frames contained in the image sequence; C is the similarity perception compression factor; MLP is a fully connected network, LN is a linear layer in the fully connected network, Rule is a Rule activation function, exp is an exponential function, and Expand represents the similarity between v r The dimension of is expanded to make it equal to v o The expansion function is consistent with the dimension of .

5. A method for constructing a text-video pair similarity assessment model according to any one of claims 1 to 4, characterized in that: The coarse-grained similarity is obtained by performing overall similarity calculation on the image sequence and text of the sample pair based on the input text features and visual features, including: Firstly, based on the similarity between the visual features of each image frame in the image sequence of the sample pair and its text features, the weights of each image frame in the image sequence of the sample pair are constructed; then, the visual features of each image frame in the sample pair are aggregated in the weighted sum manner to obtain the video coarse-grained representation of the sample pair; then, the similarity between the video coarse-grained representation of the sample pair and its text features is calculated as the coarse-grained similarity of the sample pair; The medium-grained similarity is obtained by performing frame-level similarity calculation on the image frames and texts contained in the image sequence of the sample pair based on the input text features and visual features, including: First, the query is constructed with the text features of the sample pair, and the key and value are constructed with the visual features of the sample pair. The cross-attention mechanism is used to obtain the embedded representation of each image frame contained in the image sequence of the sample pair. Then, based on the embedded representation of each image frame, the video granular representation of the sample pair is obtained. After that, the similarity between the video granular representation of the sample pair and its text features is calculated as the medium-granular similarity of the sample pair. The fine-grained similarity is obtained by calculating the factor-level similarity between the visual entities contained in the image sequence and the words contained in the text based on the input text features and visual features, including: Firstly, the text features of the sample pair are decomposed into K text factors, and the coarse-grained video representation of the input sample pair is decomposed into K video factors, and K video-text factor pairs are formed, where K is the number of words contained in the text of the input sample pair; then, the similarity between the text factors and the video factors contained in each video-text factor pair is calculated respectively; then, based on the similarity between the text factors and the video factors of each video-text factor, the fine-grained similarity of the sample pair is obtained.

6. A method for constructing a text-video pair similarity assessment model as described in claim 5, characterized in that: The calculation of the coarse-grained representation of the video includes: First, according to the following formula, based on the similarity between the visual features of each image frame contained in the image sequence of the sample pair and its text features, the weight a of each image frame contained in the image sequence of the sample pair is constructed: i : Then, according to the following formula, the visual features of each image frame contained in the sample pair are aggregated in a weighted sum manner to obtain the video coarse-grained representation v of the sample pair: cg : Afterwards, the video coarse-grained representation v of the sample pair is calculated according to the following formula: cg The similarity between its text feature t is taken as the coarse-grained similarity s of the sample pair cg : Among them, f i and f j They represent the visual features of the ith and jth image frames contained in the image sequence respectively, τ is a hyperparameter, F represents the number of image frames contained in the image sequence; the superscript T represents matrix transpose, exp is the exponential function, and |·| represents the modulo operation.

7. The method for constructing a text-video pair similarity assessment model according to claim 5, characterized in that: In the calculation of the medium-grained similarity, the query is constructed with the text features of the sample pair, the key and value are constructed with the visual features of the sample pair, and the cross-attention mechanism is used to obtain the embedded representation of each image frame contained in the image sequence of the sample pair: Q t =LN(t T )W Q K v =LN(v)W K V v =LN(v c )W V Among them, W Q , W K and W V Construct query representation Q t , Key Characterization K v The sum value represents V v The transformation matrix of p K v The feature dimension of ; softmax represents the softmax function, the superscript T represents the matrix transpose, and LN represents the linear layer; According to the following formula, the granularity representation v in the video of the sample pair is calculated mg The similarity between its text feature t is taken as the medium-grained similarity s of the sample pair mg : Among them, |·| represents the modulo operation; In the calculation of the medium-grained similarity, the video medium-grained representation v of the sample pair is obtained based on the embedded representation of each image frame according to the following formula: mg : in mg =LN(r(v|t))+FC(r(v|t))) T r(v|t)=LN(Attention(Q t ,K v ,V v )W1) in, It represents the embedded representation of each image frame contained in the image sequence, W1 is the learnable weight, LN is the linear layer, and FC represents the fully connected layer.

8. The method for constructing a text-video pair similarity assessment model according to claim 5, characterized in that: The calculation of the fine-grained similarity includes: First, the text features of the sample pair are decomposed into K text factors fw according to the following formula: k , decompose the video coarse-grained representation of the input sample pair into K video factors fe k , and form K video-text factor pairs: fw k =W k t fe k =W k v cg Among them, v cg is the coarse-grained representation of the video, t is the text feature, W k is the kth learnable decomposition factor, subscript k is the sequence number and k = 1, 2, ..., K, K is the number of words contained in the text of the sample pair; Then, the similarity between the text factor and the video factor contained in each video-text factor pair is calculated according to the following formula; s=(FW) T FE Where superscript T represents matrix transpose, FE = [fe1, fe2, …, fe k …,fe K ] is the video factor matrix, FW=[fw1,fw2,…,fw k …,fw K ] is the text factor matrix.

9. A method for constructing a text-video pair similarity assessment model as claimed in claim 8, characterized in that: In the calculation of the fine-grained similarity, the fine-grained similarity s of the sample pair is obtained based on the similarity between the text factor and the video factor of each video text factor according to the following formula: fg : s fg =s·g g = MLP(cat[FW,FE]) MLP(cat[FW,FE])=LN(Rule(LN(cat[FW,FE]))) Among them, g is the similarity confidence, MLP is the fully connected network, LN is the linear layer in the fully connected network, Rule is the Rule activation function, and cat is the concatenation function.

10. A method for constructing a text-video pair similarity assessment model as described in claim 8, characterized in that: The negative sample pairs include text negative sample pairs and video negative sample pairs; Calculate the text-to-video alignment loss as follows: Calculate the video-to-text alignment loss as follows: The alignment loss is calculated as follows: Where M represents the number of video negative samples plus one, where plus one represents the positive sample corresponding to the video negative sample; N represents the number of text negative samples plus one, where plus one represents the positive sample corresponding to the text negative sample; B is the number of positive sample pairs in this round of training, e represents the natural exponent, λ is a hyperparameter representing the scaling factor, and s(·) represents the similarity function; The total loss for: Among them, α and β are weight hyperparameters, represents the coarse-grained feature alignment loss, which is to represent the coarse-grained representation of the video v cg and text feature t as input, obtained by calculating the alignment loss; represents the mid-granularity feature alignment loss, which is to represent the mid-granularity representation v in the video mg and text feature t as input, obtained by calculating the alignment loss; represents the fine-grained feature alignment loss, which is the video factor matrix FE = [fe1, fe2, …, fe k …,fe K ] and the text factor matrix FW = [fw1,fw2,…,fw k …,fw K ] as input, obtained by alignment loss calculation; fe k is the kth video factor, fw k is the kth text factor.

Images