Text video cross-modal retrieval method and system based on two-stage neighborhood diffusion

Abstract

The invention relates to the field of cross-modal information retrieval, in particular to a text video cross-modal retrieval method and system based on two-stage neighborhood diffusion, and the method comprises the steps: carrying out the feature extraction of a to-be-retrieved text and a to-be-retrieved video, and obtaining the global feature vectors of the to-be-retrieved text and the to-be-retrieved video; calculating random neighborhoods corresponding to the text and the video based on the cosine similarity between the to-be-retrieved text and the to-be-retrieved video; performing random sampling on the global feature vector of the text / video to be retrieved in the corresponding random neighborhood to obtain random neighborhood text embedding and random neighborhood video embedding so as to perform embedding space modal alignment of the text and the video; tensor splicing is carried out on random neighborhood embedding of the to-be-retrieved text and the to-be-retrieved video, and cross-modal data distribution of the text and the video is optimized based on a pre-constructed dynamic agency attention diffusion network. Through random neighborhood modeling and a dynamic agency attention diffusion network, the cross-modal alignment precision and the model generalization ability are improved.

Description

Technical Field

[0001] The present application relates to the field of cross-modal information retrieval, and more specifically, to a text-video retrieval method and system based on two-stage neighborhood diffusion. Background Art

[0002] As a key research direction in the field of multimedia analysis and understanding, text-video cross-modal retrieval technology has the core task of establishing a bidirectional semantic mapping relationship between natural language descriptions and video content. Specifically, this task includes two main technical directions: 1) video clip retrieval based on text queries, that is, retrieving semantically matching spatiotemporal segments in the video library based on the natural language description input by the user (such as "a girl in red clothes flying a kite in the park"); 2) text description retrieval based on video content, that is, generating or retrieving text information that accurately describes the content of a given video clip. This technology is of great value in practical application scenarios such as intelligent video search, cross-media content recommendation, and video content review. However, due to the essential differences between video and text modalities in information representation and semantic expression, this task faces multiple technical challenges.

[0003] From a modality perspective, video data is essentially a continuous signal stream coupled in space and time. Its representational dimensions include static visual features in the spatial dimension (such as scene layout and target objects), dynamic motion features in the temporal dimension (such as action trajectories and event evolution), and multi-source feature streams (such as audio waveforms and subtitle text). Text, on the other hand, is a sequence of discrete symbols, and its semantic expression follows the spatial distribution of word vectors under grammatical constraints. This heterogeneity in representational dimensions leads to an essential misalignment between the two modalities in feature space, making cross-modal semantic alignment fundamentally difficult. While existing feature mapping methods based on attention mechanisms (such as the Transformer architecture) can establish coarse-grained associations, they still suffer from significant errors in fine-grained spatiotemporal alignment (such as matching specific action phases with corresponding verb phrases). From the perspective of information density, there are significant differences between video and text. Videos typically contain a large number of frames, and each frame may contain redundant information, such as static background or irrelevant action, resulting in visual information overload in video content. Especially in a continuous frame sequence, many frames repeatedly show similar scenes, and this redundant information may obscure key information in the video (such as the start of an action or important details). In contrast, text descriptions face the challenge of information sparsity. Typically, the text description of a video is relatively brief and cannot fully cover the temporal and spatial details of the video. In particular, it cannot clearly describe the visual features in the video, such as color, object position, or motion trajectory. This information asymmetry makes information transfer very difficult in cross-modal alignment of text and video.

[0004] Current mainstream methods suffer from the following technical limitations: First, fixed-point embedding strategies (such as single-point alignment based on cosine similarity) struggle to effectively represent the temporal semantic variations of videos, and the semantic expression range of short text is limited. For example, when the query text is "animals eating," existing models may incorrectly match videos depicting "humans eating" because text embeddings fail to effectively extend to relevant semantic contexts. Second, the differences in the embedding space distributions between text and video are not explicitly modeled. For example, videos often contain multi-entity scenes, while text descriptions often only involve a single subject, resulting in insufficient model performance in complex scenarios. Experiments show that existing methods perform worse in cross-dataset tests than on specific datasets. Furthermore, redundant information interferes with the model significantly. Although removing redundant frames can effectively improve retrieval accuracy, existing methods lack dynamic filtering mechanisms for core video frames. Furthermore, the performance of existing techniques in retrieval tasks for long videos (over several minutes) is significantly degraded, with retrieval accuracy much lower than that for short videos, highlighting the limitations of current techniques in handling long-span video content. Summary of the Invention

[0005] In view of this, the present application provides a text-video retrieval method and system based on two-stage neighborhood diffusion to solve the problems of insufficient text semantic coverage, video redundant information interference, and insufficient embedding space alignment in existing text-video cross-modal retrieval tasks.

[0006] This application provides the following technical solutions:

[0007] In a first aspect, the present application provides a text-video retrieval method based on two-stage neighborhood diffusion, comprising:

[0008] Perform feature extraction on the text to be retrieved and the video to be retrieved to obtain global feature vectors of the text to be retrieved and the video to be retrieved;

[0009] Calculate the random neighborhood corresponding to the text and video based on the cosine similarity between the text to be retrieved and the video to be retrieved;

[0010] Randomly sample the global feature vectors of the text / video to be retrieved in the corresponding random neighborhood to obtain random neighborhood text embedding and random neighborhood video embedding to align the embedding space modalities of text and video;

[0011] The random neighborhood embeddings of the text to be retrieved and the video to be retrieved are tensor-spliced, and the cross-modal data distribution of the text and video is optimized based on a pre-built dynamic proxy attention diffusion network; the pre-built dynamic proxy attention diffusion network includes a noise-aware projection module, a two-branch attention module, a deep convolution module and a denoising decoder.

[0012] In one possible implementation, feature extraction is performed on the text to be retrieved and the video to be retrieved, including:

[0013] Based on the pre-trained text encoder and video encoder, respectively obtain fine-grained feature representations of the text to be retrieved and the video to be retrieved;

[0014] The category labels of the fine-grained feature representations of the text to be retrieved and the video to be retrieved are extracted as global features to obtain global feature vectors of the text to be retrieved and the video to be retrieved.

[0015] In one possible implementation, the cosine similarity between the text to be retrieved and the video to be retrieved is expressed as:

[0016]

[0017]

[0018] Where, Represents the similarity between the global feature vector of the text to be retrieved and the j-th video frame, represents the average similarity between the i-th word and each video frame; M is the number of video frames contained in the video to be retrieved, and N is the number of words contained in the text to be retrieved; w i , f j ∈R d represent word-level and frame-level embeddings respectively; c t Represents the global feature vector of the text to be retrieved;

[0019] The step of calculating the random neighborhood corresponding to the text and the video based on the cosine similarity between the text to be retrieved and the video to be retrieved includes:

[0020] Based on the cosine similarity, a learnable linear layer is used Computing random neighborhoods for text and video and Expressed as:

[0021]

[0022] Where S t and S v are the neighborhood similarity tensors of text and video, respectively.

[0023] In one possible implementation, the global feature vector of the to-be-retrieved text / video is randomly sampled in the corresponding random neighborhood to obtain the random neighborhood text embedding and random neighborhood video embedding, which are expressed as:

[0024]

[0025] Where, t d is the random neighborhood text embedding, v d For random neighborhood video embedding; For random sampling, c t is the global feature vector of the text to be retrieved, c v is the global feature vector of the video to be retrieved; is a random neighborhood of the text, is a random neighborhood of the video.

[0026] One possible implementation method is to align the embedding space modalities of text and video based on random neighborhood text embedding and random neighborhood video embedding, including:

[0027] Embed the random neighborhood text into t d and random neighborhood video embedding v d Incorporating cross entropy loss In , the cross entropy loss is calculated and expressed as:

[0028]

[0029] Among them, the conditional probability p(v|t) is expressed as:

[0030]

[0031] Where the temperature hyperparameter τ is used to adjust the similarity in the embedding space, ψ t represents the text encoder, ψ v Represents a video encoder, represents the transpose of the text representation obtained by the text encoder, ψ v (v′) represents the video representation obtained by the video encoder, E (t，v) Common expected loss for text and video.

[0032] In one possible implementation, after performing tensor splicing on the random neighborhood embeddings of the text to be retrieved and the video to be retrieved to obtain the text embedding tensor and the video embedding tensor, the method includes:

[0033] Noise encoding is added to the text embedding tensor and video embedding tensor respectively, and query, key and value are generated through linear transformation;

[0034] Perform pooling operations on queries and keys, projecting them into a low-dimensional space to generate query proxies and key proxies;

[0035] Construct cross-modal data distribution, calculate the attention weights of query agents and key agents, and combine them with the candidate joint probability weight factor to perform query-agent alignment and agent-candidate alignment. Then perform deep convolution operations on the value features generated by linear transformation, and add the convolution results with the attention alignment results to extract diverse features.

[0036] The cross-modal data distribution is input into a denoising decoder, and the cross-modal data distribution of text and video is gradually generated through a reverse denoising process.

[0037] In a second aspect, the present application provides a text-video cross-modal retrieval system based on a two-stage neighborhood diffusion, comprising a random neighborhood module and a dynamic proxy attention diffusion module; the random neighborhood module comprises a text encoder, a video encoder, and a neighborhood similarity perception module;

[0038] The text encoder is used to extract features from the acquired text to be retrieved to obtain a global feature vector of the text to be retrieved;

[0039] The video encoder is used to extract features from the acquired video to be retrieved to obtain a global feature vector of the video to be retrieved;

[0040] The neighborhood similarity perception module is used to calculate the random neighborhoods corresponding to the text and video based on the cosine similarity between the text to be retrieved and the video to be retrieved; randomly sample the global feature vectors of the text to be retrieved / the video to be retrieved in the corresponding random neighborhoods to obtain random neighborhood text embedding and random neighborhood video embedding to perform embedding space modality alignment of the text and video;

[0041] The dynamic proxy attention diffusion module optimizes the cross-modal data distribution of text and video based on the text embedding tensor and video embedding tensor obtained by tensor splicing of random neighborhood embeddings of the to-be-retrieved text and video according to a pre-built dynamic proxy attention diffusion network; the dynamic proxy attention diffusion module includes a noise-aware projection module, a dual-branch attention module, a deep convolution module and a denoising decoder.

[0042] In one possible implementation, the noise-aware projection module is configured to dynamically generate queries, keys, and values ​​by adding a noise code Noise(k) and a cubic linear layer Linear;

[0043] The dual-branch attention module is used to achieve query-proxy alignment and proxy-candidate alignment respectively through two Softmax operations;

[0044] The deep convolution module is used to perform a deep convolution operation on the value features generated by the noise perception projection module through linear transformation, add the convolution result and the attention alignment result, and extract diverse features;

[0045] The denoising decoder includes a multi-layer perceptron MLp, which is used to perform feature encoding through a ReLU activation function and calculate and output cross-modal data distribution.

[0046] Compared with the existing technology, the technical solution provided by this application has the following beneficial effects:

[0047] Through random neighborhood modeling based on cosine similarity, feature regions that are highly correlated with the target modality are dynamically screened, and embedding vectors are generated using random sampling, effectively alleviating the semantic bias problem caused by modality differences in traditional methods and enhancing the robust alignment capability of the embedding space. Secondly, a dynamic proxy attention mechanism is introduced to decompose high-dimensional queries and candidate matching into two-stage alignments: query-proxy and proxy-candidate. The computational complexity is greatly reduced through low-dimensional proxy vectors, while the deep convolution module is combined to retain multi-granularity semantic features and avoid information loss. In addition, the diffusion optimization module gradually corrects the cross-modal data distribution prediction through the reverse denoising process, and combines the KL divergence loss to jointly optimize the noise modeling and the true distribution alignment, so that the model can still maintain stable retrieval performance in complex noise environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the DomainDiff cross-modal retrieval framework provided for this application.

[0049] Figure 2 This is a flowchart of a text video retrieval method based on two-stage neighborhood diffusion provided in Example 1 of the present application.

[0050] Figure 3 This is a flowchart of a text-video retrieval method based on two-stage neighborhood diffusion provided in Example 2 of the present application.

[0051] Figure 4 Schematic diagram of the random neighborhood embedding architecture provided in Example 2 of the present application.

[0052] Figure 5 A flowchart of a method for determining random neighborhoods corresponding to text and video provided in Example 2 of the present application.

[0053] Figure 6 Schematic diagram of the reverse denoising process of the DomainDiff cross-modal retrieval framework provided in Example 3 of the present application. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0055] To solve the problems mentioned in the background technology, this application provides a text video retrieval method based on two-stage neighborhood diffusion. Figure 1 As shown in , the first stage of the two-stage approach is to represent the latent neighborhood based on random sampling, and the second stage is to capture the statistical characteristics and latent dependencies of cross-modal data based on a dynamic proxy attention diffusion network. Specifically, in the first stage, the initial text-video embedding is performed by and ) are randomly sampled to generate neighborhood-specific embeddings (t d and v d ). These random embeddings are then concatenated to construct T d and V d In the second stage, these neighborhood embeddings are combined with the data distribution X k and k-level noise Together, they are input into a dynamic agent attention diffusion network (DAADN). Through iterative denoising, the network continuously optimizes the embedding, ultimately obtaining an optimized data distribution x0 for text-video alignment. Below, the specific implementation steps of the method provided by this application are described in detail with reference to specific examples.

[0056] Example 1

[0057] See also Figure 2 , is a flowchart of a text video retrieval method based on two-stage neighborhood diffusion provided in Example 1 of this application. Figure 2 As shown in , the specific implementation steps of the above method include:

[0058] Step 101: extract features from the text to be retrieved and the video to be retrieved to obtain global feature vectors of the text to be retrieved and the video to be retrieved.

[0059] Step 102: Calculate the random neighborhood corresponding to the text and the video based on the cosine similarity between the text to be retrieved and the video to be retrieved.

[0060] Step 103: Randomly sample the global feature vector of the text to be retrieved / video to be retrieved in the corresponding random neighborhood to obtain random neighborhood text embedding and random neighborhood video embedding to perform embedding space modality alignment of the text and video.

[0061] Step 104: Perform tensor splicing on the random neighborhood embeddings of the text to be retrieved and the video to be retrieved, and optimize the cross-modal data distribution of the text and video based on the pre-built dynamic proxy attention diffusion network.

[0062] Among them, such as Figure 1 As shown in , the dynamic agent attention diffusion network includes a noise-aware projection module, a two-branch attention module, a deep convolution module and a denoising decoder. The noise-aware projection module is used to dynamically generate queries, keys and values ​​by adding noise coding Noise(k) and a cubic linear layer Linear, thereby improving the robustness to noise interference. The two-branch attention module is used to respectively realize query-agent alignment and agent-candidate alignment through two Softmax operations, thereby reducing computational complexity and enhancing cross-modal semantic association. The deep convolution module (DWC) is used to perform a deep convolution operation on the value features generated by the noise-aware projection module through linear transformation, and add the convolution result to the attention alignment result to retain feature diversity and avoid semantic information loss. The denoising decoder is composed of a multi-layer perceptron MLp, whose linear layer uses the ReLU activation function for feature encoding, and then calculates and outputs the cross-modal data distribution through another linear layer.

[0063] Compared with the prior art, the technical solution provided in Example 1 of the present application has the following beneficial effects:

[0064] This application proposes the DomainDiff cross-modal retrieval framework, which realizes the joint refined alignment of text and video embedding spaces through a two-stage collaborative optimization mechanism - random neighborhood modeling and dynamic proxy attention diffusion network; it specifically solves the problems of text semantic sparsity and video information redundancy in cross-modal retrieval, and proposes a bimodal random neighborhood sampling strategy, that is, local semantic expansion of text embeddings to supplement contextual information, and key frame focusing of video embeddings to strengthen the core semantic expression; by constructing a dynamic proxy attention diffusion network, the progressive denoising idea of ​​the diffusion model is introduced into cross-modal joint distribution modeling, and the computational complexity is reduced through the proxy attention mechanism. At the same time, the diffusion optimization module is used to enhance the model's robustness to noise interference and improve the cross-scene generalization ability.

[0065] Example 2

[0066] See also Figure 3 , is a flowchart of a text video retrieval method based on two-stage neighborhood diffusion provided in Example 2 of this application. Figure 3 As shown in , the specific implementation steps of the above method include:

[0067] Step 201: Acquire multiple texts and videos to be retrieved.

[0068] In this embodiment of the present application, a corpus containing K text-video pairs is obtained. Each text t consists of N words w, and each video v contains M video frames f. The goal of text-video retrieval is to retrieve the most semantically relevant video v from the video set V (v∈V) given a text t to be retrieved.

[0069] To achieve this goal, such as Figure 4 As shown in , this application aims to learn a text encoder ψ t and a video encoder ψ v , mapping text and video into a shared embedding space, thus achieving cross-modal feature alignment.

[0070] Step 202: Based on the pre-trained text encoder ψ t and video encoder ψ v , respectively extract the fine-grained feature representations of the above-mentioned text to be retrieved and the video to be retrieved.

[0071] Specifically, given the text to be retrieved t and the video to be retrieved v, their fine-grained embedding can be expressed as:

[0072]

[0073] Among them, w i , f j ∈R d denote word-level and frame-level embeddings, respectively, d is the dimension of the feature space, N and M denote the number of words and video frames, respectively.

[0074] Step 203: By extracting the category label ([cIs]) as the global feature, the global feature vector c of the text to be retrieved and the video to be retrieved is obtained. t , c v ∈R d , expressed as:

[0075] c t =[cls] w c v =[cls] f ,

[0076] Step 204: The global feature vectors c of the text to be retrieved and the video to be retrieved are calculated in the corresponding neighborhood. t , c v ∈R d Perform random sampling to generate random neighborhood embeddings t representing random text and video embeddings d and v d .

[0077] Specifically, in order to solve the problem that text description is too concise and video data redundancy affects cross-modal alignment, the embodiment of the present application proposes a potential neighborhood representation strategy based on random sampling, such as Figure 4 As shown in Figure 2. This strategy projects text and video into random representations in the embedding stage to reduce the distance between positive samples. Using the reparameterization technique, for a given text-video global feature vector c t , c v ∈R d , introducing potential random neighbors and Random sampling is performed within these neighborhoods to obtain random neighborhood embeddings t representing random text and video embeddings d and v d , expressed as:

[0078]

[0079] Where, Used for random sampling. c t and c v is the global feature embedding of the text and video to be retrieved.

[0080] Then, the random neighborhood embedding t d and v d Used to calculate the cross entropy loss. However, there are some challenges in determining the range of the random neighborhood. On the one hand, if the random neighborhood range If is too small, it may lead to insufficient modal representation capability. On the other hand, if If the size is too large, inappropriate embedding points may be introduced, thereby reducing the retrieval performance. To solve this problem, this application proposes a neighborhood similarity awareness module (Domain Similarity Awareness Module, hereinafter referred to as DSAM). Figure 5 As shown in , in the above step 204, the method for determining the random neighborhood corresponding to the text and the video specifically further includes:

[0081] Step 2041: Calculate the fine-grained cosine similarity between the words in the text to be retrieved and the video frames of the video to be retrieved, expressed as:

[0082]

[0083] Where, Represents a sentence (text global feature vector c t ) and the similarity between the j-th frame of the video, It is worth noting that the video global feature vector c is not used in this embodiment of the application. v, because calculating similarity based on video frames is more suitable for word-level matching.

[0084] Step 2042: Based on the fine-grained cosine similarity between the words in the text to be retrieved and the video frames of the video to be retrieved, a learnable linear layer is used. Computing random neighborhoods for text and video and Expressed as:

[0085]

[0086] Among them, S t and S v are the neighborhood similarity tensors of text and video, respectively.

[0087] Furthermore, this application calculates the neighborhood range based on the learnable scalar θ, which is expressed as:

[0088]

[0089] In the formula, θ is broadcasted to d dimensions to adapt to the high-dimensional space. Specifically, as long as a point falls within the representation neighborhood, it can be considered a valid representation of the modality and used for similarity calculation.

[0090] Step 205: embed the random neighborhood into t d and v d Incorporating cross entropy loss The cross entropy loss is calculated in

[15] , and a more robust modality alignment is achieved through a dynamic sampling mechanism.

[0091] Among them, the corresponding cross entropy loss function is defined as:

[0092]

[0093] Among them, the conditional probability p(v|t) is defined as:

[0094]

[0095] The temperature hyperparameter τ is used to adjust the similarity in the embedding space and construct a normalized distribution on the candidate video set V. In the evaluation phase, the optimal embedding point is obtained through multiple sampling iterations to ensure the stability and accuracy of the representation.

[0096] Unlike traditional methods that directly use fixed-point embedding to explicitly adjust the distance between text-video embedding points in the cross-entropy loss, the dynamic random embedding strategy based on neighborhood range sampling proposed in this application enhances cross-modal representation capabilities by implicitly reducing the distance between text-video embedding points.

[0097] To overcome the limitations of concise text in capturing rich video semantics, this application uses random sampling to explore potential matching neighborhoods, thereby enhancing the semantic content of the original text embedding. Similarly, to mitigate the interference of redundant video information on the core semantic representation, random sampling is used to screen out frames or clips that are highly relevant to the text.

[0098] However, it should be noted that the random neighborhood learning method maximizes the conditional probability p(v|t; ψ t , ψ v )+p(t|v;ψ t , ψ v ) to optimize model parameters. A fundamental limitation of this approach is its inability to model the inherent distributions p(t) and p(v) of text and video data, particularly their underlying statistical characteristics and joint dependencies. This design paradigm inherently destroys the inherent structure of the original data distribution, resulting in the model's inability to effectively capture essential data patterns. Consequently, its ability to generalize across neighborhoods is significantly limited, making it difficult to adapt to unseen data.

[0099] Step 206: Perform tensor splicing on the random neighborhood embeddings of the text to be retrieved and the video to be retrieved, add noise coding to the obtained text embedding tensor and video embedding tensor respectively, and generate query, key and value through linear transformation.

[0100] In order to further optimize the text-video representation and solve the problem of ignoring the inherent distribution of data p(t) and p(v) in the first stage, the embodiment of the present application models the joint probability distribution p(v, t) from a generative perspective. Specifically, the present application proposes a Dynamic Agent Attention Diffusion Network (DAADN) to capture the statistical characteristics and potential dependencies of cross-modal data. The objective function is defined as:

[0101]

[0102] Where C represents the corpus of text-video pairs, and φ represents the parameters of the diffusion network. Unlike generation tasks, which prioritize diversity and fidelity, cross-modal retrieval requires precise modality alignment. This framework fuses a diffusion model with a dynamic proxy to bridge the gap between textual and visual distributions while preserving modality-specific features. This dual mechanism addresses the limitations of traditional methods and enables adaptive alignment through dynamic feature distribution.

[0103] Taking text-to-video retrieval as an example, v is obtained by random neighborhood modeling. d and t d Embed all text random neighborhoods in a set of B samples into t d Concatenate into text embedding tensor Td ∈R B×d , the random neighborhood embedding of the video is also spliced ​​into the video embedding tensor V d ∈R B×d .

[0104] like Figure 3 As shown in , based on the dynamic proxy diffusion framework constructed in this application, the query (Q), key (K), and value (V) required for proxy attention are generated through noise-aware projection, which are expressed as:

[0105]

[0106] Step 207: Perform a pooling operation on the query and key, and project them into a low-dimensional space to generate a query proxy and a key proxy.

[0107] Specifically, if Figure 1 As shown in , pooling is a downsampling operation, which converts Q t ∈R B×d Projection to low-dimensional space R b×d Similarly, A k By pooling key features K v Generate. In addition, the agent can be initialized as a static low-dimensional tensor. The agent (A) required to generate agent attention through noise-aware projection is expressed as:

[0108] A q =Pooling(Q t ),

[0109] A k =Pooling(K v ),

[0110] Among them, Noise(k) encodes the noise level k as a d-dimensional embedding, is the projection matrix, A q Dynamically generated by pooling query features.

[0111] Step 208: Construct a cross-modal data distribution, calculate the attention weights of the query agent and the key agent, combine the candidate joint probability weight factor, perform query-agent alignment and agent-candidate alignment, and perform deep convolution operation on the value features generated by linear transformation. Add the convolution result and the attention alignment result to extract diverse features.

[0112] In order to enhance the semantic alignment between query and candidate, this paper designs a dual-branch attention mechanism. By utilizing proxy tags, query-proxy and proxy-candidate alignment is achieved while improving computational efficiency. The complexity of proxy attention is O(Bbd), where B is the sample size and b<<B, which is significantly lower than the O(B 2d). Cross-modal data distribution The calculation is as follows:

[0113]

[0114] Among them, X k is the candidate joint probability weight factor, reflecting the joint probability of each candidate video under the previous noise level. Higher values ​​indicate higher confidence. By dynamically adjusting attention weights, the method provided in this embodiment enhances the response to high-confidence candidates. DWC(·) represents the deep convolution module, which is used to maintain feature diversity.

[0115] Step 209: Input the above cross-modal data distribution into a denoising decoder, and gradually generate the cross-modal data distribution of the text to be retrieved and the video to be retrieved through a reverse denoising process.

[0116] Specifically, the cross-modal data distribution It is considered as a high-level aligned distribution of text-video data and used as the input to the denoising decoder. The denoising decoder consists of a multi-layer perceptron (MLP) whose linear layer uses the ReLU activation function for feature encoding, followed by another linear layer to calculate the output distribution.

[0117] Similarly, in the video-to-text retrieval task, the projection of the video representation is used as the query and the projection of the text representation is used as the key and value in the attention module, and the output distribution is calculated in the same way.

[0118] In the diffusion process, it is expressed as Predicted data distribution Generate step by step. Where ∈ represents Gaussian noise in random sampling.

[0119] The loss function is designed to optimize noisy predictions and model the data distribution:

[0120]

[0121] Through the iterative process of noise injection and denoising, the joint distribution modeling can fully capture the latent semantic relationship between text-video pairs.

[0122] In addition, the generated random embedding t d and v dThe diffusion network's ability to model joint probability distributions is enhanced. Simultaneously, by modeling the joint probability distribution, the diffusion model inversely optimizes the semantic alignment of text and video features, thereby forming a collaborative optimization loop. Notably, the proposed dual-branch attention mechanism introduces proxy tags to construct a parallel computational architecture consisting of query-proxy alignment and proxy-candidate alignment: the former focuses on fine-grained cross-modal feature matching, while the latter models global relationships within the candidate set through proxy nodes. This architecture not only significantly reduces the quadratic computational complexity of the traditional cross-attention mechanism but also enhances the model's ability to capture long-tail semantic associations through the information routing function of the proxy nodes.

[0123] Example 3

[0124] Example 3 of the present application verifies the effectiveness of the text-video retrieval method based on two-stage neighborhood diffusion provided in the present application through visual analysis and case studies.

[0125] This application is evaluated in detail on five widely used standard datasets, including the MSRVTT dataset, the LSMDC dataset, the DiDeMo dataset, the ActivityNet dataset, and the VATEX dataset. These datasets cover a variety of video and text correspondences and provide rich data support.

[0126] The MSRVTT dataset contains 10,000 videos from YouTube, each accompanied by 20 manually annotated text descriptions. Based on the 1k-A split protocol, 9,000 videos are used for training and 1,000 for testing. This dataset is widely used in multimodal learning and is particularly valuable for video and text retrieval tasks.

[0127] The LSMDC dataset contains 118,081 video clips from 202 movies. The dataset is partitioned according to the [1] criteria and includes 1,000 test samples. This dataset is primarily used for multimodal tasks involving movie clips and their corresponding text descriptions, and is an important data source for research on video-text retrieval and cross-modal generation.

[0128] The DiDeMo dataset contains 10,642 video clips and 40,543 text descriptions. The data partitioning standard follows CLIP4clip and DiffusionRet. This dataset is particularly suitable for video description generation and video content retrieval tasks and is one of the classic datasets in cross-modal learning.

[0129] ActivityNet is a large-scale video dataset containing 20,000 YouTube videos covering a wide range of activity categories. In our experiments, we used the "val1" split, which consists of 10,009 training videos and 4,917 test videos. This dataset provides a rich data resource for video understanding and video retrieval tasks.

[0130] The VATEX dataset contains 34,991 video clips, each with a multi-dimensional text description. A key feature of this dataset is that each video is accompanied by a text description in a different language, making it multilingual and providing valuable data support for multilingual video retrieval and cross-lingual learning.

[0131] Several common retrieval evaluation indicators are used to measure the retrieval performance of the model, especially the effect in the video-text retrieval task. Specifically, the above-mentioned retrieval evaluation indicators include K-bit recall, median ranking and average ranking. Among them, the K-bit recall (R@K) is used to measure whether the correct answer appears in the first K returned results. The embodiment of the present application calculates the recall when K=1, 5 and 10 respectively to evaluate the performance of the model in different ranking positions. The median ranking (MdR) is used to represent the median ranking value of the correct answer in the retrieval results, which helps to measure the overall quality of the retrieval results. The average ranking (MnR) is used to calculate the average ranking of all correct answers to further evaluate the retrieval accuracy of the model.

[0132] The implementation of Example 3 of the present application is based on the CLIP framework. The present application designs a dual-backbone network architecture, in which ViT-B / 32 and ViT-B / 16 are used to process text and video features respectively and perform cross-modal alignment. The input is normalized as follows: the sequence length of the text input is set to 32, the frame sequence length of the video input is set to 12, and the spatial resolution of the video frame is 224×224. During the training process, a phased optimization strategy is adopted: Phase 1: Initialize the model using the contrastive learning method. Specifically, the learning rate of the random neighborhood module is set to 1e-5, the learning rate of the text / video encoder is 1e-6, the weight decay value is 0.2, and the Dropout value is 0.3. This stage trained for 5 cycles (batch size 128), using the Adam optimizer, and combined with the cosine annealing learning rate scheduler. Phase 2: Optimize the denoising network, which is trained based on the diffusion model. In this stage, the learning rate of the text / video encoder is set to 1e-7, the learning rate of other modules is set to 1e-3, and the temperature parameter The value of the retrieval function is set to 0.01. The second stage of optimization helps further improve the model's generative capabilities and denoising performance. All experiments were conducted on an NVIDIA A40 GPU platform. For text-to-video and video-to-text retrieval tasks, the candidate set is assumed to be known during the inference phase. During inference, the distance between video and text in the feature space and their joint probability distribution are simultaneously considered to improve retrieval accuracy.

[0133] To fully verify the superiority of DomainDiff over existing methods, we conduct an in-depth comparison with some of the latest methods in the field of text-video retrieval. These methods can be summarized into the following four categories: (1) Hybrid granularity methods based on CLIP pre-trained models, such as X-poo1, TS2-Net, UATVR, TMASS, EMCL-Net, UCoFiA, DiCoSA, HBI, etc. These methods mainly improve the performance of cross-modal retrieval by extracting features at different granularities; (2) Modal fusion methods combined with audio and subtitles, such as VAST, VALOR, vid-TLDR, etc. These methods enhance the matching degree of video and text by introducing audio information; (3) Two-stage hybrid training frameworks, such as UMT and DifFusionRet, etc., optimize the performance of the model through a two-stage training strategy; (4) Unified multimodal task large-scale models, such as UMT, VAST, VALOR, vid-TLDR, etc. These methods aim to optimize multiple tasks simultaneously through a large-scale multimodal learning framework. It is worth mentioning that this application is also compared with the GRAM method, which achieves cross-modal alignment by optimizing the Gramian determinant and improves performance by introducing additional modal information.

[0134] As can be seen in Tables 1 and 2, DomainDiff achieves significant performance improvements on all five benchmark datasets. In the text-to-video (T2V) retrieval task, experimental results show that DomainDiff achieves a 5.2% performance improvement over the state-of-the-art GRAM method on the MSRVTT dataset and a 7.4% improvement over the UMT-L method on the LSMDC dataset. While the GRAM method introduces more modal features and optimizes the Gramian determinant, and UMT-L utilizes the large-scale ViT-L / 16 architecture, DomainDiff still achieves breakthrough performance improvements thanks to its simple and efficient architecture. This performance improvement is primarily due to the synergy between its innovative random neighborhood embedding mechanism and the joint probability distribution modeling of the diffusion network. More notably, the video encoder based on ViT-B / 16 outperforms the ViT-B / 32 encoder on all datasets, demonstrating the critical importance of fine-grained features in improving model performance.

[0135] Table 1 compares DomainDiff with state-of-the-art methods on the MSRVTT dataset. “↑” indicates higher is better, and “↓” indicates lower is better. Bold indicates the best performance. “-” indicates: the result is not available.

[0136]

[0137] Table 2 compares the text-to-video retrieval performance of DomainDiff with the current state-of-the-art methods on other datasets.

[0138]

[0139] However, on the VATEX dataset, the GRAM method performs slightly better. This is because the GRAM method utilizes additional modal information, while this application avoids the introduction of additional models to save computational costs, so the results of the two cannot be directly compared. Other studies have shown that the accuracy difference between video-to-text (V2T) and text-to-video (T2V) retrieval tasks is small, indicating that for some short videos, the amount of effective information in video and text is limited, which limits the potential for semantic exploration, resulting in relatively limited performance improvements when sampling random neighborhoods.

[0140] In summary, DomainDiff achieves progressive refinement at the representation level through a two-stage joint optimization strategy: in the first stage, random neighborhood modeling provides a stable initial distribution for the diffusion network; in the second stage, the inverse denoising process dynamically optimizes the random neighborhood range through gradient backpropagation. This forms a closed-loop optimization mechanism.

[0141] In the design of phase 1, this embodiment proposes and compares three different neighborhood calculation schemes, as shown in Table 3.

[0142] Table 3 provides a discussion of the model design for neighborhood modeling. This example uses the Vi TB / 32 video encoder to conduct experiments on the MSRVTT dataset. The final performance comparison is based on the selection.

[0143] Table 3

[0144]

[0145] 1. This approach uses fixed parameters for calculation and therefore cannot be learned. The calculation process of this method is relatively simple, but due to its lack of flexibility, it cannot adapt to complex task requirements.

[0146] 2. The introduction of a learnable scalar (θ) allows the model to dynamically adjust parameters based on the specific needs of the task, improving its learnability. By training (θ), the model can adapt to different input data and improve its generalization ability.

[0147] 3. This approach uses a linear transformation matrix (W), providing stronger nonlinear modeling capabilities than the previous two approaches. By introducing linear transformations, the model can more effectively capture the complex relationships between input features, significantly improving performance, especially when dealing with high-dimensional features.

[0148] According to the experimental results, The performance of the scheme is the best, showing its strong nonlinear modeling ability. In addition, even with a single learnable scalar (θ), it also achieved quite strong performance, which shows that the random neighborhood module is very effective for The design sensitivity is low. Therefore, this application finally selected solution because it performs well in a variety of conditions.

[0149] For Phase 2: Proxy Token Initialization, this example compares two proxy token initialization strategies, as shown in Table 4. Table 4 provides a model discussion of proxy token design. Experiments were conducted using the ViT-B / 16 video encoder on the LSMDC and DiDeMo datasets, with the results focusing on the text-to-video task. The final performance comparison is based on the selection of dynamic proxy tokens.

[0150] Table 4

[0151]

[0152] (1) Static strategy: This strategy uses a fixed-dimensional zero tensor to initialize the proxy tokens, that is, the initial values ​​of all proxy tokens are zero and the dimensions are also fixed;

[0153] (2) Dynamic strategy: This strategy initializes the proxy token by generating a low-dimensional tensor based on dynamic downsampling of the query vector Q and the key vector K. In this way, the proxy token can more flexibly reflect the semantic characteristics of the input data, rather than simply relying on a fixed initial value. By comparing the experimental results of the two strategies, it can be found that the dynamic strategy performs better on most data sets and converges faster. This is because the dynamic strategy can adjust the representation of the proxy token in real time according to the characteristics of the input data, allowing the proxy token to capture more meaningful semantic information directly from Q and K. This enables the model to perform semantic alignment more effectively during training, thereby accelerating the convergence process and improving the final retrieval performance. On the contrary, due to the fixed initial value of the static strategy, the proxy token cannot fully utilize the potential information in the data in the early stage of training, which in turn affects the learning efficiency of the model. Taking all these factors into consideration, the dynamic initialization strategy was finally selected as the proxy token initialization method in this application.

[0154] Regarding training strategies, the present invention systematically compared various training strategies in ablation experiments on the text-to-video task on the MSRVTT dataset, as shown in Table 5(a). The experimental results show that a single contrastive learning strategy based on neighborhood embedding outperforms the diffusion learning method based purely on the denoising network, while the two-stage hybrid training strategy proposed in this application significantly improves performance. This verifies that an excellent representation encoder can effectively unleash the potential of the diffusion denoising network and proves that the two-stage hybrid training strategy successfully combines the advantages of the two methods, thereby improving the performance of the overall model.

[0155] Table 5(a)

[0156]

[0157]

[0158] represents the Kullback-Leibler divergence, Represents the cross entropy loss. “√” indicates that the training phase has been carried out, and “×” indicates that a failure occurred during the evaluation process.

[0159] Furthermore, the embodiment of the present application analyzes the impact of the number of diffusion steps on model performance, as shown in Table 5(b). Experimental results show that in the text-video retrieval task, 50 diffusion steps achieve the best performance, which is significantly lower than the 1000 steps commonly used in image generation tasks. The present application attributes this to the lower distribution complexity required for cross-modal alignment in the text-video retrieval task. In contrast, the image generation task needs to deal with complex pixel-level distributions. Fewer diffusion steps are sufficient to capture the required semantic information, indicating that the semantic space in this task is relatively simple and does not require a large number of diffusion steps to obtain key information.

[0160] Table 5(b)

[0161]

[0162] As shown in Table 5(c), this application studies the effect of the neighborhood sampling frequency F on the model performance during the inference phase. The experimental results show that when F increases from 5 to 15, the effect on t d and v d Multiple random samplings allow for better exploration of semantic representations, leading to improved performance. When F = 5, performance drops significantly, indicating that the sampling frequency is insufficient to fully explore the semantic space. Notably, performance stabilizes between F = 10 and F = 20. After striking a balance between performance and computational cost, the final model selected F = 15 as the optimal sampling frequency.

[0163] Table 5(c)

[0164] Frequency(F) R@1↑ R@5T R@10↑ MdR↓ MnR↓ 5 50.4 75.6 85.4 1.0 12.1 10 55.4 78.4 85.8 1.0 11.1 15 56.8 80.7 86.2 1.0 9.8 20 56.2 80.2 86.0 1.0 10.0

[0165] Figure 6 The framework's inverse denoising process is shown, and the dynamic changes in the cross-modal alignment between text and video during multiple denoising iterations are shown. As the denoising steps continue, the embedding distributions of text and video gradually converge. In particular, the distance between corresponding text-video embedding pairs decreases significantly, indicating that their positions in the semantic space are gradually approaching. This change reflects the continuous optimization of cross-modal alignment, which gradually eliminates the semantic differences between text and video. In addition, the neighborhood range ( and The uncertainty in the semantic space (represented by the size of the shape) also continues to shrink. As denoising iterations proceed, the uncertainty in the semantic space gradually decreases, indicating that the model has a more precise understanding of the semantic relationship between text and video. This process not only enhances the model's ability to capture the relationship between modalities, but also improves the accuracy of semantic alignment, ultimately leading to a consistent and high-level semantic representation between text and video. This gradual convergence trend effectively improves performance in cross-modal retrieval tasks, especially in scenarios requiring high semantic matching. The model can accurately capture the common features of different modalities, thereby achieving better performance in retrieval tasks.

[0166] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A text-video cross-modal retrieval method based on two-stage neighborhood diffusion, characterized by: include: Perform feature extraction on the text to be retrieved and the video to be retrieved to obtain global feature vectors of the text to be retrieved and the video to be retrieved; Calculate the random neighborhood corresponding to the text and video based on the cosine similarity between the text to be retrieved and the video to be retrieved; Randomly sample the global feature vectors of the text / video to be retrieved in the corresponding random neighborhood to obtain random neighborhood text embedding and random neighborhood video embedding to align the embedding space modalities of text and video; The random neighborhood embeddings of the text to be retrieved and the video to be retrieved are tensor-spliced, and the cross-modal data distribution of the text and video is optimized based on a pre-built dynamic proxy attention diffusion network; the pre-built dynamic proxy attention diffusion network includes a noise-aware projection module, a two-branch attention module, a deep convolution module and a denoising decoder.

2. The text-video cross-modal retrieval method based on two-stage neighborhood diffusion according to claim 1 is characterized in that: Perform feature extraction on the text and video to be retrieved, including: Based on the pre-trained text encoder and video encoder, respectively obtaining fine-grained feature representations of the to-be-retrieved text and the to-be-retrieved video; The category labels of the fine-grained feature representations of the text to be retrieved and the video to be retrieved are extracted as global features to obtain global feature vectors of the text to be retrieved and the video to be retrieved.

3. The text-video cross-modal retrieval method based on dual-stage neighborhood diffusion according to claim 1 is characterized in that: The cosine similarity between the text to be retrieved and the video to be retrieved is expressed as: Where, Represents the similarity between the global feature vector of the text to be retrieved and the j-th video frame, represents the average similarity between the i-th word and each video frame; M is the number of video frames contained in the video to be retrieved, and N is the number of words contained in the text to be retrieved; w i ,f j ∈R d represent word-level and frame-level embeddings respectively; c t Represents the global feature vector of the text to be retrieved; The step of calculating the random neighborhood corresponding to the text and the video based on the cosine similarity between the text to be retrieved and the video to be retrieved includes: Based on the cosine similarity, a learnable linear layer Θ is used to calculate random neighborhoods of text and video and Expressed as: Where S t and S v are the neighborhood similarity tensors of text and video, respectively.

4. The text-video cross-modal retrieval method based on two-stage neighborhood diffusion according to claim 1 is characterized in that: Randomly sample the global feature vector of the text / video to be retrieved in the corresponding random neighborhood to obtain the random neighborhood text embedding and random neighborhood video embedding, which are expressed as: Where, t d is the random neighborhood text embedding, v d For random neighborhood video embedding; For random sampling, c t is the global feature vector of the text to be retrieved, c v is the global feature vector of the video to be retrieved; is a random neighborhood of the text, is a random neighborhood of the video.

5. The text-video cross-modal retrieval method based on dual-stage neighborhood diffusion according to claim 1 is characterized in that: Based on random neighborhood text embedding and random neighborhood video embedding, the embedded space modality alignment of text and video is performed, including: Embed the random neighborhood text into t d and random neighborhood video embedding v d Incorporating cross entropy loss In , the cross entropy loss is calculated and expressed as: Among them, the conditional probability p(v|t) is expressed as: Where the temperature hyperparameter τ is used to adjust the similarity in the embedding space, ψ t represents the text encoder, ψ v Represents a video encoder, represents the transpose of the text representation obtained by the text encoder, ψ v (v′) represents the video representation obtained by the video encoder, E (t，v) Common expected loss for text and video.

6. The text-video cross-modal retrieval method based on dual-stage neighborhood diffusion according to claim 1, characterized in that: After performing tensor splicing on random neighborhood embeddings of the text to be retrieved and the video to be retrieved to obtain a text embedding tensor and a video embedding tensor, the method includes: Noise encoding is added to the text embedding tensor and video embedding tensor respectively, and query, key and value are generated through linear transformation; Perform pooling operations on queries and keys, projecting them into a low-dimensional space to generate query proxies and key proxies; Construct cross-modal data distribution, calculate the attention weights of query agents and key agents, and combine them with the candidate joint probability weight factor to perform query-agent alignment and agent-candidate alignment. Then perform deep convolution operations on the value features generated by linear transformation, and add the convolution results with the attention alignment results to extract diverse features. The cross-modal data distribution is input into a denoising decoder, and the cross-modal data distribution of text and video is gradually generated through a reverse denoising process.

7. A text-video cross-modal retrieval system based on two-stage neighborhood diffusion, characterized by: It includes a random neighborhood module and a dynamic agent attention diffusion module; the random neighborhood module includes a text encoder, a video encoder and a neighborhood similarity perception module; The text encoder is used to extract features from the acquired text to be retrieved to obtain a global feature vector of the text to be retrieved; The video encoder is used to extract features from the acquired video to be retrieved to obtain a global feature vector of the video to be retrieved; The neighborhood similarity perception module is used to calculate the random neighborhoods corresponding to the text and video based on the cosine similarity between the text to be retrieved and the video to be retrieved; randomly sample the global feature vectors of the text to be retrieved / the video to be retrieved in the corresponding random neighborhoods to obtain random neighborhood text embedding and random neighborhood video embedding to perform embedding space modality alignment of the text and video; The dynamic proxy attention diffusion module optimizes the cross-modal data distribution of text and video based on the text embedding tensor and video embedding tensor obtained by tensor splicing of random neighborhood embeddings of the to-be-retrieved text and the to-be-retrieved video according to a pre-built dynamic proxy attention diffusion network; the dynamic proxy attention diffusion module includes a noise-aware projection module, a dual-branch attention module, a deep convolution module, and a denoising decoder.

8. The text-video cross-modal retrieval system based on dual-stage neighborhood diffusion according to claim 7, characterized in that: The noise-aware projection module is used to dynamically generate queries, keys, and values ​​by adding noise encoding Noise(k) and a cubic linear layer Linear; The dual-branch attention module is used to achieve query-proxy alignment and proxy-candidate alignment respectively through two Softmax operations; The deep convolution module is used to perform a deep convolution operation on the value features generated by the noise perception projection module through linear transformation, add the convolution result and the attention alignment result, and extract diverse features; The denoising decoder includes a multi-layer perceptron (MLP) for encoding features through a ReLU activation function and calculating and outputting cross-modal data distribution.

Images