Cross-modal video clip retrieval method and device based on noise distribution learning, equipment, medium and program product

Abstract

The invention provides a cross-modal video clip retrieval method and device based on noise distribution learning, equipment, a medium and a program product, and relates to the technical field of video processing. The method comprises the following steps: acquiring a to-be-retrieved video clip set; inputting a to-be-retrieved video clip set into the video retrieval model, and outputting a similarity score between the video clip and the text clip; and comparing the similarity scores with a preset score threshold value, sorting the similarity scores greater than or equal to the preset score threshold value, and taking a preset number of video clips with high similarity scores as video clip retrieval results. The embodiment of the invention is used for solving the defects of high calculation cost and relatively large limitation of robustness in the prior art, realizing accurate recommendation of interested video clips for the user through the video retrieval model, enhancing the robustness of the video retrieval model for cross-modal alignment through noise distribution learning, and remarkably reducing the consumption of calculation resources.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method, apparatus, device, medium, and program product for cross-modal video segment retrieval based on noise distribution learning. Background Technology

[0002] With the rise and widespread adoption of short video platforms, more and more users are recording and sharing their daily lives through videos, generating and disseminating massive amounts of multimodal data, including videos, images, and text, online every day. This vast and complex volume of multimodal data has driven the rapid development of the short video industry. However, traditional single-modal retrieval technologies struggle to adapt to this trend, not only limiting users' ability to obtain comprehensive and flexible information but also significantly increasing data management costs for businesses due to low processing efficiency.

[0003] In related technologies, early video moment retrieval (VMR) primarily employed candidate segment-based methods, including sliding window and anchor point approaches. The sliding window method generates candidate segments using multi-scale anchor points and utilizes a multimodal matching module to retrieve the most matching candidate for the query. While this method has a simple architecture, its low retrieval accuracy stems from its failure to deeply explore the semantic relationships between modalities and its lack of a fine-grained multimodal fusion mechanism.

[0004] Anchor-based methods integrate candidate generation with result prediction, maintaining candidate segments through multi-level learning modules. This method generates candidates based on preset multi-scale anchors and updates them through temporal or hierarchical mechanisms. However, since candidate segments are processed independently, it fails to fully model temporal dependencies; furthermore, the duration of candidate segments is limited by the scale configuration of the preset anchors.

[0005] Subsequently, researchers proposed a VMR method based on direct localization (without candidate generation). This type of method starts directly from finer-grained video frame or segment features and directly regresses the start and end times of the target time through end-to-end learning, significantly reducing the computational overhead of candidate segment generation in traditional methods. Although it performs well on small datasets, its efficiency, robustness, and scalability remain significantly limited when facing complex scenarios such as large-scale video retrieval and multimodal alignment.

[0006] Therefore, how to quickly and accurately locate video content segments that users are interested in from a massive amount of video content is an urgent problem to be solved. Summary of the Invention

[0007] This invention provides a method, apparatus, device, medium, and program product for cross-modal video segment retrieval based on noise distribution learning, which addresses the shortcomings of existing technologies such as high computational cost and limited robustness. It enables the video retrieval model to accurately recommend video segments of interest to users, and enhances the robustness of the video retrieval model for cross-modal alignment through noise distribution learning, while significantly reducing computational resource consumption.

[0008] In a first aspect, the present invention provides a cross-modal video segment retrieval method based on noise distribution learning, comprising: Obtain a set of video clips to be retrieved; the set of video clips to be retrieved includes at least video clips and text clips; the video clips are represented as video clips to be retrieved, and the text clips are represented as retrieval information input by the user, the retrieval information being used to retrieve video clips with high similarity to the text clips from the set of video clips to be retrieved; Input the set of video segments to be retrieved into the video retrieval model, and output the similarity score between the video segments and the text segments; The similarity score is compared with a preset score threshold, and the similarity scores that are greater than or equal to the preset score threshold are sorted. A preset number of video segments with high similarity scores are used as video segment retrieval results. The video retrieval model is trained based on video samples and text samples. The video retrieval model includes at least a video encoder, a text encoder, and a similarity calculation module. The video encoder is used to extract video features based on the video samples, the text encoder is used to extract text features based on the text samples, and the similarity calculation module is used to calculate the similarity score between the video samples and the text samples based on the video features and the text features. Noise is injected into the video features and the text features respectively during the calculation of the similarity score.

[0009] Preferably, in the cross-modal video segment retrieval method based on noise distribution learning provided by the present invention, the video encoder includes at least: an attention weight module, a residual connection layer, and a normalization layer; The video encoder is used to extract video features based on the video samples, including: Frame sequences are sampled from the video samples, and each frame in the frame sequence is divided into image blocks to form an image block sequence; Trainable parameters and trainable position encoding vectors are concatenated into each image patch vector in the image patch sequence, and attention weights between each image patch vector in the image patch sequence are calculated to obtain a new image patch representation that incorporates global context information; wherein, the trainable position encoding vector is used to determine the position of the trainable parameters; After the new image patch representation is processed by the residual connection layer and the normalization layer, the trainable parameters are extracted as the video features.

[0010] Preferably, in the cross-modal video segment retrieval method based on noise distribution learning provided by the present invention, the text encoder is used to extract text features based on the text samples, including: The text sample is segmented using a word segmenter to obtain multiple word units; Add the first word element identifier at the beginning of a sentence with multiple word elements, and add the second word element identifier at the end of a sentence with multiple word elements to form a new word element sequence; The new lexical sequence is input into the decoder for decoding to obtain lexical encoding features, and the lexical encoding features are normalized to obtain the text features.

[0011] Preferably, in the cross-modal video segment retrieval method based on noise distribution learning provided by the present invention, the similarity calculation module includes at least a Transformer; The similarity calculation module is used to calculate the similarity score between the video sample and the text sample based on the video features and the text features, including: The video features and the text features are concatenated to obtain fused features, and position encoding and type encoding are added to the fused features to obtain the target fused features; wherein, position encoding is used to preserve the position information of the input text features and video features in the sequence, and type encoding is used to distinguish between text features and video features; The target fusion features are input into the Transformer for multi-layer attention interaction calculation to obtain cross-modal interaction features; The first vector output of the cross-modal interaction feature is processed by two layers of linear projection to obtain the similarity score between the video sample and the text sample.

[0012] Preferably, according to the cross-modal video segment retrieval method based on noise distribution learning provided by the present invention, before the step of concatenating the video features and the text features to obtain fused features, the method includes: Cross-attention calculation is performed based on video features and text features to obtain video noise, and multilayer perceptron processing is performed based on text features to obtain text noise; The video noise is injected into the video features to obtain injected video features, and the text noise is injected into the text features to obtain injected text features; The step of concatenating the injected video features and the injected text features to obtain injected fusion features is performed, and the similarity score is calculated based on the injected fusion features.

[0013] Preferably, according to the cross-modal video segment retrieval method based on noise distribution learning provided by the present invention, the step of calculating cross-attention based on video features and text features to obtain video noise includes: Cross-attention calculation is performed on the video features and the text features to obtain the mean and standard deviation of the video noise; The video noise is generated by calculating and processing the standard normally distributed noise, the mean of the video noise, and the standard deviation of the video noise.

[0014] Secondly, the present invention also provides a cross-modal video segment retrieval device based on noise distribution learning, comprising: The acquisition module is used to acquire a set of video segments to be retrieved; the set of video segments to be retrieved includes at least video segments and text segments; the video segments are represented as video segments to be retrieved, and the text segments are represented as retrieval information input by the user; the retrieval information is used to retrieve video segments with high similarity to the text segments from the set of video segments to be retrieved. The retrieval module is used to input the set of video segments to be retrieved into the video retrieval model and output the similarity score between the video segments and the text segments; The comparison module is used to compare the similarity score with a preset score threshold, sort the similarity scores that are greater than or equal to the preset score threshold, and take a preset number of video segments with high similarity scores as video segment retrieval results. The video retrieval model is trained based on video samples and text samples. The video retrieval model includes at least a video encoder, a text encoder, and a similarity calculation module. The video encoder is used to extract video features based on the video samples, the text encoder is used to extract text features based on the text samples, and the similarity calculation module is used to calculate the similarity score between the video samples and the text samples based on the video features and the text features. Noise is injected into the video features and the text features respectively during the calculation of the similarity score.

[0015] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cross-modal video segment retrieval method based on noise distribution learning as described above.

[0016] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-modal video segment retrieval method based on noise distribution learning as described above.

[0017] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-modal video segment retrieval method based on noise distribution learning as described above.

[0018] This invention provides a method, apparatus, device, medium, and program product for cross-modal video segment retrieval based on noise distribution learning. The method involves acquiring a set of video segments to be retrieved; this set includes at least video segments and text segments; the video segments represent the video segments to be retrieved, and the text segments represent the retrieval information input by the user; the retrieval information is used to retrieve video segments with high similarity to the text segments from the set of video segments to be retrieved; the set of video segments to be retrieved is input into a video retrieval model, which outputs a similarity score between the video segments and the text segments; the similarity score is compared with a preset score threshold, and scores greater than or equal to the preset score threshold are selected. The similarity scores are sorted, and a predetermined number of video clips with high similarity scores are used as the video clip retrieval results. The video retrieval model is trained based on video samples and text samples. The video retrieval model includes at least a video encoder, a text encoder, and a similarity calculation module. The video encoder extracts video features based on the video samples, the text encoder extracts text features based on the text samples, and the similarity calculation module calculates the similarity score between the video samples and the text samples based on the video features and the text features. Noise is injected into both the video features and the text features during the similarity score calculation process. This addresses the shortcomings of existing technologies, such as high computational cost and limited robustness, enabling the video retrieval model to accurately recommend video clips of interest to users. Furthermore, noise distribution learning enhances the robustness of the video retrieval model to cross-modal alignment and significantly reduces computational resource consumption. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1This is one of the flowcharts of the cross-modal video segment retrieval method based on noise distribution learning provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the overall framework of the cross-modal video segment retrieval method based on noise distribution learning provided by the present invention.

[0022] Figure 3 This is a schematic diagram of a basic unit of the Transformer provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the similarity calculator provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the video noise generator provided by the present invention.

[0025] Figure 6 This is a schematic diagram of the text noise generator provided by the present invention.

[0026] Figure 7 This is a schematic diagram of the cross-modal video segment retrieval device based on noise distribution learning provided by the present invention.

[0027] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The following is combined Figures 1-8 This invention describes a method, apparatus, device, medium, and program product for cross-modal video segment retrieval based on noise distribution learning, which addresses the shortcomings of existing technologies such as high computational cost and limited robustness. It enables accurate recommendation of video segments of interest to users through a video retrieval model, enhances the robustness of the video retrieval model for cross-modal alignment through noise distribution learning, and significantly reduces computational resource consumption.

[0030] Figure 1 This is one of the flowcharts illustrating a cross-modal video segment retrieval method based on noise distribution learning provided by the present invention, as shown below. Figure 1 As shown, the method may include, but is not limited to, steps S100 to S300: S100, Obtain a set of video segments to be retrieved; the set of video segments to be retrieved includes at least video segments and text segments; the video segments are represented as video segments to be retrieved, and the text segments are represented as retrieval information input by the user, the retrieval information is used to retrieve video segments with high similarity to the text segments from the set of video segments to be retrieved; S200, input the set of video segments to be retrieved into the video retrieval model, and output the similarity score between the video segments and the text segments; S300, compare the similarity score with a preset score threshold, sort the similarity scores that are greater than or equal to the preset score threshold, and take a preset number of video segments with high similarity scores as video segment retrieval results; wherein, the video retrieval model is trained based on video samples and text samples; the video retrieval model includes at least a video encoder, a text encoder, and a similarity calculation module, the video encoder is used to extract video features based on the video samples, the text encoder is used to extract text features based on the text samples, and the similarity calculation module is used to calculate the similarity score between the video samples and the text samples based on the video features and the text features, and inject noise into the video features and the text features respectively during the similarity score calculation process.

[0031] In step S100 of some embodiments, a set of video segments to be retrieved is obtained; the set of video segments to be retrieved includes at least video segments and text segments; the video segments are represented as video segments to be retrieved, and the text segments are represented as retrieval information input by the user, the retrieval information being used to retrieve video segments with high similarity to the text segments from the set of video segments to be retrieved.

[0032] It is understandable that the acquisition of the video clip to be retrieved can be a short video clip (e.g., 10 seconds to 1 minute) extracted from a video library and stored in a standardized format (e.g., MP4), or a short video clip stored on a cloud server.

[0033] In some embodiments, text segment generation involves generating corresponding text information for each video segment, such as: manually annotated keywords, tags, or descriptions (e.g., "cat backflip"); automatically generated subtitles or speech-recognition text (e.g., dialogue in the video); and user comments or title text.

[0034] It should be noted that after obtaining the set of video clips to be retrieved, the data can be cleaned to remove low-quality videos (such as blurry or black screens) or irrelevant text (such as advertising copy).

[0035] This step achieves the following technical effects: by constructing a multimodal dataset, it provides a foundation for subsequent feature extraction. The diversity of text fragments (such as keywords and captions) enhances retrieval flexibility.

[0036] In step S200 of some embodiments, the set of video segments to be retrieved is input into a video retrieval model, and a similarity score between the video segments and the text segments is output. Understandably, before inputting the set of video segments to be retrieved into the video retrieval model, the video segments to be retrieved can be standardized, including decoding, editing (such as unifying the duration), and resolution adjustment (such as unifying to 224x224), etc.

[0037] And text fragment processing, specifically cleaning the user-input text (such as removing special characters and stop words) and word segmentation (such as based on NLP tools).

[0038] Furthermore, video and text clips are converted into input formats acceptable to the model (e.g., video is converted into a frame sequence, and text is converted into a token sequence).

[0039] The data processing examples described above can eliminate the impact of video format differences (such as resolution and frame rate) on the model. They can also improve the semantic clarity of text (e.g., by removing irrelevant symbols) and prevent noise from interfering with model judgments.

[0040] Furthermore, after inputting the set of video segments to be retrieved into the video retrieval model, the video features are first extracted. Specifically, a pre-trained video encoder (such as CLIP or SlowFast) is used to extract the video feature vectors (such as RGB frame features and motion features) and audio features (such as MFCC).

[0041] Example: Given a video of a cat doing a backflip, the model extracts video feature vectors containing semantics such as "cat" and "jump".

[0042] Furthermore, the text feature extraction step can specifically involve using a pre-trained text encoder (such as BERT or RoBERTa) to convert text fragments into semantic vectors, i.e., text feature vectors.

[0043] Example: Input text "Man in black coat runs into park", model output vector containing semantics such as "person", "movement", and "scene".

[0044] This embodiment achieves at least the following technical effects: mapping video (visual / audio) and text to the same high-dimensional vector space, supporting direct comparison; and reducing computational complexity by compressing raw data (such as video frames and text words) into low-dimensional dense vectors.

[0045] Furthermore, it also includes calculating the similarity between video feature vectors and text feature vectors: First, the similarity between the video feature vector and the text feature vector is calculated (e.g., cosine similarity, dot product). Then, during the training phase, random noise (e.g., Gaussian noise, Dropout) is added to the video and text feature vectors to simulate interference in real-world scenarios (e.g., low image quality, text typos).

[0046] Example: Randomly mask parts of a video feature, or randomly replace parts of a word in text.

[0047] This embodiment achieves at least the following technical effects: noise injection makes the model insensitive to minor changes in the input data (such as blurred images or colloquial expressions). The similarity score directly reflects the semantic relevance between the video and the text, facilitating sorting and filtering.

[0048] The video retrieval model inputs video clips and text clips, and the model outputs a similarity score between the two (the value usually ranges from 0 to 1, with a larger value indicating greater similarity).

[0049] Example: The user inputs the text "cat backflip", and the model calculates its similarity to videos in the library. One video scores 0.85.

[0050] In step S300 of some embodiments, the similarity score is compared with a preset score threshold, and the similarity scores greater than or equal to the preset score threshold are sorted. A preset number of video segments with high similarity scores are used as video segment retrieval results. The video retrieval model is trained based on video samples and text samples. The video retrieval model includes at least a video encoder, a text encoder, and a similarity calculation module. The video encoder is used to extract video features based on the video samples, the text encoder is used to extract text features based on the text samples, and the similarity calculation module is used to calculate the similarity score between the video samples and the text samples based on the video features and the text features. Noise is injected into the video features and the text features during the similarity score calculation process.

[0051] Understandably, a preset score threshold (e.g., 0.7) is used to filter out scores below the threshold and retain high-scoring results. This reduces interference from low-relevance results and improves search accuracy.

[0052] Furthermore, similarity scores above the threshold are sorted in descending order (from high to low).

[0053] Example: If the scores of the three videos are 0.9, 0.8 and 0.75 respectively, the sorting order is 0.9>0.8>0.75.

[0054] Based on business requirements (such as returning the top 10 results), select the top-ranked videos as the final search results.

[0055] This implementation prioritizes displaying the most relevant videos, improving the user experience. It also controls the number of returned results, avoiding excessive irrelevant information.

[0056] It should be further noted that, in some embodiments, a video encoder is a model that converts video data (such as images and sound) into feature vectors that can be understood by a computer.

[0057] Example: The CLIP model converts videos into feature vectors containing semantics such as "cat" and "jump".

[0058] A text encoder is a model that converts text data (such as keywords and sentences) into semantic feature vectors.

[0059] Example: The BERT model converts "man in black coat" into a vector containing features such as "person" and "color".

[0060] The Similarity Calculation Module is an algorithm that measures the degree of matching between two feature vectors (video and text).

[0061] Example: Cosine similarity is calculated by taking the cosine of the angle between vectors. The closer the value is to 1, the more similar the vectors are.

[0062] In some embodiments, such as search scenarios on short video platforms Input: The user inputs "Sunrise at the beach, taken by drone".

[0063] Process: The text encoder converts the query into a vector, retrieves video features from the database, calculates similarity, and returns high-scoring videos (such as drone aerial footage of sunrise). It supports conversational queries (such as "drone sunrise") and injects noise to improve the matching ability for blurry images.

[0064] The embodiments provided by this invention achieve cross-modal retrieval through multimodal feature alignment, while noise injection enhances robustness and avoids overfitting.

[0065] The following examples illustrate the steps for training a video retrieval model: In some embodiments of the present invention, the video encoder includes at least: an attention weighting module, a residual connection layer, and a normalization layer; The video encoder is used to extract video features based on the video samples, including: Frame sequences are sampled from the video samples, and each frame in the frame sequence is divided into image blocks to form an image block sequence; Trainable parameters and trainable position encoding vectors are concatenated into each image patch vector in the image patch sequence, and attention weights between each image patch vector in the image patch sequence are calculated to obtain a new image patch representation that incorporates global context information; wherein, the trainable position encoding vector is used to determine the position of the trainable parameters; After the new image patch representation is processed by the residual connection layer and the normalization layer, the trainable parameters are extracted as the video features.

[0066] It is understandable that in some embodiments, the backbone network of the video encoder uses the VisionTransformer model (ViT-B / 32) and reuses the existing parameters of CLIP (ViT-B / 32) to extract frame-level features from video samples.

[0067] First, a fixed frame rate is set, and frame sequences are sampled uniformly from the video samples. Using a 32×32 kernel with a stride of 16, each frame... The image is divided into 32×32 blocks to form an image block sequence. Then, a linear mapping layer flattens the dimensions of the image blocks and maps them to 49×512. The process is shown in formula (1): (1) In equation (1), Let t represent the t-th frame of the video, and k represent taking k+1 consecutive frames for processing, ultimately obtaining a set of encoded image blocks for the corresponding frame within that time window. .

[0068] Furthermore, the video sample size is 224×224×12×3, the video is divided into 12 frames, and each frame is divided into 49 32×32×3 image blocks. After mapping, the feature dimension of the entire video sample is transformed to 196×512.

[0069] Subsequently, a trainable parameter is concatenated into each obtained image patch vector, and a trainable positional encoding vector is added, at which point the dimension transformation is 197×512. The ViT Transformer calculates the relationship between each image patch vector, and the embedding of each image patch is passed through three different linear mapping layers to generate query Q, key K, and value V vectors, respectively. The attention weight matrix A is obtained by calculating the dot product of the query and key, dividing by a scaling factor, and then performing Softmax. .

[0070] The attention weight matrix A represents the association strength between each image patch and other image patches. The value vectors are then weighted and summed using this attention weight matrix to obtain a new image patch representation that incorporates global contextual information.

[0071] Furthermore, after improving feature stability and representational power through residual connections and normalization layers, the concatenated trainable parameters are extracted as the final frame-level features, i.e., video features. .

[0072] In some embodiments of the present invention, the text encoder is used to extract text features based on the text sample, including: The text sample is segmented using a word segmenter to obtain multiple word units; Add the first word element identifier at the beginning of a sentence with multiple word elements, and add the second word element identifier at the end of a sentence with multiple word elements to form a new word element sequence; The new lexical sequence is input into the decoder for decoding to obtain lexical encoding features, and the lexical encoding features are normalized to obtain the text features.

[0073] Understandably, in some embodiments, the text feature extraction process relies on the Transformer decoder, with each layer containing 8 attention heads and 512 hidden units, for a total of 12 layers of decoders stacked together.

[0074] First, the input text sample, such as subtitles, is processed by a word segmenter, which divides the text sample into multiple word units. Special word units [BOS] and [EOS] are added at the beginning and end of the word units to construct a new word unit sequence. ,in Indicates the number of lexical units; [EOS] marks the end of the sentence.

[0075] The context-based embedding representation of each lexical unit is obtained by formula (2): (2) In equation (2), LN stands for Layer Normalization. These are the parameters of the mapping layer in the CLIP text encoder, and h=512 is the dimension of the hidden layer of the text encoder. It is the output of the Transformer decoder after encoding the word t. This is a standard L2 norm normalization operation used to maintain consistent vector lengths. [EOS] is used as the global representation for the entire text. , Represented as text features.

[0076] In some embodiments of the present invention, the similarity calculation module includes at least a Transformer; The similarity calculation module is used to calculate the similarity score between the video sample and the text sample based on the video features and the text features, including: The video features and the text features are concatenated to obtain fused features, and position encoding and type encoding are added to the fused features to obtain the target fused features; wherein, position encoding is used to preserve the position information of the input text features and video features in the sequence, and type encoding is used to distinguish between text features and video features; The target fusion features are input into the Transformer for multi-layer attention interaction calculation to obtain cross-modal interaction features; The first vector output of the cross-modal interaction feature is processed by two layers of linear projection to obtain the similarity score between the video sample and the text sample.

[0077] It is understandable that, in some embodiments, the similarity calculation module will extract video frame features. and text features Both modalities undergo Transform processing to perform feature interactions, and finally, a similarity score is calculated using two layers of linear projection, specifically: First, the features of the input text and video frame features By splicing together, a fusion feature is formed. As shown in equation (3): (3) Add positional encoding P and type encoding L to the concatenated fused features. Positional encoding P is used to preserve the positional information of the input text features and video features in the sequence, and type encoding L is used to distinguish between text features and video features.

[0078] Furthermore, the fused features are input into the Transformer for calculation, and the calculation process is shown in equations (4) and (5): (4) In equation (4), It is a fusion feature formed by concatenating text features and video features. P represents position encoding, L represents type encoding, and X is the target fusion feature.

[0079] (5) In equation (5), , , is a learnable weight matrix, softmax is a normalization layer used to normalize the weights, and d is the feature dimension used for scaling to prevent gradient explosion.

[0080] (6) In equation (6), FFN represents a feedforward neural network and LayerNorm represents a normalization layer.

[0081] Cross-modal interaction features are obtained through multi-layer attention interactions of Transformer. The first vector output from the last layer is... .

[0082] Finally, similarity scores are calculated using two layers of linear projection: (7) In Equation (7), the first fully connected layer (FC) is used to project the features into the hidden space and increase the nonlinear expressive power through ReLU activation; the second FC layer is used to project the features into one dimension as the final matching similarity.

[0083] In some embodiments, the video retrieval model consists of a video encoder, a text encoder, and a similarity calculation module. The video encoder and the text encoder are used to extract video feature vectors and text feature vectors of the same dimension, respectively. The similarity calculation module calculates the similarity score between the video and the text using the extracted feature vectors.

[0084] A contrastive learning loss function is used during the training of the video retrieval model. Bidirectional contrastive learning is employed to enhance cross-modal representation learning capabilities, ensuring bidirectional optimization of video-to-text and text-to-video retrieval. Simultaneously, negative samples from the batch are utilized for efficient training. During training, symmetric cross-entropy loss is used to ensure bidirectional optimization of video-to-text and text-to-video retrieval. The loss functions are shown in equations (8), (9), and (10): (8) (9) (10) Where B is the batch size. Computational Video and all text within a batch The sum of similarities; Calculate text and all videos within the batch The sum of similarities for each video. Hopefully it matches the correct text. It has the highest similarity score ,vice versa.

[0085] In some embodiments of the present invention, before the step of concatenating the video features and the text features to obtain fused features, the method includes: Cross-attention calculation is performed based on video and text features to obtain video noise.

[0086] It is understood that the video noise obtained by performing cross-attention calculation based on video features and text features includes: Cross-attention calculation is performed on the video features and the text features to obtain the mean and standard deviation of the video noise; The video noise is generated by calculating and processing the standard normally distributed noise, the mean of the video noise, and the standard deviation of the video noise.

[0087] It can be further understood that the video retrieval model provided in the embodiments of the present invention also includes a video noise generator.

[0088] Video noise generator A cross-attention architecture can be adopted, mainly composed of a linear mapping layer and a cross-attention module. This is achieved by applying cross-attention to the generated video features. Text features Capture the interaction information between video and text samples to generate the mean of video noise. and standard deviation .

[0089] First, video features Query Q is generated after a linear transformation. Text features After a linear transformation, the key K and value V are generated. , ,in , , These are the weight matrices for Q, K, and V, respectively.

[0090] The calculation process using the generated Q, K, and V attention-weighted features is shown in equations (11), (12), and (13): (11) Finally, the mean of the generated video noise. and standard deviation : (12) In equation (12), Z and T refer to video features and text features, respectively.

[0091] Introducing reparameterization techniques to generate video noise : (13) In equation (13), It's video noise. It is standard normally distributed noise, with a standard deviation of Control the scale and mean of video noise. The mean and sign of the video noise are determined. This represents element-wise multiplication.

[0092] Furthermore, text noise is obtained by performing multilayer perceptron processing based on text features.

[0093] The video retrieval model provided in this embodiment of the invention also includes a text noise generator. The architecture of a multilayer perceptron (MLP) can be used to input text features. Text features are processed using a multilayer perceptron to generate the mean of text noise. and standard deviation .

[0094] The text noise generator consists of a fully connected network. Its input layer receives text features, which are then processed by a non-linear activation function in the hidden layers. Finally, the output layer generates the mean of the text noise. and standard deviation The specific process is shown in equations (14), (15), (16), and (17): (14) (15) (16) (17) Where T represents the text feature. , This is the hidden layer dimension, and the dimension is chosen as h=1024; , and , These are the weights and biases of the hidden layer, respectively; and , and , These are the weights and biases of the output layer, respectively. This is used to ensure that the standard deviation is positive.

[0095] Introducing reparameterization techniques to generate text noise : (18) In equation (18), Represented as text noise, It is standard normally distributed noise, with a standard deviation of Control the scale and mean of noise. The mean and sign of the text noise are determined. This represents element-wise multiplication.

[0096] Furthermore, the video features are injected with the video noise to obtain injected video features. Similarly, the text features are injected with the text noise to obtain injected text features.

[0097] This invention, without altering the model architecture of the video retrieval model, can determine the injection of noise at different locations within the structure based on varying requirements of the video retrieval model, and determine the fusion method between the noise and the original data. Noise can be added or multiplicative, and can be incorporated into the original image, original text, or video features output by a visual encoder or text encoder. Text features Noise is added to the model. This is to avoid excessive perturbation of features, which could affect the stability of the model, without altering the intermediate computational process.

[0098] In embodiments of the present invention, additive noise injection is selected to inject noise into the video features output by the video encoder and the text encoder. , In the process, the corresponding injected video features and injected text features are obtained respectively: , .

[0099] In the formula, This is represented as injected video features. This is represented as injected text features. Represented as video features, Represented as text features, It's video noise. This is represented as text noise.

[0100] The step of concatenating the injected video features and the injected text features to obtain injected fusion features is performed, and the similarity score is calculated based on the injected fusion features.

[0101] Understandably, after injecting video noise and text noise to obtain injected video features and injected text features, the video retrieval model is fine-tuned again, and the similarity score is calculated through the similarity calculation module to complete the video segment retrieval task.

[0102] The similarity calculation module will fuse beneficial noise video features. Text features And add position code P and type code L, The fusion feature X is fed into the similarity calculation module to obtain the final similarity score S (the specific calculation process is shown in the calculation steps of formula (3) to formula (7) above), and the similarity threshold is set according to the similarity score. Candidates with scores higher than the threshold are retained, and these candidate scores are sorted. The top K are taken as the candidate results, which is the final similarity score.

[0103] like Figure 2 This is a schematic diagram of the overall framework of the cross-modal video segment retrieval method based on noise distribution learning provided by this invention. Video segments and text segments are input into the video retrieval model to output similarity scores between the video segments and the text segments. Specifically... Figure 2 The left side shows how video clips are processed using the structural components of a video retrieval model. Specifically, convolutional layers, flattening layers, and linear projections are input to extract image patch vectors corresponding to the video clips. Trainable parameters and trainable positional codes are then inserted. After the ViT Transformer calculates the relationship between each image patch, the stitched trainable parameters are finally extracted as video features. Video noise is then added to obtain injected video features with injected video noise.

[0104] Figure 2 The right side shows how text segments are input into a word segmenter for word segmentation, resulting in tokens. Category tags (trainable parameters) and position embeddings (trainable positional encodings) are then added, followed by encoding by a Transformer encoder and normalization to obtain text features. Finally, text noise is injected into these text features to obtain noise-injected text features.

[0105] The similarity calculator in the similarity calculation module inputs injected video features and injected text features to calculate the similarity score between the video clip and the text clip, and determines the video clip retrieval result based on the similarity score.

[0106] like Figure 3 This is a schematic diagram of a basic unit of the Transformer provided by the present invention, which encodes video segments and text segments respectively to obtain corresponding video features and text features.

[0107] Figure 4 This is a schematic diagram of the similarity calculator provided by the present invention, which is used to encode the injected text features (text embedding + position encoding) of injected text noise and the injected video features (frame embedding + position encoding) of injected video noise through a Transformer encoder and calculate the similarity score between the video segment and the text segment through a linear transformation layer.

[0108] Figure 5This is a schematic diagram of the video noise generator provided by the present invention, which is used to calculate the mean and standard deviation of video noise by applying attention weights to video features and text features.

[0109] Figure 6 This is a schematic diagram of the text noise generator provided by the present invention, which is used to process text features through a multilayer perceptron to generate the mean and standard deviation of text noise.

[0110] The present invention can achieve at least the following beneficial effects through the above embodiments: 1. The cross-modal video segment retrieval based on noise distribution learning provided by this invention treats noise as a positive stimulus, effectively reducing the uncertainty between modalities, improving the model's ability to capture key features, making the representation of video segments more distinct, and thus improving the accuracy of retrieval.

[0111] 2. This invention, based on noise distribution learning, enables cross-modal video segment retrieval. By injecting beneficial noise into the global features of the video, the model ignores unimportant background information and focuses only on the core content of the video. This not only maintains excellent performance even under high task complexity or insufficient data, but also effectively avoids mismatches caused by irrelevant information, reduces invalid computation of the model, and improves retrieval efficiency.

[0112] This invention provides a method, apparatus, device, medium, and program product for cross-modal video segment retrieval based on noise distribution learning. It addresses the shortcomings of existing technologies, such as high computational cost and limited robustness. The invention enables accurate recommendation of video segments of interest to users through a video retrieval model. Furthermore, the invention enhances the robustness of the video retrieval model for cross-modal alignment through noise distribution learning and significantly reduces computational resource consumption.

[0113] The following describes the cross-modal video segment retrieval device based on noise distribution learning provided by the present invention. The cross-modal video segment retrieval device based on noise distribution learning described below can be referred to in correspondence with the cross-modal video segment retrieval method based on noise distribution learning described above.

[0114] like Figure 7 The diagram shown is a structural schematic of the cross-modal video segment retrieval device based on noise distribution learning provided by the present invention. The cross-modal video segment retrieval device based on noise distribution learning includes the following modules: The acquisition module 710 is used to acquire a set of video segments to be retrieved; the set of video segments to be retrieved includes at least video segments and text segments; the video segments are represented as video segments to be retrieved, and the text segments are represented as retrieval information input by the user; the retrieval information is used to retrieve video segments with high similarity to the text segments from the set of video segments to be retrieved. The retrieval module 720 is used to input the set of video segments to be retrieved into the video retrieval model and output the similarity score between the video segments and the text segments. The comparison module 730 is used to compare the similarity score with a preset score threshold, sort the similarity scores that are greater than or equal to the preset score threshold, and take a preset number of video segments with high similarity scores as video segment retrieval results. The video retrieval model is trained based on video samples and text samples. The video retrieval model includes at least a video encoder, a text encoder, and a similarity calculation module. The video encoder is used to extract video features based on the video samples, the text encoder is used to extract text features based on the text samples, and the similarity calculation module is used to calculate the similarity score between the video samples and the text samples based on the video features and the text features. Noise is injected into the video features and the text features respectively during the calculation of the similarity score.

[0115] The cross-modal video segment retrieval device based on noise distribution learning provided by the present invention is further used in the video encoder, which includes at least: an attention weight module, a residual connection layer and a normalization layer; Frame sequences are sampled from the video samples, and each frame in the frame sequence is divided into image blocks to form an image block sequence; Trainable parameters and trainable position encoding vectors are concatenated into each image patch vector in the image patch sequence, and attention weights between each image patch vector in the image patch sequence are calculated to obtain a new image patch representation that incorporates global context information; wherein, the trainable position encoding vector is used to determine the position of the trainable parameters; After the new image patch representation is processed by the residual connection layer and the normalization layer, the trainable parameters are extracted as the video features.

[0116] The cross-modal video segment retrieval device based on noise distribution learning provided by the present invention is further used to segment the text sample by a word segmenter to obtain multiple word units; Add the first word element identifier at the beginning of a sentence with multiple word elements, and add the second word element identifier at the end of a sentence with multiple word elements to form a new word element sequence; The new lexical sequence is input into the decoder for decoding to obtain lexical encoding features, and the lexical encoding features are normalized to obtain the text features.

[0117] The cross-modal video segment retrieval device based on noise distribution learning provided by the present invention is further configured such that the similarity calculation module includes at least a Transformer. The video features and the text features are concatenated to obtain fused features, and position encoding and type encoding are added to the fused features to obtain the target fused features; wherein, position encoding is used to preserve the position information of the input text features and video features in the sequence, and type encoding is used to distinguish between text features and video features; The target fusion features are input into the Transformer for multi-layer attention interaction calculation to obtain cross-modal interaction features; The first vector output of the cross-modal interaction feature is processed by two layers of linear projection to obtain the similarity score between the video sample and the text sample.

[0118] The cross-modal video segment retrieval device based on noise distribution learning provided by the present invention is further used to perform cross-attention calculation based on video features and text features to obtain video noise, and to perform multilayer perceptron processing based on text features to obtain text noise; The video noise is injected into the video features to obtain injected video features, and the text noise is injected into the text features to obtain injected text features; The step of concatenating the injected video features and the injected text features to obtain injected fusion features is performed, and the similarity score is calculated based on the injected fusion features.

[0119] The cross-modal video segment retrieval device based on noise distribution learning provided by the present invention is further used to perform cross-attention calculation on the video features and the text features to obtain the mean and standard deviation of the video noise; The video noise is generated by calculating and processing the standard normally distributed noise, the mean of the video noise, and the standard deviation of the video noise.

[0120] This invention provides a method, apparatus, device, medium, and program product for cross-modal video segment retrieval based on noise distribution learning. It addresses the shortcomings of existing technologies, such as high computational cost and limited robustness. The invention enables accurate recommendation of video segments of interest to users through a video retrieval model. Furthermore, the invention enhances the robustness of the video retrieval model for cross-modal alignment through noise distribution learning and significantly reduces computational resource consumption.

[0121] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a cross-modal video segment retrieval method based on noise distribution learning. This method includes: acquiring a set of video segments to be retrieved; the set of video segments to be retrieved includes at least video segments and text segments; the video segments are represented as video segments to be retrieved, and the text segments are represented as retrieval information input by the user, the retrieval information being used to retrieve video segments with high similarity to the text segments from the set of video segments to be retrieved; inputting the set of video segments to be retrieved into a video retrieval model, and outputting a similarity score between the video segments and the text segments; comparing the similarity score with a preset score threshold, and sorting the similarity scores greater than or equal to the preset score threshold, and using a preset number of video segments with high similarity scores as the video segment retrieval results.

[0122] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the cross-modal video segment retrieval method based on noise distribution learning provided by the above methods. The method includes: obtaining a set of video segments to be retrieved; the set of video segments to be retrieved includes at least video segments and text segments; the video segments are represented as video segments to be retrieved, and the text segments are represented as retrieval information input by the user, the retrieval information being used to retrieve video segments with high similarity to the text segments from the set of video segments to be retrieved; inputting the set of video segments to be retrieved into a video retrieval model, and outputting a similarity score between the video segments and the text segments; comparing the similarity score with a preset score threshold, and sorting the similarity scores that are greater than or equal to the preset score threshold, and taking a preset number of video segments with high similarity scores as video segment retrieval results.

[0124] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the cross-modal video segment retrieval method based on noise distribution learning provided by the above methods. The method includes: acquiring a set of video segments to be retrieved; the set of video segments to be retrieved includes at least video segments and text segments; the video segments are represented as video segments to be retrieved, and the text segments are represented as retrieval information input by a user, the retrieval information being used to retrieve video segments with high similarity to the text segments from the set of video segments to be retrieved; inputting the set of video segments to be retrieved into a video retrieval model, and outputting a similarity score between the video segments and the text segments; comparing the similarity score with a preset score threshold, and sorting the similarity scores greater than or equal to the preset score threshold, and using a preset number of video segments with high similarity scores as the video segment retrieval results.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal video segment retrieval method based on noise distribution learning, characterized in that, include: Obtain a set of video clips to be retrieved; the set of video clips to be retrieved includes at least video clips and text clips; the video clips are represented as video clips to be retrieved, and the text clips are represented as retrieval information input by the user, the retrieval information being used to retrieve video clips with high similarity to the text clips from the set of video clips to be retrieved; Input the set of video segments to be retrieved into the video retrieval model, and output the similarity score between the video segments and the text segments; The similarity score is compared with a preset score threshold, and the similarity scores that are greater than or equal to the preset score threshold are sorted. A preset number of video segments with high similarity scores are used as video segment retrieval results. The video retrieval model is trained based on video samples and text samples. The video retrieval model includes at least a video encoder, a text encoder, and a similarity calculation module. The video encoder is used to extract video features based on the video samples, the text encoder is used to extract text features based on the text samples, and the similarity calculation module is used to calculate the similarity score between the video samples and the text samples based on the video features and the text features. Noise is injected into the video features and the text features respectively during the calculation of the similarity score.

2. The cross-modal video segment retrieval method based on noise distribution learning according to claim 1, characterized in that, The video encoder includes at least: an attention weight module, a residual connection layer, and a normalization layer; The video encoder is used to extract video features based on the video samples, including: Frame sequences are sampled from the video samples, and each frame in the frame sequence is divided into image blocks to form an image block sequence; Trainable parameters and trainable position encoding vectors are concatenated into each image patch vector in the image patch sequence, and attention weights between each image patch vector in the image patch sequence are calculated to obtain a new image patch representation that incorporates global context information; wherein, the trainable position encoding vector is used to determine the position of the trainable parameters; After the new image patch representation is processed by the residual connection layer and the normalization layer, the trainable parameters are extracted as the video features.

3. The cross-modal video segment retrieval method based on noise distribution learning according to claim 1, characterized in that, The text encoder is used to extract text features based on the text samples, including: The text sample is segmented using a word segmenter to obtain multiple word units; Add the first word element identifier at the beginning of a sentence with multiple word elements, and add the second word element identifier at the end of a sentence with multiple word elements to form a new word element sequence; The new lexical sequence is input into the decoder for decoding to obtain lexical encoding features, and the lexical encoding features are normalized to obtain the text features.

4. The cross-modal video segment retrieval method based on noise distribution learning according to claim 1, characterized in that, The similarity calculation module includes at least a Transformer; The similarity calculation module is used to calculate the similarity score between the video sample and the text sample based on the video features and the text features, including: The video features and the text features are concatenated to obtain fused features, and position encoding and type encoding are added to the fused features to obtain the target fused features; wherein, position encoding is used to preserve the position information of the input text features and video features in the sequence, and type encoding is used to distinguish between text features and video features; The target fusion features are input into the Transformer for multi-layer attention interaction calculation to obtain cross-modal interaction features; The first vector output of the cross-modal interaction feature is processed by two layers of linear projection to obtain the similarity score between the video sample and the text sample.

5. The cross-modal video segment retrieval method based on noise distribution learning according to claim 4, characterized in that, Before the step of concatenating the video features and the text features to obtain the fused features, the method includes: Cross-attention calculation is performed based on video features and text features to obtain video noise, and multilayer perceptron processing is performed based on text features to obtain text noise; The video noise is injected into the video features to obtain injected video features, and the text noise is injected into the text features to obtain injected text features; The step of concatenating the injected video features and the injected text features to obtain injected fusion features is performed, and the similarity score is calculated based on the injected fusion features.

6. The cross-modal video segment retrieval method based on noise distribution learning according to claim 5, characterized in that, The video noise obtained by performing cross-attention calculation based on video features and text features includes: Cross-attention calculation is performed on the video features and the text features to obtain the mean and standard deviation of the video noise; The video noise is generated by calculating and processing the standard normally distributed noise, the mean of the video noise, and the standard deviation of the video noise.

7. A cross-modal video segment retrieval device based on noise distribution learning, characterized in that, include: The acquisition module is used to acquire a set of video segments to be retrieved; the set of video segments to be retrieved includes at least video segments and text segments; the video segments are represented as video segments to be retrieved, and the text segments are represented as retrieval information input by the user; the retrieval information is used to retrieve video segments with high similarity to the text segments from the set of video segments to be retrieved. The retrieval module is used to input the set of video segments to be retrieved into the video retrieval model and output the similarity score between the video segments and the text segments; The comparison module is used to compare the similarity score with a preset score threshold, sort the similarity scores that are greater than or equal to the preset score threshold, and take a preset number of video segments with high similarity scores as video segment retrieval results. The video retrieval model is trained based on video samples and text samples. The video retrieval model includes at least a video encoder, a text encoder, and a similarity calculation module. The video encoder is used to extract video features based on the video samples, the text encoder is used to extract text features based on the text samples, and the similarity calculation module is used to calculate the similarity score between the video samples and the text samples based on the video features and the text features. Noise is injected into the video features and the text features respectively during the calculation of the similarity score.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the cross-modal video segment retrieval method based on noise distribution learning as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-modal video segment retrieval method based on noise distribution learning as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the cross-modal video segment retrieval method based on noise distribution learning as described in any one of claims 1 to 6.

Images