A text-video retrieval method based on gaussian smoothing and semantic preservation

Abstract

The application discloses a text-video retrieval method based on Gaussian smoothing and semantic preservation, which comprises the following steps: firstly, taking a text core body as an anchor point to construct a frame-level semantic matching mechanism, and introducing a one-dimensional Gaussian filter to smooth a time sequence score sequence, so as to effectively inhibit background noise interference and generate a global video aggregation representation conforming to query semantics; secondly, compressing redundant features through a visual bottleneck layer, and combining a learnable scaling residual fusion strategy to realize cross-modal semantic complementation, anchor the original discriminative structure of the text and reduce feature noise; finally, modeling a feature distribution by using a Gaussian probability model, and bridging a cross-modal semantic gap by constraining the consistency of a single-modal prior and a double-modal posterior distribution. Through Gaussian smoothing and cross-modal alignment, the application effectively improves the matching precision of text and video features, and realizes efficient and accurate video retrieval.

Description

Technical Field

[0001] This invention belongs to the field of multimodal information retrieval technology, and in particular to a text-video retrieval method based on Gaussian smoothing and semantic preservation. Background Technology

[0002] With the rapid growth of multimedia content, text-video retrieval technology has demonstrated significant application value in numerous practical scenarios. This technology is widely applicable to fields such as security monitoring, film and television entertainment, online education, content platforms, and government and enterprise offices. It can replace manual screening, accurately retrieve specific events, knowledge points, or key segments, and extends to scenarios such as healthcare and transportation. With the explosive growth of video data, text-video retrieval technology has significantly improved the efficiency of cross-modal content matching and distribution, effectively meeting the urgent needs of intelligent video resource management across various industries, and possesses extremely high practical value and broad application prospects.

[0003] However, significant differences in information density and representation granularity exist between video and text. Specifically, video data typically contains a large amount of spatiotemporal redundancy, such as complex background environments, dynamic changes in light and shadow, and long sequences of still frames or transition frames lacking significant subject movement. This redundant information constitutes the majority of video feature representations but is often weakly or completely irrelevant to the core semantics of the task. In contrast, text descriptions are highly sparsity and semantically concise. In practical applications, the same video sequence can often correspond to multiple text descriptions from different perspectives or with different granularities, and users' text queries often focus only on specific targets, local actions, or key event segments in the video. This inequality makes irrelevant background and redundant temporal information in video sequences easily introduced as interference noise into the feature matching process. When calculating cross-modal similarity, the model struggles to automatically remove visual components irrelevant to the text semantics, resulting in the alignment target being diluted by a large amount of low-value information. During cross-modal interaction, the highly discriminative semantics of the original text are easily obscured by excessive visual redundancy, and too many visual features cause the text features to deviate from their original semantic distribution. Summary of the Invention

[0004] To overcome the shortcomings of current text-video retrieval methods in addressing the differences in information density and mismatch in representation granularity between video and text, this invention proposes a text-video retrieval method based on Gaussian smoothing and semantic preservation. By suppressing major noise through Gaussian smoothing aggregation and bridging the modality gap through constrained distribution consistency, this method effectively improves the matching accuracy of text and video features, achieving efficient and accurate video retrieval.

[0005] The technical solution adopted by this invention to solve the technical problem is: A text-to-video retrieval method based on Gaussian smoothing and semantic preservation is proposed. First, frame-level semantic matching and scoring are performed using the core text as anchor points. A one-dimensional Gaussian filter is introduced to smooth the temporal score sequence, suppressing inter-frame semantic jumps and background noise interference, and generating a global video aggregation representation that fits the query semantics. Second, redundant video features are compressed through a visual bottleneck layer, and a learnable and scalable residual fusion strategy is adopted to achieve cross-modal semantic supplementation while anchoring the original discriminative structure of the text and avoiding feature noise. The feature distribution is modeled using a Gaussian probability model, and the consistency between the unimodal prior and bimodal posterior distributions is constrained to bridge the cross-modal semantic gap. Finally, the similarity score between the query text and the fused video features is calculated to retrieve videos that match the query text.

[0006] Furthermore, the method includes the following steps: Step 1: Define the text-video set as... ,in Represents the set of the first A text-video pair, This represents the total number of samples in the dataset. The text describes the information, where, Representative text description The first in vocabulary, express The number of words included; for Matching video data, among which Indicates video The first in Frame image, express Total number of frames included; Step 2: Perform data preprocessing operations from the text-video collection. Randomly select a text-video pair First of all Perform equal-interval sampling to extract Frame images, as typical visual segments, are processed using a pre-trained CLIP visual encoder to process the sampled images. Frame images are feature-encoded to generate video. Frame-level visual feature vector sequence ,in Indicates the first The visual feature vector generated after the frame is encoded. The dimension parameter represents the space in which the feature vector resides; for text... The CLIP text encoder is used to perform semantic encoding operations to obtain... Text feature vectors and text word feature vectors ,in Representative text description The first in The feature vectors of the words; traversing the text-video dataset Following the above processing method, the visual feature vector, text feature vector, and text word feature vector of each text-video pair are obtained, thus obtaining the video feature vector set. Text feature vector set and text word feature vector set .

[0007] Furthermore, the method also includes the following steps: Step 3: In the text word feature vector set Choose any one of the text word feature vectors Generate text main feature vector ; in, It is a learnable query vector; it iterates through the set of text word feature vectors. This method yields the set of feature vectors for the main body of the text. ; Step 4: In the video feature vector set Choose any video feature vector and from the text main feature vector set Obtain the corresponding text main feature vector From the set of text feature vectors Obtain the corresponding text feature vector. ,calculate Mid-frame feature vector Attention score ; in, It is a multilayer perceptron. For activation functions; Next calculation Corresponding video global feature vector ; in, This indicates the original score sequence. Smoothing score after one-dimensional Gaussian filtering It is the first The score for attention in a frame. It is the radius of the sliding window. The standard deviation of the Gaussian kernel controls the smoothing range and degree; iterates through the set of video feature vectors. This method yields the set of global feature vectors for the video. ; Step 5: In the video feature vector set Choose any video feature vector and from the text feature vector set Obtain the corresponding text feature vector. ,calculate Corresponding video core features ; in, It is a set of learnable feature vectors. for The length of the vector. This indicates a splicing operation. For the encoder layer based on the multi-head self-attention mechanism, This indicates the output processed by the Transformer. From the vectors, extract the first element by index. One vector; Will and Concatenate the vectors and compute the text-enhanced feature vectors for cross-modal interaction. ; in, For the encoder layer based on the multi-head self-attention mechanism, Indicates the first All features from the position to the end; Next, the optimized text feature vector is calculated. ; in, For element-wise multiplication, A learnable scaling vector; traversing the set of video feature vectors This method yields an optimized set of text feature vectors. ; Step 6: Calculate the text-to-video loss. ; in, Indicates the first Optimized text feature vectors With the A video global feature vector Cosine similarity; Step 7: Calculate the video-to-text loss. ; in, Indicates the first A video global feature vector With the Optimized text feature vectors The cosine similarity.

[0008] Furthermore, the method also includes the following steps: Step 8: Calculate self-attention visual features and dual-modal features ; ; in, This indicates a scaled dot product attention mechanism. and These represent the sets of video feature vectors respectively. and text word feature vector set All feature vectors in; Step 9: Calculate the prior Gaussian distribution ; in, , and They represent The mean vector and variance vector, For linear network layers; Step 10: Calculate the approximate posterior distribution ; in, , For linear network layers, and These represent the residual position and relative scale of the posterior distribution relative to the prior Gaussian distribution, respectively. Step 11: Calculate the KL divergence loss. ; Step 12: Calculate the total loss ; in, This is the proportionality coefficient; Step 13: Repeat steps 3 through 12 until... The calculation ends when the value is less than the specified minimum loss. Step 14: For the query text, select the video with the highest text-to-video similarity score as the search result.

[0009] The technical concept of this invention is as follows: Gaussian smoothing aggregation suppresses cross-modal density mismatch and noise; combined with bottleneck layer and distribution alignment, discriminative semantic preservation and deep cross-modal alignment are achieved.

[0010] The beneficial effects of this invention are: it alleviates density mismatch and temporal noise, and improves retrieval accuracy; by maintaining the text discrimination structure and combining it with distribution alignment, it achieves deep cross-modal semantic unity and enhances retrieval accuracy. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a text-video retrieval method based on Gaussian smoothing and semantic preservation. Detailed Implementation

[0012] The invention will now be further described with reference to the accompanying drawings.

[0013] Reference Figure 1 A text-to-video retrieval method based on Gaussian smoothing and semantic preservation includes the following steps: Step 1: Define the text-video set as... ,in Represents the set of the first A text-video pair, This represents the total number of samples in the dataset. The text describes the information, where, Representative text description The first in vocabulary, express The number of words included; for Matching video data, among which Indicates video The first in Frame image, express Total number of frames included; In this embodiment, taking the MSR-VTT dataset as an example, a text-video pair set is constructed; each text, such as "a dog running in the park", corresponds to a video of about 10 seconds; the dataset size is set to about 10,000 entries, and the average text length is 15 words.

[0014] Step 2: Perform data preprocessing operations from the text-video collection. Randomly select a text-video pair First of all Perform equal-interval sampling to extract Frame images, as typical visual segments, are processed using a pre-trained CLIP visual encoder to process the sampled images. Frame images are feature-encoded to generate video. Frame-level visual feature vector sequence ,in Indicates the first The visual feature vector generated after the frame is encoded. The dimension parameter represents the space in which the feature vector resides; for text... The CLIP text encoder is used to perform semantic encoding operations to obtain... Text feature vectors and text word feature vectors ,in Representative text description The first in The feature vectors of the words; traversing the text-video dataset Following the above processing method, the visual feature vector, text feature vector, and text word feature vector of each text-video pair are obtained, thus obtaining the video feature vector set. Text feature vector set and text word feature vector set ; In this embodiment, 16 frames are extracted from the video at equal intervals, and the features of each frame are extracted using a pre-trained CLIP visual model; the text is encoded through the CLIP text model to obtain the overall semantic representation and fine-grained features of each word, and all features are uniformly mapped to a 512-dimensional space.

[0015] Step 3: In the text word feature vector set Choose any one of the text word feature vectors Generate the main feature vector of the text: ; in, It is a learnable query vector; it iterates through the set of text word feature vectors. This method yields the set of feature vectors for the main body of the text. ; In this embodiment, a learnable query vector is introduced to perform attention weighting on the features of each word in the text, automatically focusing on core semantic words such as "dog" and "running", and aggregating to generate feature vectors representing the main body of the text, which serve as semantic anchors for subsequent video matching.

[0016] Step 4: In the video feature vector set Choose any video feature vector and from the text main feature vector set Obtain the corresponding text main feature vector From the set of text feature vectors Obtain the corresponding text feature vector. ,calculate Mid-frame feature vector Attention score ; in, It is a multilayer perceptron. For activation functions; Next calculation Corresponding video global feature vector ; in, This indicates the original score sequence. Smoothing score after one-dimensional Gaussian filtering It is the first The score for attention in a frame. It is the radius of the sliding window. The standard deviation of the Gaussian kernel controls the smoothing range and degree; iterates through the set of video feature vectors. This method yields the set of global feature vectors for the video. ; In this embodiment, the matching score between the video features of each frame and the main text features is calculated to form a temporal score sequence. A one-dimensional Gaussian filter is used to smooth the score sequence, effectively suppressing semantic jumps caused by background abrupt changes or irrelevant frames. Weighted aggregation is then used to obtain a global video representation that fits the query intent.

[0017] Step 5: In the video feature vector set Choose any video feature vector and from the text feature vector set Obtain the corresponding text feature vector. ,calculate Corresponding video core features ; in, It is a set of learnable feature vectors. for The length of the vector. This indicates a splicing operation. For the encoder layer based on the multi-head self-attention mechanism, This indicates the output processed by the Transformer. From the vectors, extract the first element by index. One vector; Will and Concatenate the vectors and compute the text-enhanced feature vectors for cross-modal interaction. ; in, For the encoder layer based on the multi-head self-attention mechanism, Indicates the first All features from the position to the end; Next, the optimized text feature vector is calculated. ; in, For element-wise multiplication, A learnable scaling vector; traversing the set of video feature vectors This method yields an optimized set of text feature vectors. ; In this embodiment, video features are compressed through a lightweight bottleneck layer to extract core visual information; after being concatenated with the original text features, semantic enhancement is performed through a cross-modal encoder; finally, residual fusion is performed through a learnable scaling factor to supplement the visual context while maintaining the original discriminative semantic structure of the text.

[0018] Step 6: Calculate the text-to-video loss. ; in, Indicates the first Optimized text feature vectors With the A video global feature vector Cosine similarity; Step 7: Calculate the video-to-text loss. ; in, Indicates the first A video global feature vector With the Optimized text feature vectors Cosine similarity; In this embodiment, a contrastive learning framework is used to calculate the matching loss in both the "text→video" and "video→text" directions within a batch. By bringing positive sample pairs closer together and pushing negative sample pairs apart, the model is driven to learn a robust cross-modal alignment representation.

[0019] Step 8: Calculate self-attention visual features and dual-modal features ; ; in, This indicates a scaled dot product attention mechanism. and These represent the sets of video feature vectors respectively. and text word feature vector set All feature vectors in; In this embodiment, the video frame sequence and the text word sequence interact through a scaling dot product attention mechanism, enabling visual features to perceive corresponding semantics and text features to be associated with relevant visual regions.

[0020] Step 9: Calculate the prior Gaussian distribution ; in, , and They represent The mean vector and variance vector, For linear network layers; In this embodiment, self-attention features are input into a linear network to predict the mean and variance, so that the model can make predictions only in the case of video data.

[0021] Step 10: Calculate the approximate posterior distribution ; in, , For linear network layers, and These represent the residual position and relative scale of the posterior distribution relative to the prior Gaussian distribution, respectively. In this embodiment, the bimodal interaction features are input into a linear network to predict the mean residual and variance residual relative to the prior distribution. The distribution center and range are dynamically adjusted so that the model can predict the situation where both video data and text data are seen simultaneously.

[0022] Step 11: Calculate the KL divergence loss. ; In this embodiment, the KL divergence between the posterior and prior distributions is calculated as a regularization term to constrain the feature distribution after bimodal fusion to not deviate from the unimodal semantic prior, effectively bridging the modal gap between visual redundancy and text sparsity.

[0023] Step 12: Calculate the total loss ; in, This is the proportionality coefficient; In this embodiment, the bidirectional contrast loss and KL divergence loss are weighted and summed, with the contrast loss weight set to 1.0 and the distribution constraint weight set to 0.1. The AdamW optimizer is used with an initial learning rate of 1e-4, combined with a cosine annealing strategy to balance the goals of retrieval accuracy and distribution consistency.

[0024] Step 13: Repeat steps 3 through 12 until... The calculation ends when the value is less than the specified minimum loss. In this embodiment, steps three through twelve are executed cyclically, and the total loss value is calculated after each iteration. When the total loss first falls below a preset minimum loss threshold (e.g., 1e-4), training is immediately terminated, and the current model parameters are saved.

[0025] Step 14: For the query text, select the video with the highest text-to-video similarity score as the search result.

[0026] In this embodiment, after the user inputs the query text, the optimized text features are obtained through encoding. The cosine similarity is calculated with all global video features in the video library, and the Top-K results are returned in descending order of score.

[0027] In this embodiment, for the text-video set, the CLIP text encoder and video encoder are first used to extract feature vectors of the text and video. First, frame-level semantic matching and scoring are performed using the core text as anchor points. A one-dimensional Gaussian filter is introduced to smooth the temporal score sequence, suppressing inter-frame semantic jumps and background noise interference, generating a global video aggregation representation that fits the query semantics. For the query text (e.g., "a dog running in the park"), redundant video features are compressed through a visual bottleneck layer. A learnable and scalable residual fusion strategy is adopted to achieve cross-modal semantic supplementation while anchoring the original discriminative structure of the text, avoiding feature noise. A Gaussian probability model is used to model the feature distribution, and the consistency between the unimodal prior and bimodal posterior distributions is constrained to bridge the cross-modal semantic gap. Finally, the similarity score between the query text and the fused video features is calculated, and the video with the highest similarity score is selected from the video set. The retrieved result is the video that correctly corresponds to the query text.

[0028] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A text-video retrieval method based on Gaussian smoothing and semantic preservation, characterized in that, First, a frame-level semantic matching mechanism is constructed using the core text as the anchor point, and a one-dimensional Gaussian filter is introduced to smooth the temporal score sequence, effectively suppressing background noise interference and generating a global video aggregation representation that fits the query semantics. Secondly, by compressing redundant features through a visual bottleneck layer and combining a learnable scalable residual fusion strategy, cross-modal semantic supplementation is achieved while anchoring the original discriminative structure of the text to reduce feature noise. Finally, a Gaussian probability model is used to model the feature distribution, and the cross-modal semantic gap is bridged by constraining the consistency of the unimodal prior and bimodal posterior distributions, thus enabling video retrieval.

2. The text-video retrieval method based on Gaussian smoothing and semantic preservation as described in claim 1, characterized in that, The method includes the following steps: Step 1: Define the text-video set as... ,in Represents the set of the first A text-video pair, This represents the total number of samples in the dataset. The text describes the information, where, Representative text description The first in vocabulary, express The number of words included; for Matching video data, among which Indicates video The first in Frame image, express Total number of frames included; Step 2: Perform data preprocessing operations from the text-video collection. Randomly select a text-video pair First of all Perform equal-interval sampling to extract Frame images, as typical visual segments, are processed using a pre-trained CLIP visual encoder to process the sampled images. Frame images are feature-encoded to generate video. Frame-level visual feature vector sequence ,in Indicates the first The visual feature vector generated after the frame is encoded. The dimension parameter represents the space in which the feature vector resides; for text... The CLIP text encoder is used to perform semantic encoding operations to obtain... Text feature vectors and text word feature vectors ,in Representative text description The first in The feature vectors of the words; traversing the text-video dataset Following the above processing method, the visual feature vector, text feature vector, and text word feature vector of each text-video pair are obtained, thus obtaining the video feature vector set. Text feature vector set and text word feature vector set .

3. The text-video retrieval method based on Gaussian smoothing and semantic preservation as described in claim 2, characterized in that, The method further includes the following steps: Step 3: In the text word feature vector set Choose any one of the text word feature vectors Generate the main feature vector of the text; Step 4: In the video feature vector set Choose any video feature vector and from the text main feature vector set Obtain the corresponding text main feature vector From the set of text feature vectors Obtain the corresponding text feature vector. ,calculate Mid-frame feature vector Attention score Next, calculate. The corresponding global feature vector of the video; traversing the set of video feature vectors This method yields the set of global feature vectors for the video. ; Step 5: In the video feature vector set Choose any video feature vector and from the text feature vector set Obtain the corresponding text feature vector. ,calculate The corresponding core features of the video; and The text feature vectors are concatenated and then used to calculate the enhanced text feature vectors for cross-modal interaction. Next, the optimized text feature vectors are calculated, and the video feature vector set is iterated through. This method yields an optimized set of text feature vectors. ; Step 6: Calculate the text-to-video loss; Step 7: Calculate the video-to-text loss.

4. The text-video retrieval method based on Gaussian smoothing and semantic preservation as described in claim 3, characterized in that, The method further includes the following steps: Step 8: Calculate self-attention visual features and dual-modal features ; Step 9: Calculate the prior Gaussian distribution; Step 10: Calculate the approximate posterior distribution; Step 11: Calculate the KL divergence loss. ; Step 12: Calculate the total loss; Step 13: Repeat steps 3 through 12 until... The calculation ends when the value is less than the specified minimum loss value. Step 14: For the query text, select the video with the highest text-to-video similarity score as the search result.

Images