A method and system for video moment retrieval with frequency domain guidance enhancement

Abstract

The application discloses a kind of frequency domain guidance enhanced video time retrieval method and system.Input video data and extract visual features and audio features, carry out single mode coding to obtain the video features and audio features after coding;In frequency domain space, the video features and audio features after coding are guided using user query text features, to generate cross-modal features;Using text features and cross-modal features generate time alignment query features;Using query features and cross-modal features decoding generates the feature sequence used for video time retrieval;Using prediction head to process feature sequence, and construct loss function to train, output the result of video time retrieval.The application uses the cross-modal interaction method of frequency domain guidance, captures the commonality between different modalities in frequency domain space;A novel saliency enhancement function is designed, which not only realizes cross-modal fusion, but also has lightweight characteristics, significantly reduces the model parameter quantity and computational complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video moment retrieval, visual detection and video understanding, and in particular to a frequency domain guided enhanced video moment retrieval method and system. BACKGROUND

[0002] In recent years, with the rapid increase in the number of videos, video retrieval technology has become increasingly important. Traditional text retrieval methods are based only on keywords and cannot meet the needs of complex multimedia forms of video content. Video contains a large amount of image, sound and other information, therefore, simple text retrieval is limited. In order to more comprehensively and accurately meet user needs, video moment retrieval technology has emerged, providing a richer and more intelligent video search experience through deep multimedia analysis. This technology combines image recognition, speech recognition, semantic understanding and other advanced technologies, enabling users to more conveniently and quickly find the desired video content, bringing breakthroughs in the management and use of video content.

[0003] Video moment retrieval technology is a technology that, in the face of increasing and diverse video content, through deep analysis and index establishment, enables users to more conveniently retrieve and locate specific content in videos. With the rapid development of digital media, this technology has attracted much attention in many fields. Its core lies in video content analysis and moment positioning. Video content analysis involves image processing, audio processing, video feature extraction and other technologies, which deeply mine videos and extract key information such as people, scenes, objects, etc., thereby establishing a structured representation of the content. This enables the system to understand the context and meaning of the video, laying the foundation for subsequent retrieval. Moment positioning is a key link in video moment retrieval technology, which accurately locates relevant key information according to user queries, enabling users to directly locate specific video segments through text queries. This intelligent positioning not only provides an efficient search experience, but also enables users to quickly and accurately obtain the desired information. The research and application of video moment retrieval technology is crucial for improving the efficiency of video content management and utilization. With the continuous development of technology, video moment retrieval technology will further drive innovation in the field of multimedia information processing, providing users with more intelligent and personalized audio-visual experiences.

[0004] One of the current state-of-the-art techniques is the paper "Tall: Temporal activity localization via language query" which is based on a retrieval-ranking approach. This technique first generates candidate segments, then computes scores through interaction with the query statement, and finally generates retrieval results based on the scores. Through the preliminary screening of candidate segments, further optimization of ranking through interaction with the query is achieved, realizing more accurate content retrieval. This two-stage strategy not only improves retrieval efficiency, but also better meets user needs. The disadvantages of this technology are, first, the strategy of generating candidate segments may affect the accuracy of the results. Second, interaction with the query may lead to an increase in computational complexity. Finally, for complex queries, the ranking effect may be limited. Therefore, in the design, the efficiency and accuracy need to be balanced, and various query types need to be considered.

[0005] The second current state-of-the-art technique is the paper "4d spatio-temporal convnets: Minkowski convolutional neural networks" which is based on a proposal-based approach. This technique generates candidate time proposals, uses feature embedding and similarity calculation techniques, and finally realizes the sorting of time and outputs the results related to the query. Feature embedding and similarity calculation help improve the accuracy of ranking. The comprehensive use of this method enables the retrieval system to more accurately match user queries, providing an effective solution for time retrieval. The disadvantages of this technology are, first, it may be limited by the quality and diversity of proposals, affecting the comprehensiveness of candidate time. Second, feature embedding and similarity calculation may require high computational resources, leading to performance challenges on large-scale datasets. Finally, the ranking stage may be affected by noise data or outliers, affecting the accuracy of the final output results. Taking these factors into consideration, this technology may not perform as well in some complex situations.

[0006] The third existing technology is the paper "Temporally grounding language queries in videos by contextual boundary-aware prediction", which considers video temporal retrieval as a regression task. This technology directly models and learns the complex relationship between input features and relevant time by directly modeling and learning the complex relationship between input features and relevant time to obtain more accurate prediction results. This method bypasses the traditional retrieval-ranking stage and directly outputs the relevant time through the regression model, simplifying the process and improving efficiency. This strategy emphasizes the meticulous modeling of time relevance, bringing higher accuracy and practicality to the field of video retrieval. The disadvantages of this technology are: first, directly regressing time may ignore the complex context and semantic relationships in the video, resulting in the model not understanding the context information as well as other methods. Second, it has high requirements for training data, requiring a large amount of labeled time data, limiting the application of the method on large-scale datasets. Finally, the model may be sensitive to outliers, affecting the accuracy of the time. Therefore, when choosing a method, the applicable scenarios and the degree of dependence on data need to be considered comprehensively. SUMMARY

[0007] The purpose of the present invention is to overcome the shortcomings of existing methods and propose a frequency domain guided enhanced video temporal retrieval method and system. The main problems solved by the present invention are: first, how to convert different modal data into frequency domain features, capture complementary information in the frequency domain space, and realize more accurate cross-modal interaction; second, how to further quantify the saliency of features related to the query.

[0008] To solve the above problems, the present invention proposes a frequency domain guided enhanced video temporal retrieval method, which comprises:

[0009] An input video dataset is used to extract visual and audio features of the videos in the video dataset using a trained feature extractor, obtaining a video feature sequence V and an audio feature sequence A;

[0010] The video feature sequence V and the audio feature sequence A are encoded in a single modal to increase global context features in each modal, and are mapped to the same space using a linear layer, obtaining an encoded video feature sequence V' and an encoded audio feature sequence A';

[0011] The encoded video feature sequence V' and the encoded audio feature sequence A' are guided by the text features T of the user query text in the frequency domain space, and then the commonality between the modalities is captured in the frequency domain space and cross-modal interaction is realized, the video, audio and text features are fused and the reasoning fused cross-modal features O are generated;

[0012] The text feature T is input into a query generator together with the inference fused cross-modal feature O to generate a time-aligned query feature Q.

[0013] The query feature Q is input into a query decoder together with the inference fused cross-modal feature O to generate a feature sequence D for video time retrieval.

[0014] The feature sequence D is processed by a prediction head, and a loss function is constructed for training, and finally the result of video time retrieval, i.e., a time window d, is output. p .

[0015] Preferably, the input video dataset is used to extract visual features and audio features of the videos in the video dataset using the trained feature extractor, to obtain a video feature sequence V and an audio feature sequence A, specifically:

[0016] The video dataset is from a public dataset QVHighlights, and the video features are extracted using a SlowFast model and a feature extractor CLIP, and the audio features are extracted using a PANN model trained in advance on an AudioSet training set, to obtain a video feature sequence V and an audio feature sequence A.

[0017] Preferably, the video feature sequence V and the audio feature sequence A are single-modal encoded to increase global context features in each modality, respectively, and mapped to the same space using a linear layer, to obtain an encoded video feature sequence V' and an encoded audio feature sequence A', specifically:

[0018] A single-modal feature encoder based on a Transformer is used to process the video feature sequence V and the audio feature sequence A, respectively, to increase global context features in the video modality and the audio modality, respectively.

[0019] A linear layer is applied to the video feature sequence and the audio feature sequence with increased global context features, respectively, for dimension processing, to map them to the same space to obtain an encoded video feature sequence V' and an encoded audio feature sequence A', facilitating cross-modal interaction.

[0020] Preferably, the encoded video feature sequence V' and the encoded audio feature sequence A' are guided by the text feature T of the user query text in the frequency domain space, and then the commonality between modalities is captured in the frequency domain space and cross-modal interaction is realized, and video, audio and text features are fused to generate an inference fused cross-modal feature O, specifically:

[0021] The encoded video feature sequence V' and the encoded audio feature sequence A' are respectively mapped to the frequency domain space using Fourier transform to obtain a video frequency domain feature sequence V～ and an audio frequency domain feature sequence A ～ ;

[0022] The text feature T is extracted from the user query text using the trained feature extractor CLIP, and the text feature T is used to guide the video frequency domain feature sequence V ～ and the audio frequency domain feature sequence A ～ to obtain the guided video feature sequence V^ and the guided audio feature sequence A^, as shown in the following formula, wherein represents element-wise multiplication:

[0023]

[0024] The guided video feature sequence V^ and the guided audio feature sequence A^ are cross-modal fused in the frequency domain space, that is, the commonality between the video feature sequence and the audio feature sequence is captured in the frequency domain space, to obtain the frequency domain fusion feature C va , as shown in the following formula, wherein represents element-wise multiplication:

[0025]

[0026] The frequency domain fusion feature C va is quantified for saliency using a saliency enhancement function in the following formula, to obtain a saliency feature sequence P va :

[0027]

[0028] wherein u c and respectively represent the mean and unbiased estimated variance of the frequency domain fusion feature C va , and i, j represent the element subscript in the feature sequence;

[0029] The saliency feature sequence P va is used to adaptively integrate the encoded video feature sequence V' and the encoded audio feature sequence A' to obtain the inference fused cross-modal feature O:

[0030] gate=sigmoid(P va ),

[0031] O=V'gate+A'(1-gate),

[0032] that is, the sigmoid activation function is used to generate the weight gate, and then the weight gate is used to fuse the encoded video feature sequence V' and the encoded audio feature sequence A'.

[0033] Preferably, the text feature T is input into a query generator together with the inference fused cross-modal feature O to generate a time-aligned query feature Q, specifically:

[0034] The query generator of the paper "Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection" is introduced, and the text feature T is input into the query generator together with the inference fused cross-modal feature O, taking the inference fused cross-modal feature O as the query, the text feature T as the value and the key, and using multi-head attention to generate a time-aligned query feature Q to interact with the inference fused cross-modal feature O.

[0035] Preferably, the query feature Q is input into a query decoder together with the inference fused cross-modal feature O to generate a feature sequence D for video moment retrieval, specifically:

[0036] The query decoder of the paper "Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection" is introduced, and the query feature Q is input into the query decoder together with the inference fused cross-modal feature O to generate a feature sequence D for video moment retrieval.

[0037] Preferably, the feature sequence D is processed by the prediction head, and a loss function is constructed for training, and finally the result of video moment retrieval, i.e. the moment window d p , is output, specifically:

[0038] The prediction head maps the hidden layer of the feature sequence D to a one-dimensional fully connected mapping layer; a moment is represented by a center windowing method, where the center point is estimated by a prediction time heat map and extraction of a local optimal point, and the window is obtained from the center point feature regression and an offset is added to correct the result;

[0039] During training, for each center point c∈[1,N v ], the real moment with a window size of d, the center point is quantized to c'∈[0,1], and a one-dimensional Gaussian kernel is used to fill the time heat map, where σ c is the standard deviation of the window self-adaption, x is the time coordinate, and N v represents the number of video clips; and the Gaussian focus loss is used to define the loss of the center point as follows:

[0040]

[0041] Where N represents the number of time points, α and γ represent the weights and exponents of the modulation factor in the Gaussian focus loss, respectively, and H′ x Represents the predicted Gaussian kernel;

[0042] For the regression of the window and offset, the MAE loss function is used for optimization, as follows:

[0043]

[0044] Among them, w c w′ c o c o′ c These represent the actual time window, the predicted time window, the actual offset, and the predicted offset, respectively.

[0045] Therefore, the total training loss is expressed as:

[0046] L=λ c L c +λ w L w +λ o L o ,

[0047] Where λ {c、w、o} These are the weights for adjusting the center loss, window loss, and offset loss, with the time window d being the weights. p It is obtained from the center, window, and offset.

[0048] Accordingly, the present invention also proposes a frequency-domain guided enhanced video time-of-view retrieval system, comprising:

[0049] The video feature extraction unit is used to take a video dataset as input and use a trained feature extractor to extract the visual and audio features of the video in the video dataset, resulting in a video feature sequence V and an audio feature sequence A.

[0050] A single-modal coding unit is used to perform single-modal coding on the video feature sequence V and the audio feature sequence A to add global context features in each modality, and use a linear layer to map them to the same space to obtain the encoded video feature sequence V′ and the encoded audio feature sequence A′.

[0051] The cross-modal feature fusion unit is used for guiding the coded video feature sequence V' and the coded audio feature sequence A' by using the text feature T of the user query text in the frequency domain space, and then capturing the commonality between the modes and realizing the cross-modal interaction in the frequency domain space, fusing the video, audio and text features and generating the reasoning fused cross-modal feature O;

[0052] The query feature generation unit is used for inputting the text feature T and the reasoning fused cross-modal feature O into a query generator to generate the query feature Q aligned in time;

[0053] The query feature decoding unit is used for inputting the query feature Q and the reasoning fused cross-modal feature O into a query decoder to decode and generate the feature sequence D for video time retrieval;

[0054] The training and prediction unit is used for processing the feature sequence D by using a prediction head, training by constructing a loss function, and finally outputting the result of video time retrieval, i.e., the time window d p .

[0055] The present application has the following beneficial effects:

[0056] The present application adopts the cross-modal interaction method of frequency domain guidance, guides the video features and audio features by using the text in the frequency domain space, and captures the commonality between different modes in the frequency domain space; a novel saliency enhancement function is designed, which integrates multi-modal information, not only realizes cross-modal fusion, but also has the characteristics of light weight, significantly reduces the model parameter quantity and the calculation complexity. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a general flowchart of a frequency domain guidance enhanced video time retrieval method according to an embodiment of the present application;

[0058] Figure 2 is a flowchart of generating a fused cross-modal feature according to an embodiment of the present application.

[0059] Figure 3 is a structure diagram of a frequency domain guidance enhanced video time retrieval system according to an embodiment of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0061] Figure 1 is a general flowchart of a video moment retrieval method enhanced by frequency domain guidance according to an embodiment of the present application, as shown in the figure, the method comprises: Figure 1

[0062] S1, input a video dataset, extract visual features and audio features of videos in the video dataset using a trained feature extractor, to obtain a video feature sequence V and an audio feature sequence A;

[0063] S2, encode the video feature sequence V and the audio feature sequence A of S1 to respectively add global context features in each modality, and map to the same space using a linear layer, to obtain an encoded video feature sequence V' and an encoded audio feature sequence A';

[0064] S3, guide the encoded video feature sequence V' and the encoded audio feature sequence A' of S2 in the frequency domain space using text features T of a user query text, then capture the commonality between modalities in the frequency domain space and realize cross-modal interaction, fuse video, audio and text features and generate reasoning fused cross-modal features O;

[0065] S4, input the text features T of S3 and the reasoning fused cross-modal features O into a query generator to generate query features Q aligned in time;

[0066] S5, input the query features Q of S4 and the reasoning fused cross-modal features O of S3 into a query decoder for decoding to generate a feature sequence D for video moment retrieval;

[0067] S6, process the feature sequence D of S5 using a prediction head, construct a loss function for training, and finally output the result of video moment retrieval, i.e. a time window d p .

[0068] Step S1 is as follows:

[0069] The video dataset comes from the public dataset QVHighlights, the video features are extracted using the SlowFast model and the feature extractor CLIP, and the audio features are extracted using the PANN model trained in advance on the AudioSet training set, to obtain the video feature sequence V and the audio feature sequence A.

[0070] Step S2 is as follows:

[0071] S2-1, use a Transformer-based single-modal feature encoder to process the video feature sequence V and the audio feature sequence A respectively, to add global context features in the video modality and the audio modality respectively;

[0072] ​S2-2 applies a linear layer to the video feature sequence and audio feature sequence with added global context features to perform dimensionality processing, mapping them to the same space to obtain the encoded video feature sequence V′ and the encoded audio feature sequence A′, which facilitates cross-modal interaction.

[0073] Step S3, as follows Figure 2 As shown, the details are as follows:

[0074] S3-1, the video feature sequence V′ encoded in S2 and the audio feature sequence A′ encoded in S2 are respectively mapped to the frequency domain space using Fourier transform to obtain the video frequency domain feature sequence V. ～ and audio frequency domain feature sequence A ～ ;

[0075] S3-2, using the trained feature extractor CLIP to extract text features T from the user query text, and then using these text features T to extract the video frequency domain feature sequence V in the frequency domain space. ～ and audio frequency domain feature sequence A ～ The guided video feature sequence V^ and the guided audio feature sequence A^ are obtained, as shown in the following formula, where Indicates element-wise multiplication:

[0076]

[0077] S3-3, perform cross-modal fusion of the guided video feature sequence V^ and the guided audio feature sequence A^ in the frequency domain space, that is, capture the commonalities between the video feature sequence and the audio feature sequence in the frequency domain space to obtain the frequency domain fused feature C. va As shown in the following formula, where Indicates element-wise multiplication:

[0078]

[0079] S3-4, using the significance enhancement function in the following formula to perform frequency domain fusion on the feature C va Significance quantification is performed to obtain the significant feature sequence P. va :

[0080]

[0081] Among them, u c and Representing the frequency domain fusion features C respectively va The mean and unbiased estimate variance, where i and j represent the element indices in the feature sequence;

[0082] S3-5, using the significant feature sequence P vaAdaptively integrate the encoded video feature sequence V' and the encoded audio feature sequence A' to obtain the inferentially fused cross-modal feature O:

[0083] gate = sigmoid(P va ),

[0084] O = V' gate + A' (1 - gate),

[0085] That is, first generate the weight gate using the sigmoid activation function, and then fuse the encoded video feature sequence V' and the encoded audio feature sequence A' using the weight gate.

[0086] Step S4 is as follows:

[0087] The query generator in the paper "Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection" is introduced, and the text feature T of S3 and the inferentially fused cross-modal feature O are input into the query generator, the inferentially fused cross-modal feature O is taken as the query, the text feature T is taken as the value and the key, and the multi-head attention is used to generate the time-aligned query feature Q to interact with the inferentially fused cross-modal feature O.

[0088] Step S5 is as follows:

[0089] The query decoder in the paper "Umt: Unified multi-modal transformers for joint video moment retrieval and highlight detection" is introduced, and the query feature Q of S4 and the inferentially fused cross-modal feature O of S3 are input into the query decoder for decoding to generate the feature sequence D for video time retrieval.

[0090] Step S6 is as follows:

[0091] S6-1, the prediction head maps the hidden layer of the feature sequence D of S5 to a one-dimensional fully connected mapping layer; one time is represented by a center plus a window, wherein the center point is estimated by a predicted time heat map and extraction of a local optimal point, the window is obtained from the center point feature regression, and an offset is added to correct the result;

[0092] S6-2, in the process of training, for each center point c∈[1, N v, the center point is quantized to c' e [0, 1] and filled with a one-dimensional Gaussian kernel c is the standard deviation of the window adaptation, x is the time coordinate, N v represents the number of video clips; and the loss of the center point is defined by using a Gaussian focus loss as follows:

[0093]

[0094] wherein N represents the number of time points, and alpha and gamma respectively represent the weight and the index of the modulation factor in the Gaussian focus loss, which are respectively set to 2.0 and 4.0 in the embodiment, and H' x represents the predicted Gaussian kernel.

[0095] S6-3, for the regression of the window and the offset, an MAE loss function is used for optimization as follows:

[0096]

[0097] wherein w c , w' c , o c , o' c respectively represent the real time window, the predicted time window, the real offset and the predicted offset;

[0098] S6-4, therefore, the total training loss is represented as:

[0099] L = lambda c L c + lambda w L w + lambda o L o ,

[0100] wherein lambda {c、w、o} are respectively the weights for adjusting the center loss, the window loss and the offset loss, and the time window d p is obtained from the center, the window and the offset.

[0101] Correspondingly, the application also provides a video time retrieval system with frequency domain guided enhancement, as shown in the following figure, comprising: Figure 3

[0102] A video feature extraction unit 1 is used for inputting a video data set, extracting visual features and audio features of the videos in the video data set by using a trained feature extractor, and obtaining a video feature sequence V and an audio feature sequence A.

[0103] ​​a single-modal encoding unit 2 configured to encode the video feature sequence V and the audio feature sequence A in a single-modal manner to add global context features in each modality respectively, and map to the same space using a linear layer to obtain an encoded video feature sequence V' and an encoded audio feature sequence A';

[0104] a cross-modal feature fusion unit 3 configured to guide the encoded video feature sequence V' and the encoded audio feature sequence A' in a frequency domain space using text features T of a user query text, and then capture commonalities between modalities in the frequency domain space and realize cross-modal interaction, fuse video, audio and text features, and generate reasoning fusion cross-modal features O;

[0105] a query feature generation unit 4 configured to input the text features T and the reasoning fusion cross-modal features O into a query generator to generate time-aligned query features Q;

[0106] a query feature decoding unit 5 configured to input the query features Q and the reasoning fusion cross-modal features O into a query decoder to decode and generate a feature sequence D for video time retrieval;

[0107] a training and prediction unit 6 configured to process the feature sequence D using a prediction head, construct a loss function for training, and finally output a result of video time retrieval, i.e., a time window d p .

[0108] Therefore, the present application adopts a frequency domain guided cross-modal interaction method, guides video features and audio features in a frequency domain space using text, and captures commonalities between different modalities in the frequency domain space; a novel saliency enhancement function is designed to integrate multi-modal information, which not only realizes cross-modal fusion, but also has a lightweight feature, significantly reducing the model parameter quantity and computational complexity.

[0109] The above describes in detail a frequency domain guided enhanced video time retrieval method and system provided by an embodiment of the present application, and the principle and implementation manner of the present application are described by applying specific examples; the above embodiment description is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A frequency-domain guided enhanced video time-of-view retrieval method, characterized in that, The method includes: Input a video dataset, use a trained feature extractor to extract the visual and audio features of the videos in the dataset, and obtain the video feature sequence V and the audio feature sequence A; The video feature sequence V and the audio feature sequence A are unimodal encoded to add global context features to each modality, and then mapped to the same space using a linear layer to obtain the encoded video feature sequence V′ and the encoded audio feature sequence A′. In the frequency domain space, the text features T of the user query text are used to guide the encoded video feature sequence V′ and the encoded audio feature sequence A′. Then, in the frequency domain space, the commonalities between modes are captured and cross-modal interaction is realized. Video, audio and text features are fused and cross-modal features O of inference fusion are generated. The text feature T and the cross-modal feature O fused by the inference are input together into the query generator to generate the time-aligned query feature Q; The query feature Q and the cross-modal feature O fused by the inference are input together into the query decoder for decoding, generating a feature sequence D for video moment retrieval; The feature sequence D is processed using a prediction head, and a loss function is constructed for training. Finally, the result time window d of the video time retrieval is output. p ; Specifically, the process involves using the text features T of the user query text in the frequency domain to guide the encoded video feature sequence V′ and the encoded audio feature sequence A′, then capturing the commonalities between modalities in the frequency domain and achieving cross-modal interaction, fusing video, audio, and text features to generate inference-fused cross-modal features O, specifically: The encoded video feature sequence V′ and the encoded audio feature sequence A′ are respectively mapped to the frequency domain space using Fourier transform to obtain the video frequency domain feature sequence V. ～ and audio frequency domain feature sequence A ～ ; The trained feature extractor CLIP is used to extract text features T from the user query text, and these text features T are then used to analyze the video frequency domain feature sequence V in the frequency domain space. ～ and audio frequency domain feature sequence A ～ The guided video feature sequence V^ and the guided audio feature sequence A^ are obtained, as shown in the following formula, where Indicates element-wise multiplication: Cross-modal fusion is performed on the guided video feature sequence V^ and the guided audio feature sequence A^ in the frequency domain space. Specifically, commonalities between the video feature sequence and the audio feature sequence are captured in the frequency domain space to obtain the frequency domain fused feature C. va As shown in the following formula, where Indicates element-wise multiplication: The frequency domain fusion feature C is obtained by using the significance enhancement function in the following formula. va Significance quantification is performed to obtain the significant feature sequence P. va : Among them, u c and Representing the frequency domain fusion features C respectively va The mean and unbiased estimate variance, where i and j represent the element indices in the feature sequence; Using significant feature sequence P va Adaptively integrating the encoded video feature sequence V′ and the encoded audio feature sequence A′, we obtain the inference fusion cross-modal feature O: gate=sigmoid(P va ), O=V′·gate+A′(1-gate), First, the sigmoid activation function is used to generate a weight gate. Then, the weight gate is used to fuse the encoded video feature sequence V′ and the encoded audio feature sequence A′.

2. The video time-of-view retrieval method with frequency domain guidance enhancement as described in claim 1, characterized in that, The input video dataset is used to extract visual and audio features from the videos using a trained feature extractor, resulting in a video feature sequence V and an audio feature sequence A, specifically: The video dataset comes from the public dataset QVHighlights. Video features are extracted using the SlowFast model and the CLIP feature extractor, and audio features are extracted using the PANN model pre-trained on the AudioSet training set, resulting in video feature sequence V and audio feature sequence A, respectively.

3. The video time-of-view retrieval method with frequency domain guidance enhancement as described in claim 1, characterized in that, The video feature sequence V and the audio feature sequence A are unimodal encoded to add global context features to each modality, and then mapped to the same space using a linear layer to obtain the encoded video feature sequence V′ and the encoded audio feature sequence A′, specifically: A Transformer-based single-modal feature encoder is used to process the video feature sequence V and the audio feature sequence A separately, so as to add global context features in the video modality and the audio modality respectively; Linear layers are applied to the video feature sequences and audio feature sequences with added global context features to perform dimensionality processing, mapping them to the same space to obtain the encoded video feature sequence V′ and the encoded audio feature sequence A′, which facilitates cross-modal interaction.

4. The video time-of-view retrieval method with frequency domain guidance enhancement as described in claim 1, characterized in that, The text feature T and the cross-modal feature O fused by the inference are input together into the query generator to generate the time-aligned query feature Q, specifically: A query generator is introduced, and the text feature T and the cross-modal feature O fused by inference are input into the query generator. The cross-modal feature O fused by inference is used as the query, and the text feature T is used as the value and key. Multi-head attention is used to generate a time-aligned query feature Q, which is used to interact with the cross-modal feature O fused by inference.

5. The video time-of-view retrieval method with frequency domain guidance enhancement as described in claim 1, characterized in that, The query feature Q and the cross-modal feature O fused by inference are input together into the query decoder for decoding, generating a feature sequence D for video moment retrieval, specifically: A query decoder is introduced, and the query feature Q and the cross-modal feature O fused by the inference are input into the query decoder for decoding to generate a feature sequence D for video moment retrieval.

6. The video time-of-view retrieval method with frequency domain guidance enhancement as described in claim 1, characterized in that, The feature sequence D is processed using a prediction head, and a loss function is constructed for training. Finally, the result time window d of the video time retrieval is output. p Specifically: The prediction head maps the hidden layer of the feature sequence D to a one-dimensional fully connected mapping layer; a time step is represented by a center point plus a window, where the center point is estimated by predicting the time heatmap and extracting local optima, the window is obtained by regressing the center point features, and an offset is added to correct the result; During training, for each center point z∈[1,N] v At the real time with a window size of d, the center point is quantized to z′∈[0,1] and a one-dimensional Gaussian kernel is used. To fill the time heatmap, where σ z It is the standard deviation of window adaptation, x is the time coordinate, and N is the standard deviation of window adaptation. v The number of video segments is represented; and the loss for the center point is defined using Gaussian focus loss as follows: Where N represents the number of time points, α and γ represent the weights and exponents of the modulation factor in the Gaussian focus loss, respectively, and H′ x Represents the predicted Gaussian kernel; For the regression of the window and offset, the MAE loss function is used for optimization, as follows: Among them, w z w′ z o z o′ z These represent the actual time window, the predicted time window, the actual offset, and the predicted offset, respectively. Therefore, the total training loss is expressed as: L=λ z L z +λ w L w +λ o L o , Where λ z , λ w , λ o These are the weights for adjusting the center point loss, window loss, and offset loss, with the time window d being the weight. p It is obtained from the center point, window, and offset.

7. A frequency-domain guided enhanced video time-of-view retrieval system, characterized in that, The system includes: The video feature extraction unit is used to take a video dataset as input and use a trained feature extractor to extract the visual and audio features of the video in the video dataset, resulting in a video feature sequence V and an audio feature sequence A. A single-modal coding unit is used to perform single-modal coding on the video feature sequence V and the audio feature sequence A to add global context features in each modality, and use a linear layer to map them to the same space to obtain the encoded video feature sequence V′ and the encoded audio feature sequence A′. The cross-modal feature fusion unit is used to guide the encoded video feature sequence V′ and the encoded audio feature sequence A′ in the frequency domain space using the text features T of the user query text. Then, it captures the commonalities between modalities in the frequency domain space and realizes cross-modal interaction, fuses video, audio and text features and generates inference fusion cross-modal features O. The query feature generation unit is used to input the text feature T and the cross-modal feature O fused by inference into the query generator to generate a time-aligned query feature Q. The query feature decoding unit is used to input the query feature Q and the cross-modal feature O fused by inference into the query decoder for decoding, and generate a feature sequence D for video moment retrieval; The training and prediction unit processes the feature sequence D using the prediction head, constructs a loss function for training, and finally outputs the result time window d for video time retrieval. p ; Specifically, the cross-modal feature fusion unit is as follows: The encoded video feature sequence V′ and the encoded audio feature sequence A′ are respectively mapped to the frequency domain space using Fourier transform to obtain the video frequency domain feature sequence V. ～ and audio frequency domain feature sequence A ～ ; The trained feature extractor CLIP is used to extract text features T from the user query text, and these text features T are then used to analyze the video frequency domain feature sequence V in the frequency domain space. ～ and audio frequency domain feature sequence A ～ The process involves guidance to obtain the guided video feature sequence V. ^ and the guided audio feature sequence A ^ As shown in the following formula, where Indicates element-wise multiplication: Cross-modal fusion is performed on the guided video feature sequence V^ and the guided audio feature sequence A^ in the frequency domain space. Specifically, commonalities between the video feature sequence and the audio feature sequence are captured in the frequency domain space to obtain the frequency domain fused feature C. va As shown in the following formula, where Indicates element-wise multiplication: The frequency domain fusion feature C is obtained by using the significance enhancement function in the following formula. va Significance quantification is performed to obtain the significant feature sequence P. va : Among them, u c and Representing the frequency domain fusion features C respectively va The mean and unbiased estimate variance, where i and j represent the element indices in the feature sequence; Using significant feature sequence P va Adaptively integrating the encoded video feature sequence V′ and the encoded audio feature sequence A′, we obtain the inference fusion cross-modal feature O: gate=sigmoid(P va ), O=V′·gate+A′(1-gate), First, the sigmoid activation function is used to generate a weight gate. Then, the weight gate is used to fuse the encoded video feature sequence V′ and the encoded audio feature sequence A′.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images