A deep learning-based video moment retrieval method and system

Abstract

The application discloses a video time retrieval method based on deep learning. The method comprises the following steps: inputting a video data set and extracting a feature sequence by using a trained feature extractor; performing single-mode coding on the feature sequence by using a single-mode feature encoder; performing fusion reasoning on the feature sequence by using a statistical driven heterogeneous graph reasoning method; generating time-aligned query features by using a query generator; generating a feature sequence for video time retrieval by using a query decoder; and performing final result prediction by using a prediction head, and outputting the result of video time retrieval. The application further discloses a video time retrieval system based on deep learning. Compared with other applications, the application can complete efficient video time retrieval and video information positioning under low-cost hardware conditions, facilitate users to find the required video segments, improve the utilization value of the video and the user experience, and has great practical application value.

Description

Technical Field

[0001] This invention relates to the technical field of image understanding and video retrieval, and specifically to a video moment retrieval method utilizing deep learning technology. Background Technology

[0002] In recent years, with the surge in the number of videos, video retrieval technologies have also proliferated. Traditional video retrieval techniques are primarily based on text retrieval, using keywords to search for videos. However, because video is a complex multimedia format containing a large amount of image, sound, and other information, simple text retrieval often falls short of user needs. Therefore, video time-based retrieval technology has emerged.

[0003] Video moment retrieval is a technology that analyzes and indexes video content to make it easier for users to retrieve and locate specific content within videos. With the continuous increase and diversification of video content, the research and application of video moment retrieval technology are receiving increasing attention. The core of video moment retrieval technology is video content analysis and moment location. Video content analysis mainly includes image processing, audio processing, and video feature extraction techniques. By analyzing video content, key information such as people, scenes, and objects is extracted. Moment location, on the other hand, locates relevant key information based on user queries, allowing users to pinpoint specific video content through text searches.

[0004] One current technology is a content-based video retrieval technique proposed by Gao et al. in their paper "Tall: Temporal activity localization via language query". This technique analyzes and indexes video content to make it easier for users to retrieve and locate specific content within videos. It achieves intelligent video retrieval and location by performing complex processing and analysis on the audio and image content of videos, and using machine learning techniques for indexing and classification. The drawbacks of this method are: 1) The complex processing and analysis of video content requires significant computational resources, potentially leading to slow processing speeds, especially for large-scale video data; 2) Due to the diversity and complexity of video data, accurate classification and indexing require more complex algorithms and models, further increasing the difficulty and complexity of the technology.

[0005] The second existing technology is a video localization technique based on reinforcement learning, proposed by Anne Hendricks et al. in "Localizing moments in video with natural language." This technique analyzes videos using intelligent algorithms to determine key information and features. It learns and trains on video data, allowing the algorithm to automatically explore and learn key information and features in the video, thereby achieving video localization and search. The disadvantages of this method are: 1) It requires a large amount of training data and computational resources, which may lead to long training times, especially for large-scale video data; 2) The algorithm has high complexity, requiring a deep understanding and mastery of the principles and techniques of reinforcement learning algorithms; 3) Due to the diversity and complexity of video data, the algorithm needs continuous optimization and updates to improve its accuracy and performance.

[0006] The third existing technology is a video localization technique based on visual attention mechanisms, proposed by Ma et al. in their paper "Detecting moments and highlights in videos via natural language queries." This technique simulates the visual attention mechanism humans use when watching videos, automatically identifying areas of interest based on the visual importance of different regions in the video, thereby achieving rapid video localization and search. The disadvantages of this method are: 1) It requires processing large amounts of data and complex calculations, which may lead to slow processing speeds, especially for large-scale video data; 2) It has poor adaptability to various video content and struggles to handle diverse video content. It may not be able to accurately locate key areas in dynamic scenes and fast-moving videos, thus affecting the accuracy of localization and search. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing methods and propose a video moment retrieval method and system based on deep learning. The main problems addressed by this invention are: 1) how to overcome the shortcomings of existing video retrieval technologies and improve the speed, performance, and efficiency of video retrieval models; 2) how to accurately detect and locate specific moments in situations requiring the processing of diverse video content and involving large amounts of data and complex calculations, thereby improving the accuracy and performance of video retrieval.

[0008] To address the above problems, this invention proposes a deep learning-based video time-of-view retrieval method, the method comprising:

[0009] Input a video dataset and use a trained feature extractor to extract video and audio features from the video;

[0010] The Transformer single-modal feature encoder is used to encode the video features and audio features in the video separately to obtain the encoded video sequence and audio sequence.

[0011] Input the encoded video sequence and audio sequence, construct a heterogeneous graph structure, and then use a statistically driven heterogeneous graph inference method to fuse the video and audio data, and output the cross-modal feature sequence of inference fusion;

[0012] The query text and the cross-modal feature sequence fused by the inference are input into the query generator, which outputs the time-aligned query features.

[0013] The time-aligned query features and the cross-modal feature sequence fused by inference are input together into the query decoder for decoding, and the output is a feature sequence for video time-based retrieval.

[0014] The prediction head is used to perform final processing on the feature sequence used for video time-of-view retrieval, and the video time-of-view retrieval result is output.

[0015] Preferably, the input video dataset is used to extract video and audio features from the video using a trained feature extractor, specifically as follows:

[0016] The video dataset comes from the public dataset QVHighlights. The input includes video and audio information from the video, and pre-trained features of the video and audio are extracted using SlowFast and CLIP.

[0017] Preferably, the Transformer single-modal feature encoder is specifically:

[0018] A linear layer is used to process the dimensions of the video features and audio features respectively, and the processed video features and audio features are mapped to the same space to facilitate cross-modal interaction.

[0019] Preferably, the encoded video and audio sequences are input to construct a heterogeneous graph structure. Then, a statistically driven heterogeneous graph inference method is used to fuse the video and audio data, outputting a cross-modal feature sequence of inference fusion. Specifically:

[0020] The encoded video and audio sequences are concatenated, processed through two linear layers to obtain the generated node sequence, and then the heterogeneous graph adjacency matrix is ​​obtained through Hada code product.

[0021]

[0022] Where G represents the heterogeneous graph adjacency matrix and H represents the generated node sequence;

[0023] A statistically driven cross-modal interaction method is used to generate a fused feature sequence. This method uses a third-order Taylor series to construct a saliency enhancement function and uses this enhancement function to quantify the difference between the features and the mean, thereby measuring the saliency of the features. Then, an adaptive mechanism is used to adjust the information interaction weights between the encoded video sequence and the audio sequence to obtain the initial cross-modal feature sequence.

[0024] The method uses the initial cross-modal feature sequence as a communication bridge for heterogeneous nodes in the heterogeneous graph adjacency matrix, and uses a graph attention mechanism to enhance the local correlation between sequences, thus obtaining a locally enhanced cross-modal feature sequence.

[0025] Using a gating mechanism, an adaptive embedding is performed on the initial cross-modal feature sequence and the locally enhanced cross-modal feature sequence to obtain the inference fusion cross-modal feature sequence.

[0026] Preferably, the step of inputting the query text and the cross-modal feature sequence fused with the inference into the query generator, and outputting time-aligned query features, specifically involves:

[0027] Using the CLIP text encoder, tag-based features are extracted to generate text features;

[0028] A query generator is introduced, which feeds text features and inference-fused cross-modal feature sequences into the query generator. The inference-fused cross-modal feature sequences are used as queries, and text features are used as values ​​and keys. Multi-head attention is used to generate time-aligned query features, which are used to interact with video-audio fusion features.

[0029] Preferably, the step of using the prediction head to perform final processing on the feature sequence used for video time-of-view retrieval and outputting the video time-of-view retrieval result specifically involves:

[0030] The hidden layer of the decoded features is mapped to a one-dimensional fully connected mapping layer, including using a center plus window method to represent a time point. The center point is estimated by predicting the time heatmap and extracting local optima. The window can be further obtained from the center point features by regression, and an offset is added to correct the result.

[0031] During training, for each center point and a real time step with a window size of d, the center point is quantized to c′ and then processed using a 1D Gaussian kernel. To populate the time heatmap, Gaussian focus loss is used to define the loss at the center point, as follows:

[0032]

[0033] Where N represents the number of time points, and α and γ represent the weights and exponents of the modulation factor in the focus loss, with values ​​set to 2.0 and 4.0, respectively;

[0034] For the regression of the window and offset, the MAE loss function is used for optimization, as follows:

[0035]

[0036]

[0037] Among them, w c Represents the actual time window, w c ′ represents the predicted time window, o c Represents the actual offset, o′ c Indicates the predicted offset;

[0038] The total training loss can be expressed as:

[0039] L=λ c L c +λ w L w +λ o L o

[0040] Where, λ {c、w、o} These represent the weights of the center loss, window loss, and offset loss, respectively. During testing, the time boundaries are obtained from the center, window, and offset.

[0041] The trained model was tested on the test set to obtain the final relevant video time window located based on the text data.

[0042] Accordingly, the present invention also provides a video time-of-view retrieval system based on deep learning, comprising:

[0043] The feature extraction unit is used to input a video dataset and extract video and audio features from the video using a trained feature extractor.

[0044] A single-modal coding unit is used to perform single-modal coding on the video features and audio features in the video using a Transformer single-modal feature encoder, respectively, to obtain the encoded video sequence and audio sequence;

[0045] The fusion inference unit is used to input the encoded video sequence and audio sequence, construct a heterogeneous graph structure, and then use a statistically driven heterogeneous graph inference method to fuse the video and audio data, and output the cross-modal feature sequence of inference fusion.

[0046] The query generation unit is used to input the query text and the cross-modal feature sequence fused by the inference into the query generator, and output the time-aligned query features.

[0047] The query decoding unit is used to input the time-aligned query features and the cross-modal feature sequence fused by inference into the query decoder for decoding, and output a feature sequence for video time-retrieval.

[0048] The result prediction unit is used to perform final processing on the feature sequence used for video time-of-view retrieval using the prediction head, and output the result of video time-of-view retrieval.

[0049] Implementing this invention has the following beneficial effects:

[0050] This invention employs a statistically driven cross-modal interaction method. First, a novel significance enhancement function is designed using a third-order Taylor expansion. This function quantifies the importance of eigenvalues ​​by the difference between their values ​​and the population mean. Then, based on this enhancement function, a statistically driven cross-modal interaction mechanism is designed. This mechanism not only enables cross-modal fusion but also possesses lightweight characteristics, significantly reducing the number of model parameters and computational complexity.

[0051] This invention employs a heterogeneous graph reasoning method, which generates a heterogeneous graph adjacency matrix through video and audio modal interaction, further constructs a heterogeneous graph network, and utilizes a graph attention mechanism to enhance the local correlation between feature sequences, thereby improving the model's reasoning ability.

[0052] This invention employs an end-to-end approach, combining all model components and processing steps into a unified whole, thereby simplifying model design and implementation. Compared to traditional multi-stage models, the end-to-end approach better utilizes the inherent structure and relationships of data, thus improving model performance and efficiency. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the overall process of a deep learning-based video moment retrieval method according to an embodiment of the present invention.

[0054] Figure 2 This is a flowchart of the cross-modal feature generation and inference fusion according to an embodiment of the present invention;

[0055] Figure 3 This is a system architecture diagram of video time-of-view retrieval based on deep learning, according to an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Figure 1 This is a flowchart illustrating the overall process of a deep learning-based video time-of-view retrieval method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0058] S1, Input dataset and extract features. Input a video dataset and use a trained feature extractor to extract video and audio features from the video;

[0059] S2, perform single-modal encoding on the feature sequence. Using the Transformer single-modal feature encoder, single-modal encoding is performed on the video features and audio features in the video respectively to obtain the encoded video sequence and audio sequence;

[0060] S3, perform fusion inference using a statistically driven heterogeneous graph inference method. Input the encoded video and audio sequences, construct a heterogeneous graph structure, and then use the statistically driven heterogeneous graph inference method to fuse the video and audio data, outputting a cross-modal feature sequence of the inference fusion.

[0061] S4, Generate a time-aligned query. Input the query text along with the cross-modal feature sequence fused by the inference into the query generator, and output the time-aligned query features;

[0062] S5, Decode the features to generate decoded features. The time-aligned query features and the inference-fused cross-modal feature sequence are input into the query decoder for decoding, outputting a feature sequence for video time-based retrieval;

[0063] S6, perform final result prediction. The prediction head is used to perform final processing on the feature sequence used for video time-of-view retrieval, and the video time-of-view retrieval result is output.

[0064] Step S1 is as follows:

[0065] S1-1, the obtained video dataset comes from the public dataset QVHighlights, and the input includes video information and audio information from the video.

[0066] S1-2, In order to facilitate subsequent processing of the model, this invention uses SlowFast and CLIP to extract pre-trained features of video and audio.

[0067] Step S2 is as follows:

[0068] S2-1 uses a Transformer unimodal feature encoder to process the input visual and audio features separately, so as to add global contextual features individually in each modality.

[0069] S2-2, a linear layer is used to process the video features and audio features separately, and the processed video features and audio features are mapped to the same space to facilitate cross-modal interaction.

[0070] The specific process of step S3 is as follows: Figure 2 As shown, the details are as follows:

[0071] S3-1, Construct a heterogeneous graph structure using the video and audio modal sequences generated in step S2. Concatenate the encoded video and audio sequences, process them through two linear layers to obtain the generated node sequence, and then obtain the heterogeneous graph adjacency matrix through Hada code product.

[0072]

[0073] Where G represents the adjacency matrix of the heterogeneous graph and H represents the sequence of generated nodes.

[0074] S3-2, a statistically driven cross-modal interaction method is used to generate a fused feature sequence. This method uses a third-order Taylor to construct a saliency enhancement function and uses this enhancement function to quantify the difference between the feature and the mean, thereby measuring the saliency of the feature. Then, an adaptive mechanism is used to adjust the information interaction weight between the encoded video sequence and the audio sequence to obtain the initial cross-modal feature sequence.

[0075] S3-3, due to the heterogeneous gap between different modalities, the fused cross-modal feature sequence is used as a communication bridge for heterogeneous nodes. Therefore, the initial cross-modal feature sequence obtained in S3-2 is used as a communication bridge for heterogeneous nodes in the heterogeneous graph adjacency matrix in S3-1, and the graph attention mechanism (GAT) is used to enhance the local correlation between sequences, resulting in a locally enhanced cross-modal feature sequence.

[0076] In S3-4, using a gating mechanism, an adaptive embedding is performed on the initial cross-modal feature sequence obtained in S3-2 and the locally enhanced cross-modal feature sequence obtained in S3-3 to obtain the cross-modal feature sequence for inference fusion.

[0077] Step S4 is as follows:

[0078] S4-1 utilizes the CLIP text encoder to extract tag-based features to generate text features.

[0079] In section S4-2, a query generator is introduced. Text features and the cross-modal feature sequence fused from inference are fed into the query generator. The cross-modal feature sequence from inference fusion is used as the query, and the text features are used as the value and key. Multi-head attention is employed to generate time-aligned query features. The generated time-aligned query features are then used to interact with the video-audio fusion features.

[0080] Step S5 is as follows:

[0081] S5-1: Using the time-aligned query features generated in step S4 and the cross-modal feature sequence fused with inference generated in step S3 as input, the video features are decoded, and the decoded features are used as the prediction of the video time-based retrieval results.

[0082] Step S6 is as follows:

[0083] S6-1, introduces a prediction head, including mapping the hidden layer of the decoded features to a one-dimensional fully connected mapping layer. This invention uses a center plus window method to represent a time point, where the center point is estimated by predicting the time heatmap and extracting local optima, and the window can be further obtained from the center point features by regression, and an offset is added to correct the result.

[0084] S6-2, during training, for each center point and a window size of d at a real time, the center point is quantized to c′ and a 1D Gaussian kernel is used. To populate the time heatmap, Gaussian focus loss is used to define the loss at the center point, as follows:

[0085]

[0086] Where N represents the number of time points, and α and γ represent the weights and exponents of the modulation factor in the focus loss, with values ​​set to 2.0 and 4.0, respectively;

[0087] S6-3, For the regression of window and offset, the MAE loss function is used for optimization, as follows:

[0088]

[0089]

[0090] Among them, w c w′ represents the actual time window. c Indicates the predicted time window, o c Represents the actual offset, o′ c Indicates the predicted offset;

[0091] S6-4, the total training loss can be expressed as:

[0092] L=λ c L c +λ w L w +λ o L o

[0093] Where, λ {c、w、o} These represent the weights of the center loss, window loss, and offset loss, respectively. During testing, the time boundaries are obtained from the center, window, and offset.

[0094] S6-5 uses the trained model to conduct experimental testing on the test set to obtain the final relevant video time window located based on the text data.

[0095] Accordingly, the present invention also provides a video time-of-view retrieval system based on deep learning, such as... Figure 3 As shown, it includes:

[0096] Feature extraction unit 1 is used to input a video dataset and extract video and audio features from the video using a trained feature extractor.

[0097] The single-modal coding unit 2 is used to perform single-modal coding on the video features and audio features in the video using the Transformer single-modal feature encoder, respectively, to obtain the encoded video sequence and audio sequence;

[0098] The fusion inference unit 3 is used to input the encoded video sequence and audio sequence, construct a heterogeneous graph structure, and then use a statistically driven heterogeneous graph inference method to fuse the video and audio data, and output the cross-modal feature sequence of inference fusion.

[0099] The query generation unit 4 is used to input the query text and the cross-modal feature sequence fused by the reasoning into the query generator and output the time-aligned query features.

[0100] The query decoding unit 5 is used to input the time-aligned query features and the cross-modal feature sequence fused by inference into the query decoder for decoding, and output a feature sequence for video time-retrieval.

[0101] The result prediction unit 6 is used to perform final processing on the feature sequence used for video time-of-view retrieval using the prediction head, and output the result of video time-of-view retrieval.

[0102] Therefore, this invention employs a statistically driven cross-modal interaction method. First, a novel significance enhancement function is designed using a third-order Taylor expansion. This function quantifies the importance of eigenvalues ​​based on the difference between their values ​​and the population mean. Then, based on this enhancement function, a statistically driven cross-modal interaction mechanism is designed. This mechanism not only enables cross-modal fusion but also possesses lightweight characteristics, significantly reducing the number of model parameters and computational complexity. This invention also employs a heterogeneous graph inference method. A heterogeneous graph adjacency matrix is ​​generated through video and audio modal interaction, further constructing a heterogeneous graph network. A graph attention mechanism is used to enhance the local correlations between feature sequences, improving the model's inference ability. Furthermore, this invention adopts an end-to-end approach, combining all model components and processing steps into a unified whole, thereby simplifying model design and implementation. Compared to traditional multi-stage models, the end-to-end approach better utilizes the inherent structure and correlations of data, thus improving model performance and efficiency.

[0103] The above provides a detailed description of a video moment retrieval method and system based on deep learning provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A video time-of-view retrieval method based on deep learning, characterized in that, The method includes: Input a video dataset and use a trained feature extractor to extract video and audio features from the video; The Transformer single-modal feature encoder is used to encode the video features and audio features in the video separately to obtain the encoded video sequence and audio sequence. Input the encoded video sequence and audio sequence, construct a heterogeneous graph structure, and then use a statistically driven heterogeneous graph inference method to fuse the video and audio data, and output the cross-modal feature sequence of inference fusion; The query text and the cross-modal feature sequence fused by the inference are input into the query generator, which outputs the time-aligned query features. The time-aligned query features and the cross-modal feature sequence fused by inference are input together into the query decoder for decoding, and the output is a feature sequence for video time-based retrieval. The prediction head is used to perform final processing on the feature sequence used for video time-of-view retrieval, and the video time-of-view retrieval result is output. Specifically, the encoded video and audio sequences are input to construct a heterogeneous graph structure. Then, a statistically driven heterogeneous graph inference method is used to fuse the video and audio data, outputting a cross-modal feature sequence of inference fusion. The encoded video and audio sequences are concatenated, processed through two linear layers to obtain the generated node sequence, and then the heterogeneous graph adjacency matrix is ​​obtained through Hada code product. Where G represents the heterogeneous graph adjacency matrix and H represents the generated node sequence; A statistically driven cross-modal interaction method is used to generate a fused feature sequence. A saliency enhancement function is constructed using a third-order Taylor series and used to quantify the difference between the features and the mean, thereby measuring the saliency of the features. Subsequently, an adaptive mechanism is used to adjust the information interaction weights between the encoded video sequence and the audio sequence to obtain the initial cross-modal feature sequence. The fused cross-modal feature sequence is used as a communication bridge between heterogeneous nodes. The initial cross-modal feature sequence is used as a communication bridge between heterogeneous nodes in the heterogeneous graph adjacency matrix. The graph attention mechanism is used to enhance the local correlation between sequences, resulting in a locally enhanced cross-modal feature sequence. Using a gating mechanism, an adaptive embedding is performed on the initial cross-modal feature sequence and the locally enhanced cross-modal feature sequence to obtain the inference fusion cross-modal feature sequence.

2. The video time-of-view retrieval method based on deep learning as described in claim 1, characterized in that, The input video dataset is used, and a trained feature extractor is used to extract video and audio features from the video, specifically: The video dataset comes from the public dataset QVHighlights. The input includes video and audio information from the video, and pre-trained features of the video and audio are extracted using SlowFast and CLIP.

3. The video time-of-view retrieval method based on deep learning as described in claim 1, characterized in that, The Transformer single-modal feature encoder is specifically as follows: A linear layer is used to process the dimensions of the video features and audio features respectively, and the processed video features and audio features are mapped to the same space to facilitate cross-modal interaction.

4. The video time-of-view retrieval method based on deep learning as described in claim 1, characterized in that, The process of inputting the query text and the cross-modal feature sequence fused with the inference into the query generator, and outputting time-aligned query features, specifically involves: Using the CLIP text encoder, tag-based features are extracted to generate text features; A query generator is introduced, which feeds text features and inference-fused cross-modal feature sequences into the query generator. The inference-fused cross-modal feature sequences are used as queries, and text features are used as values ​​and keys. Multi-head attention is used to generate time-aligned query features, which are used to interact with video-audio fusion features.

5. The video time-of-view retrieval method based on deep learning as described in claim 1, characterized in that, The process of using a prediction head to perform final processing on the feature sequence used for video time-of-view retrieval and outputting the video time-of-view retrieval result is as follows: The hidden layer of the decoded features is mapped to a one-dimensional fully connected mapping layer, including using a center plus window method to represent a time point. The center point is estimated by predicting the time heatmap and extracting local optima, the window is obtained by regressing the center point features, and an offset is added to correct the result. During training, for each center point and a real time step with a window size of d, the center point is quantized to c′ and then processed using a 1D Gaussian kernel. To populate the time heatmap, Gaussian focus loss is used to define the loss at the center point, as follows: Where N represents the number of time points, and α and γ represent the weights and exponents of the modulation factor in the focus loss, with values ​​set to 2.0 and 4.0, respectively; For the regression of the window and offset, the MAE loss function is used for optimization, as follows: Among them, w c Represents the actual time window, w c ′ represents the predicted time window, o c Represents the actual offset, o c ′ represents the predicted offset; The total training loss can be expressed as: L=λ c L c +λ w L w +λ o L o Where, λ {c、w、o} These represent the weights of the center loss, window loss, and offset loss, respectively. During testing, the time boundaries are obtained from the center, window, and offset. The trained model was tested on the test set to obtain the final relevant video time window located based on the text data.

6. A video time-of-view retrieval system based on deep learning, characterized in that, The system includes: The feature extraction unit is used to input a video dataset and extract video and audio features from the video using a trained feature extractor. A single-modal coding unit is used to perform single-modal coding on the video features and audio features in the video using a Transformer single-modal feature encoder, respectively, to obtain the encoded video sequence and audio sequence; The fusion inference unit is used to input the encoded video sequence and audio sequence, construct a heterogeneous graph structure, and then use a statistically driven heterogeneous graph inference method to fuse the video and audio data, and output the cross-modal feature sequence of inference fusion. The query generation unit is used to input the query text and the cross-modal feature sequence fused by the inference into the query generator, and output the time-aligned query features. The query decoding unit is used to input the time-aligned query features and the cross-modal feature sequence fused by inference into the query decoder for decoding, and output a feature sequence for video time-retrieval. The result prediction unit is used to perform final processing on the feature sequence used for video moment retrieval using the prediction head, and output the result of video moment retrieval. Specifically, the fusion reasoning unit is: The encoded video and audio sequences are concatenated, processed through two linear layers to obtain the generated node sequence, and then the heterogeneous graph adjacency matrix is ​​obtained through Hada code product. Where G represents the heterogeneous graph adjacency matrix and H represents the generated node sequence; A statistically driven cross-modal interaction method is used to generate a fused feature sequence. A saliency enhancement function is constructed using a third-order Taylor series and used to quantify the difference between the features and the mean, thereby measuring the saliency of the features. Subsequently, an adaptive mechanism is used to adjust the information interaction weights between the encoded video sequence and the audio sequence to obtain the initial cross-modal feature sequence. The fused cross-modal feature sequence is used as a communication bridge between heterogeneous nodes. The initial cross-modal feature sequence is used as a communication bridge between heterogeneous nodes in the heterogeneous graph adjacency matrix. The graph attention mechanism is used to enhance the local correlation between sequences, resulting in a locally enhanced cross-modal feature sequence. Using a gating mechanism, an adaptive embedding is performed on the initial cross-modal feature sequence and the locally enhanced cross-modal feature sequence to obtain the inference fusion cross-modal feature sequence.

7. A video time-of-view retrieval system based on deep learning as described in claim 6, characterized in that, The Transformer single-modal feature encoder is specifically as follows: A linear layer is used to process the dimensions of the video features and audio features respectively, and the processed video features and audio features are mapped to the same space to facilitate cross-modal interaction.

8. A video time-of-view retrieval system based on deep learning as described in claim 6, characterized in that, The process of inputting the query text and the cross-modal feature sequence fused with the inference into the query generator, and outputting time-aligned query features, specifically involves: Using the CLIP text encoder, tag-based features are extracted to generate text features; A query generator is introduced, which feeds text features and inference-fused cross-modal feature sequences into the query generator. The inference-fused cross-modal feature sequences are used as queries, and text features are used as values ​​and keys. Multi-head attention is used to generate time-aligned query features, which are used to interact with video-audio fusion features.

9. A video time-of-view retrieval system based on deep learning as described in claim 6, characterized in that, The process of using a prediction head to perform final processing on the feature sequence used for video time-of-view retrieval and outputting the video time-of-view retrieval result is as follows: The hidden layer of the decoded features is mapped to a one-dimensional fully connected mapping layer, including using a center plus window method to represent a time point. The center point is estimated by predicting the time heatmap and extracting local optima, the window is obtained by regressing the center point features, and an offset is added to correct the result. During training, for each center point and a real time step with a window size of d, the center point is quantized to c′ and then processed using a 1D Gaussian kernel. To populate the time heatmap, Gaussian focus loss is used to define the loss at the center point, as follows: Where N represents the number of time points, and α and γ represent the weights and exponents of the modulation factor in the focus loss, with values ​​set to 2.0 and 4.0, respectively; For the regression of the window and offset, the MAE loss function is used for optimization, as follows: Among them, w c Represents the actual time window, w c ′ represents the predicted time window, o c Represents the actual offset, o c ′ represents the predicted offset; The total training loss can be expressed as: L=λ c L c +λ w L w +λ o L o Where, λ {c、w、o} These represent the weights of the center loss, window loss, and offset loss, respectively. During testing, the time boundaries are obtained from the center, window, and offset. The trained model was tested on the test set to obtain the final relevant video time window located based on the text data.

Images