A method and system for video moment retrieval and highlight clip detection

Abstract

The application discloses a video moment retrieval and highlight segment detection method, which comprises the following steps: extracting original features in a video by using a pre-trained feature extraction network, passing the original features through a feature activation layer and a hole convolution layer, and performing cross-modal feature fusion by using a convolution-driven multi-modal fusion module; generating negative sample pairs by using given video-text pairs, and assisting training by using the difference between the positive and negative samples; calculating the interval of moment retrieval and the saliency score of highlight detection by using a saliency prediction head and a feature decoder, calculating a loss, and generating an overall loss function of the model; and finally, training and optimizing the model by using the loss, and generating a prediction result. The application makes full use of the complementarity between different modalities, thereby improving the understanding and representation ability of the model for data. Compared with a traditional model, the method adopted by the application significantly reduces the calculation amount and the parameter amount, can make full use of the complementarity between modalities, and improves the accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and video retrieval, and particularly relates to a video moment retrieval and highlight segment detection method and system. BACKGROUND

[0002] With the explosive growth of video data, how to efficiently retrieve relevant moments from massive videos and detect highlight moments in videos has become a problem to be solved. Video moment retrieval refers to retrieving a time segment containing a specific event or action from a given long video. Highlight detection is to identify the exciting moments of interest from the video, which is usually applied to sports events, game live streaming and other scenes. Traditional video retrieval and highlight detection methods usually rely on manual annotation and feature extraction, which have high computational complexity, poor real-time performance, high annotation cost and other problems, and are difficult to meet the actual application requirements.

[0003] In recent years, with the rise of deep learning technology, a new solution has been provided for video moment retrieval and highlight detection technology. Existing deep learning methods usually use complex attention mechanisms for feature extraction and interaction of video and text, which introduces a huge number of parameters and high hardware cost. In addition, text features play a decisive role in moment retrieval, while current research methods use mixed weighting or late fusion methods to interact with text features, which do not fully play the important role of text features in cross-modal feature fusion.

[0004] Therefore, the present application proposes a convolution-driven joint video moment retrieval and highlight detection method, which uses a convolution-based cross-modal feature fusion mechanism to replace the complex attention layer to fuse the encoded video and text features, fully utilizes the complementarity between modalities, improves the understanding and representation ability of the model to data, and obtains lower hardware cost and better performance.

[0005] One of the current prior arts is a video moment retrieval model Moment-DETR based on Transformer architecture in Lei J et al. Qvhighlights: Detecting moments and highlights in videos via natural language queries, which is composed of a feature encoder, a temporal encoder, a query generator and a decoder module, uses attention mechanism in deep learning for efficient moment retrieval, wherein the feature encoder is responsible for extracting the multi-modal feature representation of the video, the temporal encoder models the video features in time sequence, the query generator generates the query vector required for retrieval, and the decoder module interacts the query vector with the video features to finally generate accurate moment retrieval results. The disadvantage of this model is that the video features and the text features are only simply spliced in the fusion process, resulting in confusion in feature fusion, thereby affecting the performance of the model, and the computational load of the model is large and the hardware cost is high.

[0006] The second current prior art is a query-dependent detection model QD-DETR based on attention mechanism in Moon W J et al. Query-dependent video representation for moment retrieval and highlight detection, which is composed of a feature encoder, a query generator, a decoder module and a loss calculation module, and uses a query vector to guide target detection. Among them, the feature encoder is responsible for extracting the visual feature representation of the image or video, the query generator generates the query vector related to the target detection task, the decoder module generates the target detection result by interacting with the visual features, and the loss calculation module is used to optimize the detection accuracy of the model, so as to obtain more accurate target detection result. The disadvantage of this model is that the model focuses on the complex query-guided attention mechanism, pays attention to the global features, ignores the important influence of local features on the performance, and the computational load and parameter quantity of the model are large, and the hardware cost is high. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings of the existing method, and a video moment retrieval and highlight segment detection method and system are proposed. The main problems solved by the present application are: 1) how to overcome the problem that the existing video moment retrieval and highlight detection method pays too much attention to the complex attention mechanism and the dependence between global features, thereby introducing a large number of parameters and computational load, which is low in efficiency and not suitable for large-scale popularization in practical applications; 2) how to use a multi-modal feature fusion mechanism driven by convolution to replace the attention mechanism, while ensuring the continuity of local features and fully utilizing the role of multi-modal features, so as to obtain a model with high accuracy and light weight.

[0008] To solve the above problems, the present application provides a video time retrieval and highlight segment detection method, which comprises:

[0009] Inputting a data set, using a pre-trained feature extraction network to extract original features in the video, including video features and text features in the video;

[0010] Using a convolution-driven multi-modal fusion module to input the extracted original features in the video, passing through a feature activation layer and a dilated convolution layer, and outputting fused cross-modal features;

[0011] Inputting the extracted original features in the video, generating negative sample pairs using a given video-text pair, and finally generating a negative sample set, so that the association between the video and the text has uniqueness and the diversity of the training samples is increased;

[0012] Inputting the fused cross-modal features and the negative sample set, calculating the interval of time retrieval and the saliency score of highlight detection through a saliency prediction head and a feature decoder, and calculating the loss of time retrieval and highlight detection, respectively, to generate a model overall loss function;

[0013] Using the model overall loss function to optimize the joint video time retrieval and highlight detection model through loss training, and outputting the trained model and the final time retrieval prediction and highlight detection prediction results.

[0014] Preferably, the input data set uses a pre-trained feature extraction network to extract original features in the video, including video features and text features in the video, specifically:

[0015] The input data set contains visual information and text information in the video, and a video encoder composed of a pre-trained feature extraction network SlowFast and a CLIP is used to extract original video features A pre-trained CLIP text encoder is used to extract original text features Wherein N v and N t represent the lengths of the video and text feature sequences, respectively.

[0016] Preferably, the convolution-driven multi-modal fusion module inputs the extracted original features in the video, passes through a feature activation layer and a dilated convolution layer, and outputs fused cross-modal features, specifically:

[0017] The extracted original video features and text features The text feature T is compressed and expanded in the feature activation layer, so as to fully utilize the complementarity between the modalities to capture the interaction between more complex multi-modal features; three different feature compression methods, namely average pooling, maximum pooling and GRU, are respectively used, the processed features are expanded to the same dimension as the video feature V, and then are spliced with the video feature and mapped back to the original dimension N of the video feature v , and the formula is as follows:

[0018] X = Linear (Concat [V, T'])

[0019] wherein, X is the processed fusion feature, T' represents the text feature compressed and expanded by one of the feature compression methods, Concat represents the splicing operation, and Linear represents the full connection layer;

[0020] Through the mutual embedding of the video feature and the text feature, the specific moment region of the visual feature is fully activated under the guidance of the text feature, so that the visual information and the text information in the video can be more accurately reflected;

[0021] The processed fusion feature X is input, X is input into a multi-layer dilated convolution network layer, i.e. an interval (or called dilated rate) is introduced between the convolution kernels to expand the receptive field, so as to increase the perception ability of the network, and at the same time, in order to reduce overfitting and solve the gradient vanishing problem, BatchNorm and Relu layers are added in the middle of the convolution layer, and the initial input x (0) = X, and the calculation process between the convolution layers is as follows:

[0022] z (i) = Conv (x (i-1) )

[0023] z (i) = BatchNorm (z (i) )

[0024] x (i) = Relu (z (i) )

[0025] wherein, G = x (i) is a multi-modal feature, and represents the output of the i-th layer, and the output feature In the multi-layer convolution operation, the interaction information of the video feature and the text feature is transmitted layer by layer, so as to realize more fine inter-modal fusion;

[0026] The multi-modal feature G is input, in order to screen and control the information, the gating mechanism is used, the output obtained by the multi-layer convolution layer is used as a weight, the network selectively transmits or shields some information, and the redundant features are filtered, and the formula is as follows:

[0027] weight = sigmoid(G)

[0028] F * = weight · V + (1 - weight) · X

[0029] wherein, weight is a weight, sigmoid is an activation function for processing the feature into a weight, is a fusion feature filtered according to the weight; for the three fusion features F * obtained by the three different feature compression methods, the three features are combined by using Hadamard product, specifically:

[0030] F = F meanpooling F maxpooling F GRU

[0031] wherein, F represents the fusion cross-modal feature, which not only contains the spatio-temporal information of the video, but also contains the semantic information of the text, and can more accurately reflect the corresponding relationship between the video content and the text query.

[0032] Preferably, the original features in the extracted video are input, a negative sample set is generated by using a given video-text pair to generate a negative sample pair, so that the association between the video and the text has uniqueness, and the diversity of the training samples is increased, specifically:

[0033] a given training video query pair is defined as a positive sample pair, and a video and a query from different sample pairs are mixed to form the negative sample pair, and the difference between the negative sample pair and the positive sample pair is used to assist training, the positive video query pair produces different saliency scores according to query relevance, and the irrelevant negative video query pair is forced to produce the lowest saliency score; the positive sample pair will make the highlight detection loss L hd and the negative sample pair loss L neg of the negative sample pair rises, and the negative sample pair will make it fall, and the loss function of the negative sample pair is expressed as follows:

[0034]

[0035] wherein, represents the generated negative sample set, and S(·) represents a saliency prediction head.

[0036] Preferably, the input fusion cross-modal feature and negative sample set are input, the saliency score of the time instance retrieval and the highlight detection are calculated by the saliency prediction head and the feature decoder, which are respectively used to calculate the loss of the time instance retrieval and the highlight detection, and the overall loss function of the model is generated, specifically:

[0037] In the saliency prediction head S(·), the randomly initialized learnable parameter y is input into the DETR encoder-decoder structure together with the fused cross-modal feature, so that the saliency identifier can learn the relevant context information, as shown in the formula:

[0038]

[0039] where S(F) is the calculated saliency score, which is used as the prediction result of highlight detection, W x and W F are learnable matrices, and d is a tuning factor parameter. The interval is represented by the dynamic two-dimensional anchor center point center and offset offset, which is the result of the moment retrieval;

[0040] The loss function includes moment retrieval L mr , highlight detection L hd , and contrast loss L neg , where L mr mainly includes the mean absolute error loss L1 and the improved intersection over union loss gIoU, whose formulas are as follows:

[0041] L mr = λ L1 ‖m c -m′ c ‖+λ gIoU L gIoU (m,m′)

[0042] where m and m' are the true value and the predicted value, the mean absolute error loss L1 = ||m c -m′ c || is calculated from the average absolute error of the interval center point, and the improved intersection over union loss gIoU is calculated from the intersection relationship of the real interval and the predicted interval composed of the center point and the offset, λ * all represent hyperparameters for balancing the loss;

[0043] The loss of HD includes the margin ranking loss L margin and the ranking-aware contrast loss L cons . The margin ranking loss is a loss function used to train the ranking model, which is usually used to learn how to correctly rank the given positive sample pairs and negative sample pairs. The basic idea is to ensure that the score of the positive sample is higher than that of the negative sample by a fixed margin, which is as follows:

[0044] L margin = max(0, δ + S(Mlow) - S(M high ))

[0045] wherein delta represents the difference, M low and M high respectively represent the significance of the positive sample and the negative sample, and S represents the significance score;

[0046] The ranking-aware loss is used to minimize the loss of all sample pairs, so that the representation of the positive sample is closer in the feature space, and the ranking of the positive sample is higher than that of the negative sample in the whole data set, given a maximum significance score threshold R in a batch, and then R times are iterated in the batch, and each time an iterative index r is used to construct a positive sample set and a negative sample set The dividing line of the positive and negative sample sets is the iterative index r, and the specific calculation method is as follows:

[0047]

[0048] Wherein tau is a regulation parameter, and the loss of highlight detection is represented as follows:

[0049] L hd =lambda margin L margin +lambda cons L cons

[0050] Wherein lambda * is a hyperparameter, and the final overall loss is represented as follows:

[0051] L=L mr +L hd +lambda neg L neg

[0052] Wherein lambda is a hyperparameter, and L represents the overall loss function of the model.

[0053] Correspondingly, the application also provides a video time retrieval and highlight segment detection system, comprising:

[0054] A data extraction unit is configured to input a data set, extract original features in a video using a pre-trained feature extraction network, and extract video features and text features in the video.

[0055] A feature fusion unit is configured to input the extracted original features in the video into a convolution-driven multi-modal fusion module, pass the original features through a feature activation layer and a dilated convolution layer, and output fused cross-modal features.

[0056] A sample construction unit is configured to input the extracted original features in the video, generate negative sample pairs using a given video-text pair, and finally generate a negative sample set, so that the association between the video and the text is unique and the diversity of the training samples is increased.

[0057] a loss calculation unit configured to input the fused cross-modal feature and the negative sample set, calculate the saliency score of the interval of the moment retrieval and the highlight detection through a saliency prediction head and a feature decoder, and calculate the loss of the moment retrieval and the highlight detection, respectively, to generate a model overall loss function;

[0058] a result generation unit configured to optimize the joint video moment retrieval and highlight detection model through loss training by using the model overall loss function, and output the trained model and the final moment retrieval prediction and highlight detection prediction results.

[0059] The present application has the following beneficial effects:

[0060] The present application makes full use of the complementarity between modalities, thereby improving the understanding and representation ability of the model for data. In terms of feature fusion, compared with the traditional bilinear model, the present application adopts a feature activation method of feature compression and expansion after mutual embedding, which can more fully mine the complex information in the interaction of modal features. Moreover, with the help of the idea of contrastive learning, the present application processes the positive video-query sample pairs to generate irrelevant negative sample pairs, which will produce a lower saliency score in training, thereby increasing the diversity of the training samples and assisting the training through the difference between the positive and negative samples. In addition, in the query result generation part, the present application improves the architecture design of the saliency prediction head, discards the text query generator module in the traditional architecture, and further lightens the model.

[0061] Compared with the general video moment detection and highlight detection method using a complex attention mechanism in the modal interaction part, the method adopted by the present application significantly reduces the amount of calculation and the amount of parameters, reduces the hardware cost and has a faster calculation speed; compared with the existing video moment detection and highlight detection method, the method adopted by the present application can fully utilize the modal complementarity, which is conducive to improving the accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is a video moment retrieval and highlight segment detection method flowchart of an embodiment of the present application;

[0063] Figure 2 is a video moment retrieval and highlight segment detection system structure diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0065] Figure 1 is a flow chart of a video moment retrieval and highlight segment detection method according to an embodiment of the present application, as shown in the figure, the method comprises the following steps. Figure 1

[0066] S1, input a data set, and extract original features in a video using a pre-trained feature extraction network, including video features and text features in the video;

[0067] S2, input the extracted original features in the video into a convolution-driven multi-modal fusion module, output fused cross-modal features through a feature activation layer and a dilated convolution layer;

[0068] S3, input the extracted original features in the video, generate negative sample pairs using given video-text pairs, and finally generate a negative sample set, so that the association between the video and the text has uniqueness, and the diversity of the training samples is increased;

[0069] S4, input the fused cross-modal features and the negative sample set, calculate the interval of moment retrieval and the saliency score of highlight detection through a saliency prediction head and a feature decoder, and calculate the loss of moment retrieval and highlight detection, respectively, to generate a model overall loss function;

[0070] S5, use the model overall loss function to optimize the joint video moment retrieval and highlight detection model through loss training, and output the trained model and the final moment retrieval prediction and highlight detection prediction results.

[0071] Step S1 is as follows:

[0072] S1-1: The obtained video data set comes from public video data sets QVHighlights and TVSum, and the input data set contains visual information and text information in the video.

[0073] S1-2: Use a pre-trained feature extraction network SlowFast and a CLIP composed video encoder to extract original video features Use a pre-trained CLIP text encoder to extract original text features Wherein N v and N t represent the lengths of the video and text feature sequences, respectively.​

[0074] Step S2, as follows:

[0075] S2-1: adopt the extracted original video features and text features Since the length of the text features is dynamically changing (related to the length of the query sentence), N v and N t are not equal in most cases; in the process of multi-modal feature fusion, the difference and complementarity of video and text features are one of the key problems; traditional fusion methods such as simple splicing and weighted average can combine the modes to some extent, but often there is a problem of information loss or deficiency; in order to fully utilize the complementarity between modes, the text features T are compressed and expanded in the feature activation layer, so as to fully utilize the complementarity between modes to capture the interaction between more complex multi-modal features; the present application adopts three different feature compression methods, average pooling, maximum pooling and GRU, and expands to the same dimension as the video features V after processing, and then splices and maps back to the original dimension N v of the video features V, and the formula is as follows:

[0076] X = Linear (Concat [V, T'])

[0077] Where X is the processed fusion feature, T' represents the text feature compressed and expanded by one of the feature compression methods, Concat represents the splicing operation, and Linear represents the full connection layer;

[0078] Through the mutual embedding of the above-mentioned video features and text features, the specific moment region of the visual features is fully activated under the guidance of the text features, so as to more accurately reflect the visual information and text information in the video;

[0079] S2-2: input the processed fusion feature X, input X into a multi-layer hollow convolution network layer, wherein the convolution kernel size of the hollow convolution is set to 3, the convolution stacking parameter is set to 2, and the expansion rate is set to 2, that is, an interval (or called expansion rate) is introduced between the convolution kernels to expand the receptive field, thereby increasing the perception ability of the network, and at the same time, in order to reduce overfitting and solve the problem of gradient disappearance, BatchNorm and Relu layers are added in the middle of the convolution layer, and the initial input x (0) = X, and the calculation process between the convolution layers is as follows:

[0080] z (i) = Conv (x (i-1) )

[0081] z (i)= BatchNorm(z (i) )

[0082] x (i) = Relu(z (i) )

[0083] wherein G = x (i) is a multi-modal feature, representing the output of the i-th layer, and the output feature obtained after multi-layer convolutional layer In the multi-layer convolutional operation, the interaction information of video features and text features is transmitted layer by layer, so as to realize more fine inter-modal fusion;

[0084] S2-3: input the multi-modal feature G, in order to filter and control the information, use the gating mechanism, use the output obtained by the multi-layer convolutional layer as the weight, the network selectively transmits or shields some information, filters the redundant features, and the formula is as follows:

[0085] Weight = sigmoid(G)

[0086] F * = weight · V + (1 - weight) · X

[0087] wherein weight is a weight, sigmoid is an activation function for processing features into weights, is the fusion feature filtered according to the weight; for the three fusion features F * obtained by the three different feature compression methods, the three features are combined by using Hadamard product, specifically:

[0088] F = F mean pooling ⊙ F max pooling ⊙ F GRU

[0089] wherein F represents the fusion cross-modal feature, which not only contains the space-time information of the video, but also contains the semantic information of the text, and can more accurately reflect the corresponding relationship between the video content and the text query.

[0090] Step S3, specifically as follows:

[0091] S3-1: Due to the existence of many similar segments in the video, the association of the video and the text is not unique, in order to increase the diversity of the training samples, the present application introduces a negative sample pair, specifically, a given training video query pair is defined as a positive sample pair, and the video and query from different sample pairs are mixed to form the negative sample pair, and the difference between the negative sample pair and the positive sample pair is used to assist training, the positive video query pair produces different saliency scores according to query relevance, and the irrelevant negative video query pair is forced to produce the lowest saliency score; the positive sample pair will make the highlight detection loss L hd and the negative sample pair loss L neg of the negative sample pair rises, and the negative sample pair makes it fall, and the loss function of the negative sample pair is expressed as follows:

[0092]

[0093] wherein, represents the generated negative sample set, and S(·) represents the saliency prediction head.

[0094] Step S4, specifically as follows:

[0095] S4-1: The traditional saliency prediction head is often a simple fully connected layer, however, such a general head provides the same standard for saliency prediction of each video query pair, ignoring the diversity of video and natural language query pairs. Therefore, a randomly initialized learnable parameter y is defined as a saliency identifier in this paper, which is input into the DETR encoder-decoder structure together with the fused cross-modal feature, so that the saliency identifier can learn relevant context information, as shown in the formula:

[0096]

[0097] wherein S(F) is the calculated saliency score, which is used as the prediction result of highlight detection, and W x and W F are learnable matrices, and d is a tuning factor parameter,

[0098] Since the convolution-driven multi-modal fusion mechanism is used to fuse video features and text features, the traditional step of feeding back the text query to the decoder is not needed, and the dynamic two-dimensional anchor box center point center and offset can be used to represent the interval as the result of moment retrieval;

[0099] S4-2: The loss function includes moment retrieval L mr , highlight detection L hd and contrast loss L neg , wherein L mrThe loss function mainly includes the mean absolute error loss L1 and the improved intersection over union loss gloU, whose formulas are as follows:

[0100] L mr =λ L1 ||m c -m′ c ||+λ gIoU L gIoU (m,m′)

[0101] wherein m and m' are the true value and the predicted value, the mean absolute error loss L1 = ||m c -m′ c || is calculated by the mean absolute error of the interval center point, the improved intersection over union loss gIoU is calculated by the intersection and union relationship of the real interval and the predicted interval composed of the center point and the offset, and λ * all represent the hyperparameters for balancing the loss;

[0102] The loss of the HD includes the margin ranking loss L margin and the ranking-aware contrastive loss L cons , the margin ranking loss is a loss function for training a ranking model, and is usually used to learn how to correctly rank the given positive sample pair and negative sample pair, and the basic idea is to ensure that the score of the positive sample is higher than that of the negative sample by a fixed margin, and the specific formula is as follows:

[0103] L margin = max(0,δ+S(M low )-S(M high ))

[0104] wherein δ represents the margin, M low and M high represent the saliency labels of the positive sample and the negative sample respectively, and S represents the saliency score;

[0105] The ranking-aware contrastive loss minimizes the loss of all sample pairs, so that the representation of the positive sample is closer in the feature space, and the ranking of the positive sample is higher than that of the negative sample in the overall data set, and is often used to learn the relative order relationship between the samples in the feature space, and is an extension of the contrastive loss, and is particularly suitable for ranking tasks and learning the relative relationship between the samples with order, given a maximum saliency score threshold R in a batch, and then iterating R times in the batch, and each time using the iterative index r∈{0,1,...,R-1} to construct a positive sample set and a negative sample set The dividing line of the positive and negative sample sets is the iterative index r, and the specific calculation method is as follows:

[0106]

[0107] wherein, tau is a tuning parameter, the loss of highlight detection is represented as follows:

[0108] L hd = lambda margin L margin + lambda cons L cons

[0109] wherein, lambda * is a hyperparameter, the final overall loss is represented as follows:

[0110] L = L mr + L hd + lambda neg L neg

[0111] wherein lambda is a hyperparameter, L represents the overall loss function of the model.

[0112] Correspondingly, the application also provides a video time retrieval and highlight segment detection system, as shown in the accompanying drawings, comprising: Figure 2 A data extraction unit 1 is configured to input a data set, extract original features in a video using a pre-trained feature extraction network, and extract video features and text features in the video;

[0113] A feature fusion unit 2 is configured to input the extracted original features in the video using a convolution-driven multi-modal fusion module, output fused cross-modal features through a feature activation layer and a hole convolution layer;

[0114] A sample construction unit 3 is configured to input the extracted original features in the video, generate negative sample pairs using a given video-text pair, and finally generate a negative sample set, so that the association between the video and the text has uniqueness and the diversity of the training samples is increased;

[0115] A loss calculation unit 4 is configured to input the fused cross-modal features and the negative sample set, calculate the interval of time retrieval and the saliency score of highlight detection through a saliency prediction head and a feature decoder, and calculate the loss of time retrieval and highlight detection, respectively, to generate an overall loss function of the model;

[0116] A result generation unit 5 is configured to use the overall loss function of the model to optimize the joint video time retrieval and highlight detection model through loss training, and output the trained model and the final time retrieval prediction and highlight detection prediction results.

[0117]

[0118] ​Therefore, the application takes full advantage of the complementarity between each modality, thereby improving the understanding and representation ability of the model for data. In terms of feature fusion, compared with the traditional bilinear model, the feature compression and expansion are used in the feature activation method of the application, which can more fully mine the complex information in the interaction of modal features. Moreover, with the help of the idea of contrastive learning, the application processes the positive video-query sample pairs to generate irrelevant negative sample pairs, which will produce lower saliency scores during training, thereby increasing the diversity of training samples and assisting training through the difference between positive and negative samples. In addition, in the query result generation part, the application improves the architecture design of the saliency prediction head, discards the text query generator module in the traditional architecture, and further lightens the model. Compared with the general video time detection and highlight detection method using complex attention mechanism in the modal interaction part, the method used by the application significantly reduces the amount of calculation and the amount of parameters, reduces the hardware cost and has a faster calculation speed; compared with the existing video time detection and highlight detection method, the method used by the application can fully utilize the modality complementarity, which is conducive to improving the accuracy of the model.

[0119] The above describes in detail a video time retrieval and highlight segment detection method and system provided by an embodiment of the application. The principles and implementation modes of the application are described by applying specific examples in this paper. The above description of the embodiments is only used to help understand the method of the application and its core idea; at the same time, for those skilled in the art, according to the idea of the application, the specific implementation mode and application range will be changed; in summary, the content of the specification should not be understood as a limitation of the application.

Claims

1. A method for video moment retrieval and highlight segment detection, characterized in that, The method includes: The input dataset is used to extract the original features from the video using a pre-trained feature extraction network, including video features and text features. A convolution-driven multimodal fusion module is used. The original features extracted from the video are input, and the fused cross-modal features are output through feature activation layers and dilated convolutional layers. Input the original features from the extracted video, generate negative sample pairs using the given video-text pairs, and finally generate a negative sample set, thereby making the association between video and text unique and increasing the diversity of training samples; Input the fused cross-modal features and negative sample set, and calculate the saliency scores of the time retrieval interval and the highlight detection through the saliency prediction head and feature decoder. These scores are used to calculate the loss of time retrieval and highlight detection, respectively, and generate the overall loss function of the model. Using the overall loss function of the model, the joint video time-of-view retrieval and highlight detection model is optimized through loss training, and the trained model and the final time-of-view retrieval prediction and highlight detection prediction results are output. Specifically, the convolution-driven multimodal fusion module takes the extracted original features from the video as input, passes them through a feature activation layer and a dilated convolutional layer, and outputs the fused cross-modal features, as follows: Using the extracted video features V and text features T, the text features T are compressed and expanded at the feature activation layer. Leveraging the complementarity between modalities, the interactions between multimodal features are captured. Three different feature compression methods—average pooling, max pooling, and GRU—are used to compress the text features T. After compression, the text features T are expanded to the same dimension as the video features V, then concatenated with the video features and mapped back to the original dimension of the video features. The formula is as follows: X = Linear(Concat[V,T′]) Where X represents the processed fusion feature, T′ represents the text feature after compression and expansion by one of the feature compression methods, Concat represents the concatenation operation, and Linear represents the fully connected layer; Through the mutual embedding of video features and text features, under the guidance of text features, specific time regions of visual features are fully activated, thereby more accurately reflecting the visual and text information in the video. The processed fused feature X is input into a multi-layer dilated convolutional network. Spacing is introduced between the convolutional kernels to expand the receptive field. BatchNorm and ReLU layers are added in the middle of the convolutional layers. In the multi-layer dilated convolution operation, the interaction information between video features and text features is passed layer by layer. After passing through the multi-layer dilated convolutional network, the output multimodal feature G is obtained.

2. The video moment retrieval and highlight segment detection method as described in claim 1, characterized in that, The input dataset is used to extract raw features from the video using a pre-trained feature extraction network, including video features and text features, specifically: The input dataset contains visual and textual information from videos. Video features V are extracted using a video encoder composed of a pre-trained feature extraction network SlowFast and CLIP, and textual features T are extracted using a pre-trained CLIP text encoder.

3. The video moment retrieval and highlight segment detection method as described in claim 1, characterized in that, The convolution-driven multimodal fusion module takes the extracted original features from the video as input, passes them through a feature activation layer and a dilated convolutional layer, and outputs the fused cross-modal features. It also includes: The multimodal features G are input, and in order to filter and control the information, a gating mechanism is used. The outputs of multiple dilated convolutional network layers are used as weights, and the network selectively transmits or masks some information to filter redundant features. The specific formula is as follows: weight = sigmoid(G) F * =weight·V+(1-weight)·X Where weight is the weight, sigmoid is the activation function that processes multimodal features into weights, and F * It is based on the fused features after weight filtering; for the three fused features F obtained by the three different feature compression methods * , including F meanpooling F maxpooling F GRU The three features are combined using the Hadamard product, specifically: F=F meanpooling ⊙F maxpooling ⊙F GRU Here, F represents the fused cross-modal feature, which includes not only the spatiotemporal information of the video but also the semantic information of the text, reflecting the correspondence between video content and text query.

4. The video moment retrieval and highlight segment detection method as described in claim 1, characterized in that, The original features extracted from the video are input, and negative sample pairs are generated using the given video-text pairs, ultimately generating a negative sample set. This ensures the uniqueness of the association between the video and the text, increasing the diversity of the training samples. Specifically: A given training video query pair is defined as a positive sample pair, and videos and queries from different sample pairs are mixed to form the negative sample pair. The difference between the negative sample pair and the positive sample pair is used to assist training. Positive video query pairs generate different significance scores based on query relevance, while irrelevant negative video query pairs are forced to generate the lowest significance score. Positive sample pairs will cause a loss of L in specular detection. hd And the negative sample pair loss L that suppresses negative sample pairs neg An increase in the number of positive samples will be offset by a decrease in the number of negative samples. The loss function for suppressing negative samples is expressed as follows: in, S represents the generated negative sample set, S(·) represents the significance prediction head, and r is the index.

5. The video moment retrieval and highlight segment detection method as described in claim 4, characterized in that, The input consists of the fused cross-modal features and negative sample set. Through a saliency prediction head and a feature decoder, the saliency scores for time-time retrieval and highlight detection are calculated. These scores are then used to calculate the losses for time-time retrieval and highlight detection, respectively, generating the overall model loss function. Specifically: In the saliency prediction head S(·), the randomly initialized learnable parameter y is used as the saliency label, and together with the fused cross-modal features, it is input into the DETR encoder-decoder structure, so that the saliency label can learn relevant contextual information, as shown in the formula: Where S(F) is the calculated significance score, which is used as the prediction result for highlight detection, and W x and W F All are learnable matrices, d is an adjustment factor parameter, and the interval is represented by the dynamic two-dimensional anchor box center point center and offset, which serve as the result of the time-time retrieval; The loss function includes time-limited retrieval L mr Highlight detection L hd And contrast loss L neg L mr This mainly includes the mean absolute error loss L1 and the improved crossover ratio loss L. gIoU The formula is as follows: L mr =λ L1 ‖m c -m′ c ‖+λ gIoU L gIoU (m,m′) Where m and m′ are the true value and the predicted value, respectively, and the mean absolute error loss L1 = ||m c -m′ c The improved crossover ratio loss L is calculated from the mean absolute error of the interval center point. gIoU λ is calculated from the intersection and union relationship between the true interval (composed of the center point and the offset) and the predicted interval. L1 , λ gIoU This represents the hyperparameters used to balance the loss; HD's loss includes the difference ranking loss L margin Contrast loss L with order awareness cons The differential ranking loss is a loss function used to train a ranking model. It learns how to rank given pairs of positive and negative samples, ensuring that the score of a positive sample is higher than the score of a negative sample by a fixed difference, as follows: L margin =max(0,δ+S(M low )-S(M high )) Where δ represents the difference, M low and M high These represent the significance indicators for positive and negative samples, respectively, and S represents the significance score. We utilize ranking-aware loss to minimize the loss of all sample pairs, where positive samples rank higher than negative samples in the overall dataset. Given a maximum significance score threshold R in a batch, we iterate R times within the batch, constructing a positive sample set each time using the iteration index r∈{0,1,...,R-1}. and negative sample set The boundary between positive and negative sample sets is the iteration index r, which is calculated as follows: Where τ is an adjustment parameter, the loss for specular detection is expressed as follows: L hd =λ margin L margin +λ cons L cons Where, λ margin , λ cons As a hyperparameter, the final overall loss is expressed as follows: L=L mr +L hd +λ neg L neg Where, λ neg Here, L is a hyperparameter, and L represents the overall loss function of the model.

6. A video time-lapse retrieval and highlight segment detection system, characterized in that, The system includes: The data extraction unit is used to extract the original features from the video using a pre-trained feature extraction network, including video features and text features. The feature fusion unit is used to employ a convolution-driven multimodal fusion module, which takes the original features extracted from the video as input, and outputs the fused cross-modal features through a feature activation layer and a dilated convolutional layer. The sample construction unit is used to take the original features from the extracted video as input, generate negative sample pairs using the given video-text pairs, and finally generate a negative sample set, thereby making the association between video and text unique and increasing the diversity of training samples. The loss calculation unit is used to take into input the fused cross-modal features and negative sample set, and through the saliency prediction head and feature decoder, calculates the saliency scores of the time-retrieval interval and the highlight detection, respectively, to calculate the loss of time-retrieval and highlight detection, and generate the overall loss function of the model. The result generation unit is used to optimize the joint video time-of-view retrieval and highlight detection model by using the overall loss function of the model through loss training, and outputs the trained model and the final time-of-view retrieval prediction and highlight detection prediction results. Specifically, the feature fusion unit is used to employ a convolution-driven multimodal fusion module, taking the extracted original features from the video as input, and outputting the fused cross-modal features through a feature activation layer and a dilated convolutional layer, specifically: Using the extracted video features V and text features T, the text features T are compressed and expanded at the feature activation layer. Leveraging the complementarity between modalities, the interactions between multimodal features are captured. Three different feature compression methods—average pooling, max pooling, and GRU—are used to compress the text features T. After compression, the text features T are expanded to the same dimension as the video features V, then concatenated with the video features and mapped back to the original dimension of the video features. The formula is as follows: X = Linear(Concat[V,T′]) Where X represents the processed fusion feature, T′ represents the text feature after compression and expansion by one of the feature compression methods, Concat represents the concatenation operation, and Linear represents the fully connected layer; Through the mutual embedding of video features and text features, under the guidance of text features, specific time regions of visual features are fully activated, thereby more accurately reflecting the visual and text information in the video. The processed fused feature X is input into a multi-layer dilated convolutional network. Spacing is introduced between the convolutional kernels to expand the receptive field. BatchNorm and ReLU layers are added in the middle of the convolutional layers. In the multi-layer convolutional operation, the interaction information between video features and text features is passed layer by layer. After passing through the multi-layer dilated convolutional network, the output multimodal feature G is obtained.

7. The video moment retrieval and highlight segment detection system as described in claim 6, characterized in that, The data extraction unit is used to input the dataset and use a pre-trained feature extraction network to extract the original features from the video, including video features and text features, specifically: The input dataset contains visual and textual information from videos. Video features V are extracted using a video encoder composed of a pre-trained feature extraction network SlowFast and CLIP, and textual features T are extracted using a pre-trained CLIP text encoder.

8. The video moment retrieval and highlight segment detection system as described in claim 6, characterized in that, The feature fusion unit, configured to employ a convolution-driven multimodal fusion module, takes the extracted original features from the video as input, passes them through a feature activation layer and a dilated convolutional layer, and outputs the fused cross-modal features. It also includes: The multimodal features G are input, and in order to filter and control the information, a gating mechanism is used. The outputs of multiple dilated convolutional network layers are used as weights, and the network selectively transmits or masks some information to filter redundant features. The specific formula is as follows: weight = sigmoid(G) F * =weight·V+(1-weight)·X Where weight is the weight, sigmoid is the activation function that processes multimodal features into weights, and F * It is based on the fused features after weight filtering; for the three fused features F obtained by the three different feature compression methods * , including F meanpooling F maxpooling F GRU The three features are combined using the Hadamard product, specifically: F=F meanpooling ⊙F maxpooling ⊙F GRU Here, F represents the fused cross-modal feature, which includes not only the spatiotemporal information of the video but also the semantic information of the text, reflecting the correspondence between video content and text query.

9. The video moment retrieval and highlight segment detection system as described in claim 6, characterized in that, The sample construction unit is used to take the original features from the extracted video as input, generate negative sample pairs using the given video-text pairs, and finally generate a negative sample set, thereby making the association between video and text unique and increasing the diversity of training samples. Specifically: A given training video query pair is defined as a positive sample pair, and videos and queries from different sample pairs are mixed to form the negative sample pair. The difference between the negative sample pair and the positive sample pair is used to assist training. Positive video query pairs generate different significance scores based on query relevance, while irrelevant negative video query pairs are forced to generate the lowest significance score. Positive sample pairs will cause a loss of L in specular detection. hd And the negative sample pair loss L that suppresses negative sample pairs neg An increase in the number of positive samples will be offset by a decrease in the number of negative samples. The loss function for suppressing negative samples is expressed as follows: in, S represents the generated negative sample set, S(·) represents the significance prediction head, and r is the index.

10. The video moment retrieval and highlight segment detection system as described in claim 9, characterized in that, The loss calculation unit is used to take into input the fused cross-modal features and negative sample set, and through the saliency prediction head and feature decoder, calculates the saliency scores of the time-retrieved interval and the highlight detection, respectively, to calculate the loss of time-retrieved interval and highlight detection, and generates the overall model loss function, specifically: In the saliency prediction head S(·), the randomly initialized learnable parameter y is used as the saliency label, and together with the fused cross-modal features, it is input into the DETR encoder-decoder structure, so that the saliency label can learn relevant contextual information, as shown in the formula: Where S(F) is the calculated significance score, which is used as the prediction result for highlight detection, and W x and W F All are learnable matrices, d is an adjustment factor parameter, and the interval is represented by the dynamic two-dimensional anchor box center point center and offset, which serve as the result of the time-time retrieval; The loss function includes time-limited retrieval L mr Highlight detection L hd And contrast loss L neg L mr This mainly includes the mean absolute error loss L1 and the improved crossover ratio loss L. gIoU The formula is as follows: L mr =λ L1 ‖m c -m′ c ‖+λ gIoU L gIoU (m,m′) Where m and m′ are the true value and the predicted value, respectively, and the mean absolute error loss L1 = ||m c -m′ c The improved crossover ratio loss L is calculated from the mean absolute error of the interval center point. gIoU λ is calculated from the intersection and union relationship between the true interval (composed of the center point and the offset) and the predicted interval. L1 , λ gIoU This represents the hyperparameters used to balance the loss; HD's loss includes the difference ranking loss L margin Contrast loss L with order awareness cons The differential ranking loss is a loss function used to train a ranking model. It learns how to rank given pairs of positive and negative samples, ensuring that the score of a positive sample is higher than the score of a negative sample by a fixed difference, as follows: L margin =max(0,δ+S(M low )-S(M high )) Where δ represents the difference, M low and M high These represent the significance indicators for positive and negative samples, respectively, and S represents the significance score. We utilize ranking-aware loss to minimize the loss of all sample pairs, where positive samples rank higher than negative samples in the overall dataset. Given a maximum significance score threshold R in a batch, we iterate R times within the batch, constructing a positive sample set each time using the iteration index r∈{0,1,...,R-1}. and negative sample set The boundary between positive and negative sample sets is the iteration index r, which is calculated as follows: Where τ is an adjustment parameter, the loss for specular detection is expressed as follows: L hd =λ margin L margin +λ cons L cons Where, λ margin , λ cons As a hyperparameter, the final overall loss is expressed as follows: L=L mr +L hd +λ neg L neg Where, λ neg Here, L is a hyperparameter, and L represents the overall loss function of the model.

Images