Video time retrieval and highlight segment detection method and system

Abstract

The invention discloses a video time retrieval and highlight segment detection method, which comprises the following steps of: extracting original features in a video by using a pre-trained feature extraction network, and performing cross-modal feature fusion on the original features through a feature activation layer and a cavity convolution layer by using a convolution-driven multi-modal fusion module; generating a negative sample pair by using a given video-text pair, and assisting training by means of the difference between positive and negative samples; calculating a time retrieval interval and a saliency score of highlight detection through a saliency prediction head and a feature decoder so as to calculate loss, and generating a model overall loss function; and finally, a prediction result is generated through a loss training optimization model. According to the method, the complementarity among all modes is fully utilized, so that the data understanding and characterization capability of the model is improved. Compared with a traditional model, the method has the advantages that the calculation amount and the parameter amount are remarkably reduced, the mode complementarity can be fully utilized, and the accuracy of the model is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and video retrieval, and in particular to a method and system for video moment retrieval and highlight segment detection. Background Art

[0002] With the explosive growth of video data, how to efficiently retrieve relevant moments and detect highlight moments in videos from massive videos has become an urgent problem to be solved. Video moment retrieval refers to retrieving time segments containing specific events or actions from a given long video. Highlight detection is to identify the wonderful moments that the audience is interested in from the video, which is usually used in scenarios such as sports events and game live broadcasts. Traditional video retrieval and highlight detection methods usually rely on manual annotation and feature extraction, which have problems such as high computational complexity, poor real-time performance, and high annotation costs, making it difficult to meet actual application needs.

[0003] In recent years, with the rise of deep learning technology, new solutions have been provided for video moment retrieval and highlight detection technology. Existing deep learning methods usually use complex attention mechanisms to extract and interact with video and text features, which introduces a huge amount of parameters and high hardware costs. In addition, text features play a decisive role in moment retrieval, while current research methods use a hybrid weighted approach or a late fusion approach to interact with text features, which does not give full play to the important role of text features in the cross-modal feature fusion process.

[0004] Therefore, the present invention proposes a convolution-driven joint video moment retrieval and highlight detection method, which adopts a convolution-based cross-modal feature fusion mechanism to replace the complex attention layer to fuse the encoded video and text features, making full use of the complementarity between the modalities, improving the model's ability to understand and represent data, and achieving lower hardware cost and better performance.

[0005] One of the current existing technologies is Lei J et al.'s "Qvhighlights: Detecting moments and highlights in videos via natural language queries", a video moment retrieval model Moment-DETR based on the Transformer architecture. The model consists of a feature encoder, a temporal encoder, a query generator, and a decoder module. It uses the attention mechanism in deep learning for efficient moment retrieval, where the feature encoder is responsible for extracting the multimodal feature representation of the video, the temporal encoder performs temporal modeling on the video features, the query generator generates the query vector required for retrieval, and the decoder module interacts the query vector with the video features to finally generate accurate moment retrieval results. The disadvantages of this model are that the video features and text features are only simply spliced ​​during the fusion process, resulting in confusion in feature fusion, which affects the performance of the model. In addition, the model has a large amount of computation and high hardware cost.

[0006] The second existing technology is a query-driven detection model QD-DETR based on the attention mechanism in "Query-dependent video representation for moment retrieval and highlight detection" by Moon WJ et al. The model consists of a feature encoder, a query generator, a decoder module and a loss calculation module, and uses a query vector to guide target detection. The feature encoder is responsible for extracting the visual feature representation of the image or video, the query generator generates a query vector related to the target detection task, the decoder module generates target detection results by interacting with the visual features, and the loss calculation module is used to optimize the detection accuracy of the model to obtain more accurate target detection results. The disadvantages of this model are: the model focuses on the complex query-guided attention mechanism, pays attention to global features, and ignores the important impact of local features on performance. In addition, the model has a large amount of computation and parameters, and the hardware cost is high. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of existing methods and propose a method and system for video moment retrieval and highlight segment detection. The main problems solved by the present invention are: 1) How to overcome the problem that existing video moment retrieval and highlight detection methods are too focused on complex attention mechanisms and the dependencies between global features, thereby introducing a huge amount of parameters and calculations, and having low efficiency and being unsuitable for large-scale popularization in practical applications; 2) How to use a multimodal feature fusion mechanism driven by convolution to replace the attention mechanism, while ensuring the continuity of local features, giving full play to the role of multimodal features, thereby obtaining a highly accurate and more lightweight model.

[0008] In order to solve the above problems, the present invention proposes a video moment retrieval and highlight segment detection method, the method comprising:

[0009] Input the data set and use the pre-trained feature extraction network to extract the original features in the video, including video features and text features in the video;

[0010] A convolution-driven multimodal fusion module is used to input the original features in the extracted video, and output the fused cross-modal features through a feature activation layer and a hole convolution layer;

[0011] Inputting the original features in the extracted video, generating negative sample pairs using given video-text pairs, and finally generating a negative sample set, so that the association between the video and the text is unique and the diversity of the training samples is increased;

[0012] Input the fused cross-modal features and negative sample set, calculate the saliency scores of the interval of moment retrieval and highlight detection through the saliency prediction head and feature decoder, and use them to calculate the losses of moment retrieval and highlight detection respectively, and generate the overall loss function of the model;

[0013] The overall loss function of the model is used to optimize the joint video moment retrieval and highlight detection model through loss training, and the trained model and the final moment retrieval prediction and highlight detection prediction results are output.

[0014] Preferably, the input data set uses a pre-trained feature extraction network to extract original features in the video, including video features and text features in the video, specifically:

[0015] The input dataset contains visual information and text information in the video. The video encoder composed of pre-trained feature extraction network SlowFast and CLIP is used to extract the original video features. Use the pre-trained CLIP text encoder to extract the original text features Where N v and N t Represent the length of video and text feature sequences respectively.

[0016] Preferably, the multimodal fusion module driven by convolution inputs the original features in the extracted video, and outputs the fused cross-modal features through the feature activation layer and the hole convolution layer, specifically:

[0017] Using the extracted original video features and text features In the feature activation layer, the text feature T is compressed and expanded to make full use of the complementarity between the modes to capture the interaction between more complex multimodal features; three different feature compression methods, average pooling, maximum pooling and GRU, are used respectively. After processing, it is expanded to the same dimension as the video feature V, and then spliced ​​with the video feature and mapped back to the original dimension N of the video feature. v , the formula is as follows:

[0018] X = Linear(Concat[V, T′])

[0019] Wherein, X is the processed fusion feature, T′ represents the text feature after being compressed and expanded by one of the feature compression methods, Concat represents the concatenation operation, and Linear represents the fully connected layer;

[0020] Through the mutual embedding of the above-mentioned video features and text features, under the guidance of the text features, the specific time region of the visual features is fully activated, so that the visual information and text information in the video can be more accurately reflected;

[0021] Input the processed fusion feature X, and input X into the multi-layer dilated convolutional network layer, that is, introduce intervals (or called dilation rates) between convolution kernels to expand the receptive field, thereby increasing the perception ability of the network. At the same time, in order to reduce overfitting and solve the gradient disappearance problem, add BatchNorm and Relu layers in the middle of the convolution layer to define the initial input x (0) =X, the calculation process between convolutional layers is as follows:

[0022] z (i) =Conv(x (i-1) )

[0023] z (i) =BatchNorm(z (i) )

[0024] x (i) =Relu(z (i) )

[0025] Where G = x (i) It is a multimodal feature, representing the output of the i-th layer, and the output feature is obtained after multiple convolutional layers. In multi-layer convolution operations, the interactive information between video features and text features is transmitted layer by layer, thus achieving more sophisticated inter-modal fusion;

[0026] The multimodal feature G is input. In order to filter and control the information, a gating mechanism is used to selectively transmit or shield certain information and filter redundant features by using the outputs of multiple convolutional layers as weights. The formula is as follows:

[0027] weight = sigmoid(G)

[0028] F * =weight·V+(1-weight)·X

[0029] Among them, weight is the weight, sigmoid is the activation function that processes the feature into the weight, is the fusion feature after filtering according to the weight; for the three fusion features F obtained by the three different feature compression methods * , the three features are combined using the Hadamard product, specifically:

[0030] F=F meanpooling ⊙F maxpooling ⊙F GRU

[0031] Among them, F represents the fused cross-modal feature, which not only includes the spatiotemporal information of the video, but also includes the semantic information of the text, and can more accurately reflect the correspondence between the video content and the text query.

[0032] Preferably, the original features in the extracted video are input, and a negative sample pair is generated using a given video-text pair, and finally a negative sample set is generated, so that the association between the video and the text is unique and the diversity of the training samples is increased, specifically:

[0033] A given training video query pair is defined as a positive sample pair, and videos and queries from different sample pairs are mixed to form the negative sample pair, and the difference between the negative sample pair and the positive sample pair is used to assist training. The positive video query pair generates different saliency scores according to the query relevance, while the irrelevant negative video query pair is forced to generate the lowest saliency score; the positive sample pair will cause the highlight detection loss L hd and the negative pair loss L that suppresses negative pairs neg The negative sample pairs will make it rise, and the loss function of suppressing the negative sample pairs is expressed as follows:

[0034]

[0035] in, represents the generated negative sample set, and S(·) represents the saliency prediction head.

[0036] Preferably, the fused cross-modal features and negative sample set are input, and the saliency scores of the interval of moment retrieval and highlight detection are calculated through the saliency prediction head and feature decoder, which are used to calculate the losses of moment retrieval and highlight detection respectively, and generate the overall loss function of the model, which is specifically:

[0037] In the saliency prediction head S(·), the randomly initialized learnable parameter y is used as a saliency marker and is input into the DETR encoder-decoder structure together with the fused cross-modal features, so that the saliency marker can learn relevant contextual information, as shown in the formula:

[0038]

[0039] Among them, S(F) is the calculated significance score, which is used as the prediction result of highlight detection, and W x and W F They are all learnable matrices, d is the adjustment factor parameter, and the interval is represented by the dynamic two-dimensional anchor box center point center and offset offset as the result of moment retrieval;

[0040] The loss function includes the time retrieval L mr , High light detection L hd and contrast loss L neg , where L mr It mainly includes the mean absolute error loss L1 and the improved intersection-over-union loss gIoU, and its formula is as follows:

[0041] L mr =λ L1 ‖m c -m′ c ‖+λ gIoU L gIoU (m,m′)

[0042] Among them, m and m′ are the true value and the predicted value, and the mean absolute error loss L1 = ||m c -m′ c || is calculated from the mean absolute error of the center point of the interval, and the improved intersection-over-union loss gIoU is calculated from the intersection of the true interval and the predicted interval composed of the center point and the offset, λ * All represent hyperparameters used to balance the loss;

[0043] The loss of HD includes the difference sorting loss L margin and the ranking-aware contrast loss L cons , the difference ranking loss is a loss function used to train the ranking model. It is usually used to learn how to correctly sort them given positive and negative sample pairs. The basic idea is to ensure that the score of the positive sample is higher than the score of the negative sample by a fixed difference, as follows:

[0044] L margin =max(0,δ+S(Mlow)-S(M high ))

[0045] Where δ represents the difference, M low and M high Represent the significance marks of positive samples and negative samples respectively, and S represents the significance score;

[0046] The ranking-aware loss is used to minimize the loss of all sample pairs, so that the representation of positive samples is closer in the feature space and the ranking of positive samples is higher than that of negative samples in the entire dataset. Given a maximum significance score threshold R in a batch, it is iterated R times in the batch, and each time a positive sample set is constructed using the iteration index r∈{0, 1, ..., R-1} and negative sample set The dividing line between the positive and negative sample sets is the iterative index r, which is calculated as follows:

[0047]

[0048] Among them, τ is a tuning parameter, and the loss of highlight detection is expressed as follows:

[0049] L hd =λ margin L margin +λ cons L cons

[0050] Among them, λ * is a hyperparameter, and the final overall loss is expressed as follows:

[0051] L=L mr +L hd +λ neg L neg

[0052] Where λ is a hyperparameter, and L represents the overall loss function of the model.

[0053] Accordingly, the present invention also provides a video moment retrieval and highlight segment detection system, comprising:

[0054] A data extraction unit, used for inputting a data set, and extracting original features in the video using a pre-trained feature extraction network, including video features and text features in the video;

[0055] A feature fusion unit, which is used to adopt a convolution-driven multimodal fusion module, input the original features in the extracted video, and output fused cross-modal features through a feature activation layer and a hole convolution layer;

[0056] A sample construction unit, used to input the original features in the extracted video, generate negative sample pairs using given video-text pairs, and finally generate a negative sample set, so that the association between the video and the text is unique and the diversity of the training samples is increased;

[0057] A loss calculation unit, used to input the fused cross-modal features and negative sample set, calculate the saliency scores of the interval of moment retrieval and highlight detection through the saliency prediction head and feature decoder, and respectively calculate the losses of moment retrieval and highlight detection to generate the overall loss function of the model;

[0058] The result generation unit is used to optimize the joint video moment retrieval and highlight detection model through loss training by using the overall loss function of the model, and output the trained model and the final moment retrieval prediction and highlight detection prediction results.

[0059] The implementation of the present invention has the following beneficial effects:

[0060] The present invention makes full use of the complementarity between the various modalities, thereby improving the model's ability to understand and characterize data. In terms of feature fusion, compared with the traditional bilinear model, the present invention adopts a feature activation method that embeds features after compression and expansion, which can more fully mine the complex information in the interaction of modal features. In addition, with the help of the idea of ​​contrastive learning, the present invention processes positive video-query sample pairs to generate irrelevant negative sample pairs, which will produce lower significance scores during training, thereby increasing the diversity of training samples and assisting training through the differences between positive and negative samples. In addition, in the part of generating query results, the present invention improves the architectural design of the significance prediction head, abandons the text query generator module in the traditional architecture, and further lightweights the model.

[0061] Compared with general video moment detection and highlight detection methods that use complex attention mechanisms in the modal interaction part, the method adopted by the present invention significantly reduces the amount of calculation and parameters, reduces hardware costs and has a faster calculation speed; compared with existing video moment detection and highlight detection methods, the method adopted by the present invention can make full use of modal complementarity, which is conducive to improving the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of a method for video moment retrieval and highlight segment detection according to an embodiment of the present invention;

[0063] Figure 2 It is a structural diagram of a video moment retrieval and highlight segment detection system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] Figure 1 FIG. 1 is a flow chart of a method for video moment retrieval and highlight segment detection according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0066] S1, input data set, uses the pre-trained feature extraction network to extract the original features in the video, including video features and text features in the video;

[0067] S2, using a convolution-driven multimodal fusion module, inputting the original features in the extracted video, and outputting fused cross-modal features through a feature activation layer and a hole convolution layer;

[0068] S3, inputting the original features in the extracted video, generating negative sample pairs using the given video-text pairs, and finally generating a negative sample set, so that the association between the video and the text is unique and the diversity of the training samples is increased;

[0069] S4, inputting the fused cross-modal features and negative sample set, calculating the saliency scores of the interval of moment retrieval and highlight detection through the saliency prediction head and feature decoder, which are used to calculate the losses of moment retrieval and highlight detection respectively, and generating the overall loss function of the model;

[0070] S5, using the overall loss function of the model, optimizes the joint video moment retrieval and highlight detection model through loss training, and outputs the trained model and the final moment retrieval prediction and highlight detection prediction results.

[0071] Step S1 is as follows:

[0072] S1-1: The acquired video dataset comes from the public video datasets QVHighlights and TVSum. The input dataset contains the visual information and text information in the video.

[0073] S1-2: Use the pre-trained feature extraction network SlowFast and CLIP to form a video encoder to extract the original video features Use the pre-trained CLIP text encoder to extract the original text features Where N v and N t Represent the length of video and text feature sequences respectively.

[0074] Step S2 is as follows:

[0075] S2-1: Using the original video features extracted and text features Since the length of text features changes dynamically (related to the length of the query sentence), in most cases N v and N t are not equal; in the process of multimodal feature fusion, the difference and complementarity of video and text features are one of the key issues; although traditional fusion methods such as simple splicing and weighted averaging can achieve the combination of modalities to a certain extent, there are often problems of information loss or insufficiency; in order to make full use of the complementarity between the modalities, the text feature T is compressed and expanded in the feature activation layer, so as to make full use of the complementarity between the modalities to capture the interaction between more complex multimodal features; the present invention adopts three different feature compression methods, average pooling, maximum pooling and GRU, which are expanded to the same dimension as the video feature V after processing, and then spliced ​​with the video feature and mapped back to the original dimension N of the video feature v , the formula is as follows:

[0076] X = Linear(Concat[V,T′])

[0077] Wherein, X is the processed fusion feature, T′ represents the text feature after being compressed and expanded by one of the feature compression methods, Concat represents the concatenation operation, and Linear represents the fully connected layer;

[0078] Through the mutual embedding of the above-mentioned video features and text features, under the guidance of the text features, the specific time region of the visual features is fully activated, so that the visual information and text information in the video can be more accurately reflected;

[0079] S2-2: Input the processed fusion feature X, and input X into the multi-layer dilated convolution network layer, where the convolution kernel size of the dilated convolution is set to 3, the convolution stacking parameter is set to 2, and the dilation rate is set to 2, that is, an interval (or dilation rate) is introduced between the convolution kernels to expand the receptive field, thereby increasing the perception ability of the network. At the same time, in order to reduce overfitting and solve the gradient disappearance problem, BatchNorm and Relu layers are added in the middle of the convolution layer to define the initial input x (0) =X, the calculation process between convolutional layers is as follows:

[0080] z (i) =Conv(x (i-1) )

[0081] z (i)=BatchNorm(z (i) )

[0082] x (i) =Relu(z (i) )

[0083] Where G = x (i) It is a multimodal feature, representing the output of the i-th layer, and the output feature is obtained after multiple convolutional layers. In multi-layer convolution operations, the interactive information between video features and text features is transmitted layer by layer, thus achieving more sophisticated inter-modal fusion;

[0084] S2-3: Input the multimodal feature G. In order to filter and control the information, a gating mechanism is used to selectively transmit or shield certain information and filter redundant features by using the outputs of multiple convolutional layers as weights. The formula is as follows:

[0085] Weight = sigmoid(G)

[0086] F * =weight·V+(1-weight)·X

[0087] Among them, weight is the weight, sigmoid is the activation function that processes the feature into the weight, is the fusion feature after filtering according to the weight; for the three fusion features F obtained by the three different feature compression methods * , the three features are combined using the Hadamard product, specifically:

[0088] F=F mean pooling ⊙F max pooling ⊙F GRU

[0089] Among them, F represents the fused cross-modal feature, which not only includes the spatiotemporal information of the video, but also includes the semantic information of the text, and can more accurately reflect the correspondence between the video content and the text query.

[0090] Step S3 is as follows:

[0091] S3-1: Since there are many similar clips in the video, the association between the video and the text is not unique. In order to increase the diversity of training samples, the present invention introduces negative sample pairs. Specifically, a given training video query pair is defined as a positive sample pair, and videos and queries from different sample pairs are mixed to form the negative sample pair. The difference between the negative sample pair and the positive sample pair is used to assist in training. The positive video query pair generates different significance scores according to the query relevance, while the irrelevant negative video query pair is forced to generate the lowest significance score; the positive sample pair will cause the highlight detection loss L hd and the negative pair loss L that suppresses negative pairs neg The negative sample pairs will make it rise, and the loss function of suppressing the negative sample pairs is expressed as follows:

[0092]

[0093] in, represents the generated negative sample set, and S(·) represents the saliency prediction head.

[0094] Step S4 is as follows:

[0095] S4-1: Traditional saliency prediction heads are often simple fully connected layers. However, this universal head provides the same standard for saliency prediction for each video query pair, ignoring the diversity of video and natural language query pairs. Therefore, this paper defines a randomly initialized learnable parameter y as a saliency marker, which is input into the DETR encoder-decoder structure together with the fused cross-modal features, so that the saliency marker can learn relevant contextual information, as shown in the formula:

[0096]

[0097] Among them, S(F) is the calculated significance score, which is used as the prediction result of highlight detection, and W x and W F are all learnable matrices, d is the adjustment factor parameter,

[0098] Since the convolution-driven multimodal fusion mechanism is used to fuse video features and text features, there is no need to feed the text query back to the decoder in the traditional way. The interval can be represented by the dynamic two-dimensional anchor box center point and offset as the result of the moment retrieval.

[0099] S4-2: The loss function includes the time retrieval L mr , High light detection L hd and contrast loss L neg , where L mrIt mainly includes the mean absolute error loss L1 and the improved intersection-over-union loss gloU, and its formula is as follows:

[0100] L mr =λ L1 ||m c -m′ c ||+λ gIoU L gIoU (m, m′)

[0101] Among them, m and m′ are the true value and the predicted value, and the mean absolute error loss L1 = ||m c -m′ c || is calculated from the mean absolute error of the center point of the interval, and the improved intersection-over-union loss gIoU is calculated from the intersection of the true interval and the predicted interval composed of the center point and the offset, λ * All represent hyperparameters used to balance the loss;

[0102] The loss of HD includes the difference sorting loss L margin and the ranking-aware contrast loss L cons , the difference ranking loss is a loss function used to train the ranking model. It is usually used to learn how to correctly sort them given positive and negative sample pairs. The basic idea is to ensure that the score of the positive sample is higher than the score of the negative sample by a fixed difference, as follows:

[0103] L margin =max(0,δ+S(M low )-S(M high ))

[0104] Where δ represents the difference, M low and M high Represent the significance marks of positive samples and negative samples respectively, and S represents the significance score;

[0105] The ranking-aware loss is used to minimize the loss of all sample pairs, so that the representation of positive samples is closer in the feature space, and the ranking of positive samples in the overall data set is higher than that of negative samples. It is often used to learn the relative order relationship between samples in the feature space. It is an extension of the contrast loss, especially suitable for sorting tasks and learning the relative relationship between sequential samples. Given a maximum significance score threshold R in a batch, it is iterated R times in the batch, and each time the iteration index r∈{0, 1, ..., R-1} is used to construct a positive sample set and negative sample set The dividing line between the positive and negative sample sets is the iterative index r, which is calculated as follows:

[0106]

[0107] Among them, τ is a tuning parameter, and the loss of highlight detection is expressed as follows:

[0108] L hd =λ margin L margin +λ cons L cons

[0109] Among them, λ * is a hyperparameter, and the final overall loss is expressed as follows:

[0110] L=L mr +L hd +λ neg L neg

[0111] Where λ is a hyperparameter, and L represents the overall loss function of the model.

[0112] Accordingly, the present invention also provides a video moment retrieval and highlight segment detection system, such as Figure 2 As shown, including:

[0113] A data extraction unit 1 is used to input a data set and use a pre-trained feature extraction network to extract original features in the video, including video features and text features in the video;

[0114] A feature fusion unit 2 is used to adopt a convolution-driven multimodal fusion module, input the original features in the extracted video, and output the fused cross-modal features through a feature activation layer and a hole convolution layer;

[0115] The sample construction unit 3 is used to input the original features in the extracted video, generate negative sample pairs using the given video-text pairs, and finally generate a negative sample set, so that the association between the video and the text is unique and the diversity of the training samples is increased;

[0116] The loss calculation unit 4 is used to input the fused cross-modal features and negative sample set, calculate the saliency scores of the interval of moment retrieval and highlight detection through the saliency prediction head and feature decoder, and respectively calculate the losses of moment retrieval and highlight detection to generate the overall loss function of the model;

[0117] The result generating unit 5 is used to optimize the joint video moment retrieval and highlight detection model through loss training by using the overall loss function of the model, and output the trained model and the final moment retrieval prediction and highlight detection prediction results.

[0118] Therefore, the present invention makes full use of the complementarity between the modalities, thereby improving the model's ability to understand and characterize data. In terms of feature fusion, compared with the traditional bilinear model, the present invention adopts a feature activation method that embeds each other after feature compression and expansion, which can more fully mine the complex information in the interaction of modal features. In addition, with the help of the idea of ​​contrastive learning, the present invention processes the positive video-query sample pairs to generate irrelevant negative sample pairs, which will produce lower significance scores during training, thereby increasing the diversity of training samples and assisting training through the difference between positive and negative samples. In addition, in the query result generation part, the present invention improves the architecture design of the significance prediction head, abandons the text query generator module in the traditional architecture, and further lightweights the model. Compared with the general video moment detection and highlight detection methods that use complex attention mechanisms in the modal interaction part, the method adopted by the present invention significantly reduces the amount of calculation and parameters, reduces the hardware cost and has a faster calculation speed; compared with the existing video moment detection and highlight detection methods, the method adopted by the present invention can make full use of modal complementarity, which is conducive to improving the accuracy of the model.

[0119] The above is a detailed introduction to a video moment retrieval and highlight segment detection method and system provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for video moment retrieval and highlight segment detection, characterized in that: The method comprises: Input the data set and use the pre-trained feature extraction network to extract the original features in the video, including video features and text features in the video; A convolution-driven multimodal fusion module is used to input the original features in the extracted video, and output the fused cross-modal features through a feature activation layer and a hole convolution layer; Inputting the original features in the extracted video, generating negative sample pairs using given video-text pairs, and finally generating a negative sample set, so that the association between the video and the text is unique and the diversity of the training samples is increased; Input the fused cross-modal features and negative sample set, calculate the saliency scores of the interval of moment retrieval and highlight detection through the saliency prediction head and feature decoder, and use them to calculate the losses of moment retrieval and highlight detection respectively, and generate the overall loss function of the model; The overall loss function of the model is used to optimize the joint video moment retrieval and highlight detection model through loss training, and the trained model and the final moment retrieval prediction and highlight detection prediction results are output.

2. A method for video moment retrieval and highlight segment detection as claimed in claim 1, characterized in that: The input data set uses a pre-trained feature extraction network to extract the original features in the video, including video features and text features in the video, specifically: The input dataset contains visual information and text information in the video. The video encoder composed of pre-trained feature extraction network SlowFast and CLIP is used to extract the original video features. Use the pre-trained CLIP text encoder to extract the original text features Where N v and N t Represent the length of video and text feature sequences respectively.

3. A method for video moment retrieval and highlight segment detection as claimed in claim 2, characterized in that: The multimodal fusion module driven by convolution inputs the original features in the extracted video, and outputs the fused cross-modal features through the feature activation layer and the hole convolution layer, specifically: Using the extracted original video features and text features In the feature activation layer, the text feature T is compressed and expanded to make full use of the complementarity between the modes to capture the interaction between more complex multimodal features; three different feature compression methods, average pooling, maximum pooling and GRU, are used respectively. After processing, it is expanded to the same dimension as the video feature V, and then spliced ​​with the video feature and mapped back to the original dimension N of the video feature. v , the formula is as follows: X = Linear(Concat[V,T′]) Among them, X is the processed fusion feature, T ′ represents the text feature after being compressed and expanded by one of the feature compression methods, Concat represents the concatenation operation, and Linear represents the fully connected layer; Through the mutual embedding of the above-mentioned video features and text features, under the guidance of the text features, the specific time region of the visual features is fully activated, so that the visual information and text information in the video can be more accurately reflected; Input the processed fusion feature X, and input X into the multi-layer hole convolution network layer, that is, introduce intervals between the convolution kernels to expand the receptive field, thereby increasing the perception ability of the network. At the same time, in order to reduce overfitting and solve the gradient disappearance problem, add BatchNorm and Relu layers in the middle of the convolution layer to define the initial input x (0) =X, the calculation process between convolutional layers is as follows: z (i) =Conv(x (i-1) ) With (i) =BatchNorm(from (i) ) x (i) =Reli(z (i) ) Where G = x (i) It is a multimodal feature, representing the output of the i-th layer, and the output feature is obtained after multiple convolutional layers. In multi-layer convolution operations, the interactive information between video features and text features is transmitted layer by layer, thus achieving more sophisticated inter-modal fusion; The multimodal feature G is input. In order to filter and control the information, a gating mechanism is used to selectively transmit or shield certain information and filter redundant features by using the outputs of multiple convolutional layers as weights. The formula is as follows: weight = sigmoid(G) F * =weight·V+(1-weight)·X Among them, weight is the weight, sigmoid is the activation function that processes the feature into the weight, is the fusion feature after filtering according to the weight; for the three fusion features F obtained by the three different feature compression methods * , the three features are combined using the Hadamard product, specifically: F=F meanpooling ⊙F maxpooling ⊙F GRU Among them, F represents the fused cross-modal feature, which not only includes the spatiotemporal information of the video, but also includes the semantic information of the text, and can more accurately reflect the correspondence between the video content and the text query.

4. A method for video moment retrieval and highlight segment detection as claimed in claim 1, characterized in that: The input of the original features in the extracted video, using the given video-text pair to generate a negative sample pair, and finally generating a negative sample set, so that the association between the video and the text is unique and the diversity of the training samples is increased, specifically: A given training video query pair is defined as a positive sample pair, and videos and queries from different sample pairs are mixed to form the negative sample pair, and the difference between the negative sample pair and the positive sample pair is used to assist the training, the positive video query pair generates different significance scores according to the query relevance, and the irrelevant negative video query pair is forced to generate the lowest significance score; Positive sample pairs will cause highlight detection loss L hd and the negative pair loss L that suppresses negative pairs neg The negative sample pairs will make it rise, and the loss function of suppressing the negative sample pairs is expressed as follows: in, represents the generated negative sample set, and S(·) represents the saliency prediction head.

5. A method for video moment retrieval and highlight segment detection as claimed in claim 4, characterized in that: The input of the fused cross-modal features and negative sample set is used to calculate the saliency scores of the interval of moment retrieval and highlight detection through the saliency prediction head and feature decoder, which are used to calculate the losses of moment retrieval and highlight detection respectively, and generate the overall loss function of the model, which is specifically: In the saliency prediction head S(·), the randomly initialized learnable parameter y is used as a saliency marker and is input into the DETR encoder-decoder structure together with the fused cross-modal features, so that the saliency marker can learn relevant contextual information, as shown in the formula: Among them, S(F) is the calculated significance score, which is used as the prediction result of highlight detection, and W x and W F They are all learnable matrices, d is the adjustment factor parameter, and the interval is represented by the dynamic two-dimensional anchor box center point center and offset offset as the result of moment retrieval; The loss function includes the time retrieval L mr , High light detection L hd and contrast loss L neg , where L mr It mainly includes the mean absolute error loss L1 and the improved intersection-over-union loss gIoU, and its formula is as follows: L mr =λ L1 ‖m c -m′ c ‖+λ gIoU L gIoU (m,m′) Among them, m and m ′ is the true value and the predicted value, the mean absolute error loss L1 = ‖m c -m ′ c ‖ is calculated from the mean absolute error of the center point of the interval, and the improved intersection-over-union loss gIoU is calculated from the intersection of the true interval and the predicted interval composed of the center point and the offset, λ * All represent hyperparameters used to balance the loss; The loss of HD includes the difference sorting loss L margin and the ranking-aware contrast loss L cons , the difference ranking loss is a loss function used to train the ranking model. It is usually used to learn how to correctly sort them given positive and negative sample pairs. The basic idea is to ensure that the score of the positive sample is higher than the score of the negative sample by a fixed difference, as follows: L margin =max(0,δ+S(M low )-S(M high )) Where δ represents the difference, M low and M high Represent the significance marks of positive samples and negative samples respectively, and S represents the significance score; The ranking-aware loss is used to minimize the loss of all sample pairs, so that the representation of positive samples is closer in the feature space, and the ranking of positive samples in the entire dataset is higher than that of negative samples. Given a maximum significance score threshold R in a batch, it is iterated R times in the batch, and each time a positive sample set is constructed using the iteration index r∈{0,1,...,R-1} and negative sample set The dividing line between the positive and negative sample sets is the iterative index r, which is calculated as follows: Among them, τ is a tuning parameter, and the loss of highlight detection is expressed as follows: L hd =λ margin L margin +λ cons L cons Among them, λ * is a hyperparameter, and the final overall loss is expressed as follows: L=L mr +L hd +λ neg L neg Where λ is a hyperparameter, and L represents the overall loss function of the model.

6. A video moment retrieval and highlight segment detection system, characterized in that: The system comprises: A data extraction unit, used for inputting a data set, and extracting original features in the video using a pre-trained feature extraction network, including video features and text features in the video; A feature fusion unit, which is used to adopt a convolution-driven multimodal fusion module, input the original features in the extracted video, and output fused cross-modal features through a feature activation layer and a hole convolution layer; A sample construction unit, used to input the original features in the extracted video, generate negative sample pairs using given video-text pairs, and finally generate a negative sample set, so that the association between the video and the text is unique and the diversity of the training samples is increased; A loss calculation unit, used to input the fused cross-modal features and negative sample set, calculate the saliency scores of the interval of moment retrieval and highlight detection through the saliency prediction head and feature decoder, and respectively calculate the losses of moment retrieval and highlight detection to generate the overall loss function of the model; The result generation unit is used to optimize the joint video moment retrieval and highlight detection model through loss training by using the overall loss function of the model, and output the trained model and the final moment retrieval prediction and highlight detection prediction results.

7. A video moment retrieval and highlight segment detection system as claimed in claim 6, characterized in that: The data extraction unit is used to input a data set and use a pre-trained feature extraction network to extract original features in the video, including video features and text features in the video, specifically: The input dataset contains visual information and text information in the video. The video encoder composed of pre-trained feature extraction network SlowFast and CLIP is used to extract the original video features. Use the pre-trained CLIP text encoder to extract the original text features Where N v and N t Represent the length of video and text feature sequences respectively.

8. A video moment retrieval and highlight segment detection system as claimed in claim 7, characterized in that: The feature fusion unit is used to adopt a convolution-driven multimodal fusion module, input the original features in the extracted video, and output the fused cross-modal features through a feature activation layer and a hole convolution layer, specifically: Using the extracted original video features and text features In the feature activation layer, the text feature T is compressed and expanded to make full use of the complementarity between the modes to capture the interaction between more complex multimodal features; three different feature compression methods, average pooling, maximum pooling and GRU, are used respectively. After processing, it is expanded to the same dimension as the video feature V, and then spliced ​​with the video feature and mapped back to the original dimension N of the video feature. v , the formula is as follows: X = Linear(Concat[V,T′]) Among them, X is the processed fusion feature, T ′ represents the text feature after being compressed and expanded by one of the feature compression methods, Concat represents the concatenation operation, and Linear represents the fully connected layer; Through the mutual embedding of the above-mentioned video features and text features, under the guidance of the text features, the specific time region of the visual features is fully activated, so that the visual information and text information in the video can be more accurately reflected; Input the processed fusion feature X, and input X into the multi-layer hole convolution network layer, that is, introduce intervals between the convolution kernels to expand the receptive field, thereby increasing the perception ability of the network. At the same time, in order to reduce overfitting and solve the gradient disappearance problem, add BatchNorm and Relu layers in the middle of the convolution layer to define the initial input x (0) =X, the calculation process between convolutional layers is as follows: z (i) =Conv(x (i-1) ) With (i) =BatchNorm(from (i) ) x (i) =Relu(from (i) ) Where G = x (i) It is a multimodal feature, representing the output of the i-th layer, and the output feature is obtained after multiple convolutional layers. In multi-layer convolution operations, the interactive information between video features and text features is transmitted layer by layer, thus achieving more sophisticated inter-modal fusion; The multimodal feature G is input. In order to filter and control the information, a gating mechanism is used to selectively transmit or shield certain information and filter redundant features by using the outputs of multiple convolutional layers as weights. The formula is as follows: weight = sigmoid(G) F * =weight·V+(1-weight)·X Among them, weight is the weight, sigmoid is the activation function that processes the feature into the weight, is the fusion feature after filtering according to the weight; for the three fusion features F obtained by the three different feature compression methods * , the three features are combined using the Hadamard product, specifically: F=F meanpooling ⊙F maxpooling ⊙F GRU Among them, F represents the fused cross-modal feature, which not only includes the spatiotemporal information of the video, but also includes the semantic information of the text, and can more accurately reflect the correspondence between the video content and the text query.

9. A video moment retrieval and highlight segment detection system as claimed in claim 6, characterized in that: The sample construction unit is used to input the original features in the extracted video, generate negative sample pairs using given video-text pairs, and finally generate a negative sample set, so that the association between the video and the text is unique and the diversity of the training samples is increased, specifically: A given training video query pair is defined as a positive sample pair, and videos and queries from different sample pairs are mixed to form the negative sample pair, and the difference between the negative sample pair and the positive sample pair is used to assist the training, the positive video query pair generates different significance scores according to the query relevance, and the irrelevant negative video query pair is forced to generate the lowest significance score; Positive sample pairs will cause highlight detection loss L hd and the negative pair loss L that suppresses negative pairs neg The negative sample pairs will make it rise, and the loss function of suppressing the negative sample pairs is expressed as follows: in, represents the generated negative sample set, and S(·) represents the saliency prediction head.

10. A video moment retrieval and highlight segment detection system as claimed in claim 9, characterized in that: The loss calculation unit is used to input the fused cross-modal features and negative sample set, calculate the saliency scores of the interval of moment retrieval and highlight detection through the saliency prediction head and feature decoder, and respectively calculate the losses of moment retrieval and highlight detection to generate the overall loss function of the model, which is specifically: In the saliency prediction head S(·), the randomly initialized learnable parameter y is used as a saliency marker and is input into the DETR encoder-decoder structure together with the fused cross-modal features, so that the saliency marker can learn relevant contextual information, as shown in the formula: Among them, S(F) is the calculated significance score, which is used as the prediction result of highlight detection, and W x and W F They are all learnable matrices, d is the adjustment factor parameter, and the interval is represented by the dynamic two-dimensional anchor box center point center and offset offset as the result of moment retrieval; The loss function includes the time retrieval L mr , High light detection L hd and contrast loss L neg , where L mr It mainly includes the mean absolute error loss L1 and the improved intersection-over-union loss gIoU, and its formula is as follows: L mr =λ L1 ‖m c -m′ c ‖+λ gIoU L gIoU (m,m′) Among them, m and m ′ is the true value and the predicted value, the mean absolute error loss L1 = ‖m c -m′ c ‖ is calculated from the mean absolute error of the center point of the interval, and the improved intersection-over-union loss gIoU is calculated from the intersection of the true interval and the predicted interval composed of the center point and the offset, λ * All represent hyperparameters used to balance the loss; The loss of HD includes the difference sorting loss L margin and the ranking-aware contrast loss L cons , the difference ranking loss is a loss function used to train the ranking model. It is usually used to learn how to correctly sort them given positive and negative sample pairs. The basic idea is to ensure that the score of the positive sample is higher than the score of the negative sample by a fixed difference, as follows: L margin =max(0,δ+S(M low )-S(M high )) Where δ represents the difference, M low and M high Represent the significance marks of positive samples and negative samples respectively, and S represents the significance score; The ranking-aware loss is used to minimize the loss of all sample pairs, so that the representation of positive samples is closer in the feature space, and the ranking of positive samples in the entire dataset is higher than that of negative samples. Given a maximum significance score threshold R in a batch, it is iterated R times in the batch, and each time a positive sample set is constructed using the iteration index r∈{0,1,...,R-1} and negative sample set The dividing line between the positive and negative sample sets is the iterative index r, which is calculated as follows: Among them, τ is a tuning parameter, and the loss of highlight detection is expressed as follows: L hd =λ margin L margin +λ cons L cons Among them, λ * is a hyperparameter, and the final overall loss is expressed as follows: L=L mr +L hd +λ neg L neg Where λ is a hyperparameter, and L represents the overall loss function of the model.

Images