Video highlight detection method based on weak supervision multi-mode large model

Abstract

The invention discloses a video highlight detection method based on a weak supervision multi-mode large model, and the method comprises the steps: constructing the pairing data of a video clip and text description based on a video language large model, and configuring a highlight detection module for carrying out the significance score calculation of a video frame according to a highlight detection instruction, a highlight detection model is obtained through training in a multi-stage progressive training mode and is used for highlight detection of a video. According to the method, the precision and efficiency of highlight detection are remarkably improved by generating a high-quality pseudo tag and designing a time-sequence-sensitive highlight detection module and a progressive multi-stage training strategy, and the method has wide practical application value in scenes such as content optimization and personalized recommendation, video editing and manufacturing, social media, education and training and the like.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, relates to video understanding technology, and is a video highlight detection method based on a weakly supervised multimodal large model. Background Art

[0002] Video Highlight Detection (VHD) is a technology designed to automatically extract the most attractive or most critical segments from long videos. It is widely used in video summarization, content recommendation, automatic editing, and sports event analysis. With the explosive growth of video content on the Internet and social media platforms, how to efficiently screen out highlights has become a major challenge. Early methods relied on manual features and traditional machine learning models, with limited results; with the development of deep learning and multimodal models, highlight detection technology that combines visual and language information has significantly improved detection accuracy and efficiency, providing new possibilities for improving user experience and assisting content creation.

[0003] Existing video highlight detection solutions are usually implemented through deep learning models, mainly including methods based on visual features and multimodal information fusion. First, the visual features of the video are extracted using convolutional neural networks (CNN) or long short-term memory networks (LSTM) to capture object, action and scene information in key frames. Then, some solutions also pair videos with text, such as subtitles or descriptions, and use multimodal models such as Transformer to fuse visual and language information to enhance the recognition of highlight clips. Finally, the model is trained through supervised learning or weakly supervised learning using manually annotated datasets (such as TVSum, YouTube HL, etc.) to automatically predict the saliency score of each video clip, thereby identifying and extracting the most attractive highlight moments. Some existing technical solutions are as follows:

[0004] (1) Patent application CN112801762A “Multimodal video highlight detection method and system based on product perception”. This method extracts the self-attention semantic representation of video clips through the Bi-GRU network; extracts the lexical features of video titles, product tags and product names through text preprocessing, and then constructs and updates the graph structure to obtain a syntax-aware text representation. Next, the video and text information are fused through the common attention module to obtain a cross-modal semantic representation set, and the confidence of each video clip is calculated using a fully connected layer containing a sigmoid function. Finally, candidate clips are screened based on the confidence, an undirected graph is constructed, and a graph convolutional network is used to predict the final highlight moment. However, the text information of this method comes from product names, tags and video titles. If these texts are noisy or erroneous, the model output may be unstable or biased, thereby reducing the detection accuracy. This method relies heavily on rich information such as product names, tags and video titles. If the video data lacks relevant text label information or the text information cannot accurately reflect the video content, the model performance may be greatly reduced.

[0005] (2) Patent application CN115294506A “A video highlight detection method and device”. This method combines manual annotation of the video, playback information and barrage information to determine the frame-level highlight annotation of the video, and obtains the focus position information of the annotator during the video playback process to determine the pixel-level position annotation of the video. The video data is feature encoded by the convolution module, and the video highlight detection module is trained. This invention can achieve pixel-level highlight detection and improve the accuracy of video highlight detection. The frame-level and pixel-level highlight annotation of this method requires detailed manual annotation of each video, which is very time-consuming and labor-intensive on large-scale data sets and requires a lot of human resources. The frame-level and pixel-level highlight annotation requires detailed manual annotation of each video, which is very time-consuming and labor-intensive on large-scale data sets and requires a lot of human resources. The fine-grained requirements of pixel-level annotation are prone to introduce annotation deviations, and how to accurately define the highlight boundary may vary due to the subjective judgment of the annotator. This inconsistent annotation may cause the model to learn inconsistent features, thereby reducing the detection effect.

[0006] (3) Patent application CN117058571A “A noise-robust multimodal video highlight detection learning method”. This method takes the visual and audio modal representations of the video as input, and compresses the high-dimensional features to low dimensions through linear mapping to reduce the number of parameters and video memory usage, and improve the running speed. This method combines the global and local spatial features within the video clips and the temporal features between the clips, and uses a multimodal interactive fusion module to fuse the visual and audio representations. This method effectively improves the accuracy and robustness of video highlight detection, has a wide range of applications, and is particularly suitable for processing data with noisy labels. This method uses linear mapping to compress high-dimensional features to low dimensions to reduce the number of parameters and video memory usage, but this compression process may cause the loss of some important information, thereby affecting the final detection performance of the model, especially when detecting fine-grained highlight clips. Summary of the invention

[0007] The technical problem to be solved by the present invention is: as described in the background technology, the existing video highlight detection technology mainly relies on a large amount of manually annotated data and a single-stage training method, which makes it difficult to fully utilize the massive amount of unlabeled data, resulting in insufficient generalization ability and data utilization of the model. In addition, the existing methods are not deep enough in fusion of multimodal information, are not sensitive to numerical errors in saliency scores, and are difficult to accurately capture key segments in complex scenes.

[0008] The technical solution of the present invention is: a video highlight detection method based on a weakly supervised multimodal large model, based on a video language large model, constructing paired data of video clips and text descriptions, and configuring a highlight detection module for calculating the significance score of video frames according to highlight detection instructions, and training a highlight detection model through a multi-stage progressive training method for highlight detection of videos, including the following steps:

[0009] S1. Taking video frames, video description instructions and corresponding subtitles as input, the visual language big model is used to generate highlight detection pseudo-label data consisting of video clips and description subtitles: the attention score between the visual token in the video and the text token of the corresponding subtitle is calculated, and the contribution of the visual token to the core element is quantified using the attention score. The total contribution score of each visual token is accumulated to obtain its total contribution score; at the frame level, the average of the total contribution scores of all visual tokens is calculated to obtain the pseudo-saliency score of the frame, and then the pseudo-saliency score at the frame level is normalized to obtain the highlight pseudo-label;

[0010] S2. Construct a time-series numerically sensitive highlight detection module, take video frames and highlight detection instructions as inputs of the visual language large model, and the highlight detection module predicts the significance score of the output, introduces a learnable conditional embedding, and the learnable conditional embedding is input into the visual language large model together with the highlight detection instructions and video frames. The highlight detection module is based on the Transformer decoder architecture, and uses the output of the corresponding conditional embedding part of the visual language large model as the input part of the cross attention of the highlight detection module, that is, as the key and value of the cross attention. Through autoregression, the highlight detection module adaptively processes the video frames of the variable length sequence and outputs the significance score of each frame.

[0011] S3, multi-stage progressive training stage 1, weakly supervised learning based on pseudo-labels: The highlight detection model is composed of a large video language model and a highlight detection module, and weakly supervised training is performed using the highlight pseudo-label data generated in step S1. During the training process, hinge loss is used as the loss function to optimize the gap between the significance score of the highlight detection module and the highlight pseudo-label;

[0012] S4, multi-stage progressive training stage 2, supervised learning based on manually labeled data: introduce manual labeling of highlight detection, and perform supervised learning on the highlight detection module in the highlight detection model trained in step S3 to improve the prediction accuracy of the highlight detection module for the significance score;

[0013] The highlight detection model, which has undergone multi-stage progressive training, is used to perform highlight detection on input videos and highlight detection instructions.

[0014] Furthermore, the prediction results of the highlight detection model are normalized according to step S1 to obtain updated highlight pseudo-labels; then, these updated highlight pseudo-labels are used to replace the pseudo-label data used in step S3, and the training processes of stage one and stage two are executed. The pseudo-label updating and multi-stage progressive training are repeated several times, and the highlight detection model is continuously trained during multiple iterations until the task indicators of the highlight detection model on the specified validation set converge.

[0015] Furthermore, the highlight detection model outputs a saliency score for each frame of the video and selects the K frames with the highest scores to form a highlight video clip.

[0016] Furthermore, the highlight detection model is implemented by at least one instruction or at least one program, and the at least one instruction or at least one program is used to be executed by a data processor through remote call or local call to implement highlight detection on the video.

[0017] The present invention proposes a method for highlight detection based on a large multimodal model of video language. By generating high-quality pseudo labels, designing a highlight detection module that is sensitive to temporal values, and using a progressive multi-stage training strategy, the accuracy and efficiency of highlight detection are significantly improved, effectively solving the problems of high data dependence, insufficient information fusion, and weak temporal processing capabilities in the prior art. Compared with the prior art, the present invention has the following advantages.

[0018] 1) This paper proposes a pseudo-label generation strategy based on a multimodal large model, which automatically generates pseudo-label data by calculating the attention score between vision and text. This method significantly reduces the cost of manual labeling and utilizes a large amount of unlabeled data, providing rich materials for model training.

[0019] 2) This paper designs a temporal value-sensitive highlight detection module that can process variable-length sequences of video frames and introduces learnable conditional embedding to enhance the interaction between visual information and query information. This design enables the model to more accurately capture the temporal features and dynamic changes in the video, thereby improving the accuracy of highlight detection.

[0020] 3) The present invention adopts a multi-stage progressive training scheme, including weakly supervised learning based on pseudo-labels and supervised learning based on manually annotated data, as well as an iterative self-training strategy. This training scheme fully utilizes the advantages of pseudo-label data and manually annotated data, and realizes coarse-to-fine highlight detection by gradually optimizing model parameters and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the highlight detection model constructed in the method of the present invention.

[0022] Figure 2 Schematic diagram of the multi-stage progressive training process of the highlight detection model in the method of the present invention. DETAILED DESCRIPTION

[0023] The present invention provides a highlight detection method based on a multimodal large model, designs a highlight detection module that is sensitive to pseudo-label generation, temporal numerical values, and a multi-stage progressive training scheme.

[0024] First, the present invention generates pseudo-label data of video clips and text descriptions through a multimodal large model, and uses attention scores to quantify the contribution of visual tokens to core elements such as objects, actions, events, etc. This pseudo-label generation method combines visual and language information and performs normalization of saliency scores to form pseudo-labels for highlight detection.

[0025] Secondly, the present invention designs a highlight detection module that is sensitive to temporal values, adopts learnable conditional embedding, interacts with query statements and video features, and predicts significance scores based on the Transformer decoder architecture. The highlight detection module introduces a cross-attention mechanism to adapt to the video frame input of variable-length sequences, and realizes sensitivity to temporal values ​​of videos of arbitrary lengths. Ensuring the accuracy of significance score prediction and the ability to process variable-length sequences is an important step in improving the accuracy of highlight detection. The present invention constructs pairing data of video clips and text descriptions through a large video language model, and combines the significance score calculation of the highlight detection module to provide effective training data and optimization strategies for subsequent video highlight detection models.

[0026] The present invention also proposes a progressive multi-stage training strategy, which is divided into weakly supervised learning based on pseudo-labels and supervised learning based on manually labeled data, and further improves the model performance through iterative self-training. This training scheme not only optimizes the initial weights of the model, but also improves the model performance through continuous iteration, effectively utilizing massive unlabeled data, and is the key to achieving high model accuracy.

[0027] The implementation of the present invention will be described in detail in the following steps. The model structure corresponding to the method of the present invention is as follows: Figure 1 As shown in Figure 2, the process of the training phase is as follows: Figure 2 shown.

[0028] Step S1: Pseudo-label data preparation and pseudo-label construction for highlight detection.

[0029] First, collect a large number of video clips and their corresponding captions, and use them as input to mature video language models, such as LLaVA-NeXT-Video. These models can handle paired inputs of video and text, and the data format is usually as follows:

[0030]

[0031] Among them, "video" is the original video path, "i" in "qa" is the query statement, and "a" is the stored text description. Then, the multimodal large model is used to calculate the attention score between the visual token in the video and the text token in the corresponding caption. The attention score is used to quantify the contribution of each visual token in the caption generation process. Figure 1 The video-level annotations shown in are the description subtitles corresponding to the video.

[0032] In particular, in the highlight detection task, the saliency score is usually closely related to the objects, actions, or events involved in the query sentence. In order to extract this key information, we first use natural language processing tool libraries such as spaCy and nltk to process the captions of the video and extract the core elements in the sentence, such as nouns and verbs. Next, we use the attention scores of the visual tokens in the video language model when generating these key information to calculate the contribution value of each visual token to these elements.

[0033] Specifically, assume that the key information set extracted from the caption is K = {k1, k2, …, k n}, the visual token set corresponding to a certain video frame is denoted as V = {v1,v2,…,v m}, then each visual tokenv j Generate key information k i The attention score is A(v j ,k i ), where A is the attention module calculation function of the output layer of the selected multimodal video language model. By accumulating the contribution of each visual token, its total contribution score is obtained.

[0034]

[0035] At the frame level, the average contribution value of all visual tokens is calculated to obtain the pseudo-saliency score of the frame.

[0036]

[0037] After that, the frame-level saliency scores are normalized to adjust the scores to the standard range of highlight detection, usually 1 to 5, to reflect the importance of different frames. The normalization formula is:

[0038]

[0039] Where S' frame Represents the normalized pseudo-saliency score. The highlight detection saliency score is usually 1-5, so the formula requires values ​​1 and 4 to expand and shift the value range. The values ​​1 and 4 change as the standard range changes. These saliency scores form pseudo-labels for highlight detection, which are used to identify the most attractive segments in the video.

[0040] Step S2: Design of a timing value-sensitive highlight detection module.

[0041] During the pre-training and fine-tuning process, the large video language model is often trained using the cross entropy loss function, which is insensitive to the numerical error of the significance score. Directly introducing loss functions such as mean square error can improve the accuracy of significance score prediction, but it is easy to destroy the generation ability of the large model itself. In order to solve this problem, the present invention designs a time series value-sensitive highlight detection module, which is specifically used for the prediction of significance scores.

[0042] The highlight detection module introduces learnable conditional embeddings, which are input into the multimodal large model together with query statements and video features so that the model can fully interact with visual information and query information. Specifically, the highlight detection module is based on a simple Transformer decoder architecture, and uses the output of the conditional embedding part of the large model as the input of the cross-attention of the highlight detection module, which is mapped to keys and values ​​through different linear layers of the cross-attention layer. Through autoregression, the module can adaptively process variable-length sequences of video frames and output the saliency score of each frame.

[0043] Step S3: Progressive multi-stage training - Stage 1, weakly supervised learning based on pseudo labels.

[0044] The training of the highlight detection module is divided into multiple stages. The first stage is weakly supervised learning based on pseudo-labels. The pseudo-label data generated in step S1 is used for training. During the training process, hinge loss is used as the loss function to help optimize the gap between the prediction of the highlight detection module and the pseudo-label. At this stage, although the pseudo-label data has not been manually annotated, its quantity is huge, which can help the model obtain good initialization weights in the early training. Through pseudo-label training, the model gradually acquires a certain ability to predict the significance score.

[0045] Step S4: Progressive multi-stage training - Stage 2, supervised learning based on manually labeled data.

[0046] Although pseudo-label data can provide a large number of training samples to help the model perform preliminary training, the model's prediction still needs further calibration due to the inevitable noise in the pseudo-label data. In the second stage, the present invention introduces manually annotated high-quality datasets, such as TVSum, YouTube HL, QVHighlights and other public highlight detection datasets. Similar to the first stage, this stage uses the same training process and uses the good weights obtained in the first stage training as initialization, which greatly reduces the demand for high-quality annotated data.

[0047] At this stage, the accuracy of the highlight detection module is further improved by training on manually annotated high-quality data to ensure the accuracy and stability of the significance score prediction results.

[0048] After two stages of training, the resulting highlight detection model has a high prediction accuracy. During detection, the video and highlight detection instructions are input. The highlight detection instructions are used to indicate the scene that the user wants to highlight. For example, for the provided video, the highlighted scenes are marked according to the description, i.e., the highlight key content, the timestamps corresponding to these highlight contents are recorded and their significance scores are evaluated to complete the highlight detection task.

[0049] Step S5: Iterative self-training.

[0050] In order to further improve the performance of the model, the method of the present invention can optionally introduce an iterative self-training strategy. In this process, the prediction results of the highlight detection model that has undergone two-stage training for the first time, that is, the significance score of each frame, are normalized according to S1 to obtain new highlight pseudo-labels, generating higher quality pseudo-label data. Subsequently, these updated pseudo-labels are used to replace the pseudo-label data used in step S3, and stage one and stage two are executed, corresponding to the training process of steps S3-S4. The above steps are repeated several times, and the model is continuously trained during multiple iterations until its task indicators on the specified validation set converge.

[0051] This iterative self-training method can effectively utilize massive video text data and alleviate the noise problem in the initial pseudo-labels, so that the model can gradually improve its ability to recognize highlight clips in each round of iteration.

[0052] Finally, the highlight detection model outputs a saliency score for each frame of the video and selects the K frames with the highest scores to form a highlight video clip.

[0053] The present invention is implemented based on a computer program, and the highlight detection model is implemented by at least one instruction or at least one program. The at least one instruction or at least one program is executed by a data processor through a remote call or a local call to realize highlight detection of the video. The instructions or program codes for implementing the method of the present invention can be written in any combination of one or more programming languages. The instructions or program codes can be executed completely on the data processor, partially on the processor, partially on the processor as an independent software package and partially on the remote device, or completely on the remote device or server.

[0054] The video highlight detection method based on the multimodal large model of video language of the present invention integrates technical solutions such as multimodal fusion, automatic pseudo-label generation and efficient detection module, and has a wide range of practical application value in scenarios such as content optimization and personalized recommendation, video editing and production, social media, education and training. By accurately capturing the highlight moments in the video, it can be used to provide users with more personalized content recommendations and enhance the user experience. For the video editing and production industry, the present invention can quickly locate and extract key key segments in the video, saving a lot of time for video editing and improving production efficiency. The highlight detection method of the present invention can also be used on social media platforms to automatically generate highlight short video clips to assist users in quickly extracting and understanding video content. In the field of education and training, it can help teachers quickly find the key points and highlights in the course, and provide students with a more efficient learning experience.

Claims

1. A video highlight detection method based on a weakly supervised multimodal large model, characterized by Based on the video language model, we build paired data of video clips and text descriptions, and configure a highlight detection module to calculate the saliency score of video frames according to the highlight detection instructions. We train the highlight detection model through a multi-stage progressive training method to detect highlights in the video, including the following steps: S1. Taking video frames, video description instructions and corresponding subtitles as input, the visual language big model is used to generate highlight detection pseudo-label data consisting of video clips and description subtitles: the attention score between the visual token in the video and the text token of the corresponding subtitle is calculated, and the contribution of the visual token to the core element is quantified using the attention score. The total contribution score of each visual token is accumulated to obtain its total contribution score; at the frame level, the average of the total contribution scores of all visual tokens is calculated to obtain the pseudo-saliency score of the frame, and then the pseudo-saliency score at the frame level is normalized to obtain the highlight pseudo-label; S2. Construct a time-series numerically sensitive highlight detection module, take video frames and highlight detection instructions as inputs of the visual language large model, and the highlight detection module predicts the significance score of the output, introduces a learnable conditional embedding, and the learnable conditional embedding is input into the visual language large model together with the highlight detection instructions and video frames. The highlight detection module is based on the Transformer decoder architecture, and uses the output of the corresponding conditional embedding part of the visual language large model as the input part of the cross attention of the highlight detection module, that is, as the key and value of the cross attention. Through autoregression, the highlight detection module adaptively processes the video frames of the variable length sequence and outputs the significance score of each frame. S3, multi-stage progressive training stage 1, weakly supervised learning based on pseudo-labels: The highlight detection model is composed of a large video language model and a highlight detection module, and weakly supervised training is performed using the highlight pseudo-label data generated in step S1. During the training process, hinge loss is used as the loss function to optimize the gap between the significance score of the highlight detection module and the highlight pseudo-label; S4, multi-stage progressive training stage 2, supervised learning based on manually labeled data: introduce manual labeling of highlight detection, and perform supervised learning on the highlight detection module in the highlight detection model trained in step S3 to improve the prediction accuracy of the highlight detection module for the significance score; The highlight detection model, which has undergone multi-stage progressive training, is used to perform highlight detection on input videos and highlight detection instructions.

2. According to claim 1, a video highlight detection method based on a weakly supervised multimodal large model is characterized by: The method also includes step S5: normalizing the prediction results of the highlight detection model according to step S1 to obtain updated highlight pseudo-labels; then, using these updated highlight pseudo-labels to replace the pseudo-label data used in step S3, executing the training processes of stage one and stage two, updating the pseudo-labels and repeating the multi-stage progressive training for several times, and continuously training the highlight detection model during multiple iterations until the task indicators of the highlight detection model on the specified validation set converge.

3. According to the method of video highlight detection based on weakly supervised multimodal large model in claim 1 or 2, it is characterized in that in step S1, the subtitles of the video are first processed using the natural language processing tool library to extract the core elements in the sentence, and the attention score of the visual token in the multimodal model when generating these key information is used to calculate the contribution value of each visual token to these elements: let the key information set extracted from the subtitles be K = {k1, k2, ..., k n }, the visual token set corresponding to a certain video frame is denoted as V = {v1,v2,…,v m }, then each visual token v j Generate key information k i The attention score is A(v j ,k i ), where A is the attention module calculation function of the output layer of the selected multimodal video language model.

4. According to claim 3, a video highlight detection method based on a weakly supervised multimodal large model is characterized by: The total contribution score of visual tokens is: At the frame level, the pseudo-saliency score of a frame is: The normalization of the frame-level pseudo-saliency score is: S' frame represents the normalized pseudo-significance score.

5. The video highlight detection method based on weakly supervised multimodal large model according to claim 1 or 2, characterized in that The highlight detection model outputs a saliency score for each frame of the video and selects the K frames with the highest scores to form a highlight video clip.

6. The video highlight detection method based on weakly supervised multimodal large model according to claim 1 or 2, characterized in that The highlight detection model is implemented by at least one instruction or at least one program, and the at least one instruction or at least one program is used to be executed by a data processor through a remote call or a local call to implement highlight detection on the video.

Images