A Video Spectrum Detection Method Based on Weakly Supervised Multimodal Large Model

Abstract

A video specular highlight detection method based on a weakly supervised multimodal large model is proposed. This method utilizes a video language large model to construct paired data between video clips and text descriptions. A specular highlight detection module is configured to calculate saliency scores for video frames based on specular highlight detection instructions. The specular highlight detection model is trained through a multi-stage progressive training approach for specular highlight detection in videos. This invention significantly improves the accuracy and efficiency of specular highlight detection by generating high-quality pseudo-labels, designing a time-sensitive specular highlight detection module, and employing a progressive multi-stage training strategy. It has broad practical application value in scenarios such as content optimization and personalized recommendation, video editing and production, social media, and education and training.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically video understanding technology, and is a video specular detection method based on a weakly supervised multimodal large model. Background Technology

[0002] Video Highlight Detection (VHD) is a technique designed to automatically extract the most attractive or crucial segments from long videos, widely used in video summarization, content recommendation, automatic editing, and sports event analysis. With the explosive growth of video content on internet and social media platforms, efficiently filtering out the most exciting moments has become a major challenge. Early methods relied on handcrafted features and traditional machine learning models, with limited effectiveness. However, with the development of deep learning and multimodal models, VHD techniques combining visual and linguistic information have significantly improved detection accuracy and efficiency, offering new possibilities for improving user experience and assisting content creation.

[0003] Existing video highlight detection solutions typically employ deep learning models, primarily including methods based on visual features and multimodal information fusion. First, convolutional neural networks (CNNs) or long short-term memory networks (LSTMs) are used to extract visual features from the video, capturing object, action, and scene information in keyframes. Next, some solutions pair the video with text, such as captions or descriptions, using multimodal models like the Transformer to fuse visual and linguistic information, enhancing the ability to recognize highlight segments. Finally, the model is trained using manually labeled datasets (such as TVSum, YouTube HL, etc.) through supervised or weakly supervised learning to automatically predict the saliency score of each video segment, thereby identifying and extracting the most attractive highlight moments. Some existing technical solutions are as follows:

[0004] (1) Patent application CN112801762A, "A Multimodal Video Highlight Detection Method and System Based on Product Awareness." This method extracts self-attention semantic representations of video segments through a Bi-GRU network; through text preprocessing, it extracts lexical features of video titles, product tags, and product names, and then constructs and updates the graph structure to obtain syntactic-aware text representations. Next, through a common attention module, the video and text information are fused to obtain a cross-modal semantic representation set, and the confidence score of each video segment is calculated using a fully connected layer containing a sigmoid function. Finally, candidate segments are selected based on the confidence scores, an undirected graph is constructed, and a graph convolutional network is used to predict the final highlight moments. However, the text information of this method comes from product names, tags, and video titles. If these texts contain noise or errors, it may cause unstable or biased model outputs, thereby reducing the detection accuracy. This method is highly dependent on rich information such as product names, tags, and video titles. If the video data lacks relevant text tag information, or if the text information cannot accurately reflect the video content, the model performance may drop significantly.

[0005] (2) Patent application CN115294506A, "A Video Highlight Detection Method and Apparatus." This method combines manual annotation of the video, playback information, and bullet screen information to determine the frame-level highlight annotation of the video, and obtains the attention position information of the annotator during video playback to determine the pixel-level position annotation of the video. The video data is feature-encoded using a convolutional module to train the video highlight detection module. This invention can achieve pixel-level highlight detection, improving the accuracy of video highlight detection. However, this method requires detailed manual annotation for each video at both the frame-level and pixel-level highlight annotation, which is very time-consuming and labor-intensive on large-scale datasets and requires a large amount of human resources. The fine-grained requirement of pixel-level annotation can easily introduce annotation bias; how to accurately define the boundary of the highlight may differ due to the annotator's subjective judgment. This inconsistency in annotation may cause the model to learn inconsistent features, thereby reducing the detection effect.

[0006] (3) Patent application CN117058571A, "A Noise-Robust Multimodal Video Spectrum Detection Learning Method." This method takes the visual and audio modal representations of the video as input, and compresses high-dimensional features to low dimensions through linear mapping to reduce the number of parameters and memory usage, and improve the running speed. This method combines global and local spatial features within video segments and temporal features between segments, and uses a multimodal interactive fusion module to fuse visual and audio representations. This method effectively improves the accuracy and robustness of video spectrum detection, has a wide range of applications, and is especially suitable for processing data with noisy labels. This method uses linear mapping to compress high-dimensional features to low dimensions to reduce the number of parameters and memory usage, but this compression process may lead to the loss of some important information, thus affecting the final detection performance of the model, especially in fine-grained spectrum segment detection. Summary of the Invention

[0007] The technical problem this invention aims to solve is that, as described in the background section, existing video specular detection technologies mainly rely on large amounts of manually labeled data and single-stage training methods, making it difficult to fully utilize massive amounts of unlabeled data, resulting in insufficient generalization ability and data utilization of the model. Furthermore, existing methods lack sufficient depth in fusing multimodal information and are insensitive to numerical errors in saliency scores, making it difficult to accurately capture key segments in complex scenes.

[0008] The technical solution of this invention is as follows: a video specular highlight detection method based on a weakly supervised multimodal large model. Based on a video language large model, it constructs paired data of video segments and text descriptions, and configures a specular highlight detection module to calculate the saliency score of video frames according to specular highlight detection instructions. A specular highlight detection model is trained through a multi-stage progressive training method for specular highlight detection in videos, including the following steps:

[0009] S1. Using video frames, video description instructions, and corresponding subtitles as input, a visual language big data model is used to generate specular detection pseudo-label data consisting of video clips and description subtitles: the attention score between visual tokens in the video and text tokens of corresponding subtitles is calculated, and the contribution of visual tokens to core elements is quantified using the attention score. The total contribution score is obtained by accumulating the contributions of each visual token. At the frame level, the average of the total contribution scores of all visual tokens is calculated to obtain the pseudo-salience score of the frame. The frame-level pseudo-salience scores are then normalized to obtain specular pseudo-labels.

[0010] S2. Construct a time-series numerically sensitive specular detection module. The video frames and specular detection instructions are used as inputs to the visual language big model. The specular detection module predicts the saliency score of the output and introduces learnable conditional embeddings. The learnable conditional embeddings are input into the visual language big model along with the specular detection instructions and video frames. The specular detection module is based on the Transformer decoder architecture. The output of the corresponding conditional embedding part of the visual language big model is used as the input part of the cross-attention of the specular detection module, that is, as the key and value of cross-attention. Through an autoregressive method, the specular detection module adaptively processes the variable-length sequence of video frames and outputs the saliency score of each frame.

[0011] S3, Multi-stage progressive training stage one, weakly supervised learning based on pseudo-labels: The highlight detection model is composed of a large video language model and a highlight detection module. Weakly supervised training is performed using the highlight pseudo-label data generated in step S1. During the training process, hinge loss is used as the loss function to optimize the gap between the significance score of the highlight detection module and the highlight pseudo-label.

[0012] S4. Multi-stage progressive training stage two: supervised learning based on manually labeled data: introduce manually labeled specular detection data, and supervise the specular detection module in the specular detection model trained in step S3 to improve the prediction accuracy of the specular detection module for saliency score.

[0013] The specular detection model, after multi-stage progressive training, is used to perform specular detection on the input video and specular detection commands.

[0014] Furthermore, the prediction results of the specular detection model are normalized according to step S1 to obtain updated specular pseudo-labels; then, these updated specular pseudo-labels are used to replace the pseudo-label data used in step S3, and the training process of stage one and stage two is executed. The pseudo-labels are updated and the multi-stage progressive training is repeated several times. The specular detection model is continuously trained in multiple iterations until the task index of the specular detection model converges on the specified validation set.

[0015] Furthermore, the highlight detection model scores the saliency of each frame in the video output and selects the K frames with the highest scores to form a highlight video clip.

[0016] Furthermore, the highlight detection model is implemented by at least one instruction or at least one program segment, wherein the at least one instruction or at least one program segment is executed by the data processor via remote or local invocation to perform highlight detection on the video.

[0017] This invention proposes a specular highlight detection method based on a multimodal large model of video language. By generating high-quality pseudo-labels, designing a time-sensitive specular highlight detection module, and employing a progressive multi-stage training strategy, it significantly improves the accuracy and efficiency of specular highlight detection, effectively solving the problems of high data dependence, insufficient information fusion, and weak temporal processing capabilities in existing technologies. Compared with existing technologies, it has the following advantages.

[0018] 1) This invention proposes a pseudo-label generation strategy based on a multimodal large model, which automatically generates pseudo-label data by calculating the attention score between visual and textual data. This method significantly reduces the cost of manual annotation while utilizing a large amount of unlabeled data, providing rich material for model training.

[0019] 2) This invention designs a time-sensitive specular detection module that can process video frames of variable length and introduces learnable conditional embeddings to enhance the interaction between visual and query information. This design enables the model to more accurately capture temporal features and dynamic changes in the video, thereby improving the accuracy of specular detection.

[0020] 3) This invention employs a multi-stage progressive training scheme, including weakly supervised learning based on pseudo-labels and supervised learning based on manually labeled data, as well as an iterative self-training strategy. This training scheme fully utilizes the advantages of pseudo-labeled data and manually labeled data, achieving coarse-to-fine specular detection by progressively optimizing model parameters and performance. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the specular detection model constructed in the method of the present invention.

[0022] Figure 2 This is a schematic diagram of the multi-stage progressive training process of the specular detection model in the method of the present invention. Detailed Implementation

[0023] This invention provides a specular detection method based on a multimodal large model, which designs a pseudo-label generation, a specular detection module sensitive to time-series values, and a multi-stage progressive training scheme.

[0024] First, this invention generates pseudo-label data of video clips and text descriptions using a multimodal large model, and quantifies the contribution of visual tokens to core elements such as objects, actions, and events using attention scores. This pseudo-label generation method combines visual and linguistic information and performs saliency score normalization processing to form pseudo-labels for highlight detection.

[0025] Secondly, this invention designs a time-sensitive specular detection module that employs learnable conditional embeddings to interact with query statements and video features, and predicts saliency scores based on a Transformer decoder architecture. The specular detection module introduces a cross-attention mechanism to adapt to video frame inputs of variable length, achieving time-sensitive performance for videos of arbitrary length. This ensures the accuracy of saliency score prediction and the ability to handle variable-length sequences, which is a crucial step in improving specular detection accuracy. This invention constructs paired data between video segments and text descriptions using a large video language model, and combines this with the saliency score calculation from the specular detection module to provide effective training data and optimization strategies for subsequent video specular detection models.

[0026] This invention also proposes a progressive multi-stage training strategy, which consists of weakly supervised learning based on pseudo-labels and supervised learning based on manually labeled data, and further improves model performance through iterative self-training. This training scheme not only optimizes the initial weights of the model, but also improves model performance through continuous iteration, effectively utilizing massive amounts of unlabeled data, which is key to achieving high model accuracy.

[0027] The implementation of this invention will be described in detail step by step below. The model structure corresponding to the method of this invention is as follows: Figure 1 As shown, the training phase process is as follows: Figure 2 As shown.

[0028] Step S1: Pseudo-label data preparation and pseudo-label construction for specular detection.

[0029] First, a large number of video clips and their corresponding captions are collected and used as input to mature video language models, such as LLaVA-NeXT-Video. These models can handle paired video and text inputs, and the data format is typically as follows:

[0030]

[0031] Here, "video" represents the original video path, "qa" contains the query term "i" and the stored text description "a". Subsequently, a multimodal large model is used to calculate the attention score between the visual tokens in the video and the corresponding text tokens in the caption. The attention score quantifies the degree to which each visual token contributes to the caption generation process. Figure 1 The video-level annotations shown are the descriptive captions corresponding to the videos.

[0032] Specifically, in highlight detection tasks, saliency scores are typically closely related to the objects, actions, or events mentioned in the query. To extract this key information, we first use natural language processing libraries such as spaCy and nltk to process the video captions, extracting core elements such as nouns and verbs from the sentences. Next, we utilize the attention scores of visual tokens in the video language model during the generation of this key information to calculate the contribution of each visual token to these elements.

[0033] Specifically, suppose the set of key information extracted from the caption is K = {k1, k2, ..., k n Let V = {v1, v2, ..., v} be the set of visual tokens corresponding to a certain video frame. m}, then each visual tokenv j Generate key information k i The attention score is A(v j ,k i ), where A is the attention module computation function of the selected multimodal video language large model output layer. The total contribution score is obtained by summing the contributions of each visual token.

[0034]

[0035] At the frame level, the average contribution value of all visual tokens is calculated to obtain the pseudo-salience score for that frame.

[0036]

[0037] Next, the frame-level saliency scores are normalized, adjusting them to the standard range for specular detection, typically 1–5, to reflect the importance of different frames. The normalization formula is:

[0038]

[0039] Where S' frame This represents the normalized pseudo-significance score. Highlight detection significance scores are typically between 1 and 5, therefore the formula requires values ​​1 and 4 for range expansion and translation. These values ​​vary with the standard range. These significance scores form pseudo-labels for highlight detection, used to identify the most attractive segments in a video.

[0040] Step S2: Design of a time-sensitive specular detection module.

[0041] In the pre-training and fine-tuning processes of large-scale video language models, cross-entropy loss is often used for training. However, cross-entropy loss is insensitive to numerical errors in saliency scores. Directly introducing loss functions such as mean squared error can improve the accuracy of saliency score prediction, but it can easily compromise the generative capabilities of the large model itself. To address this issue, this invention designs a time-series numerically sensitive specular detection module specifically for saliency score prediction.

[0042] The specular highlight detection module introduces learnable conditional embeddings, which are input into a multimodal large model along with the query and video features, allowing the model to fully interact with visual and query information. Specifically, the specular highlight detection module is based on a naive Transformer decoder architecture, using the output of the large model's conditional embedding as the input to the module's cross-attention layer. This cross-attention layer maps the data to keys and values ​​through different linear layers. Through an autoregressive approach, the module can adaptively process variable-length video frames, outputting a saliency score for each frame.

[0043] Step S3: Progressive multi-stage training – Stage 1, weakly supervised learning based on pseudo-labels.

[0044] The training of the specular detection module is divided into several stages. The first stage is weakly supervised learning based on pseudo-labels. The pseudo-label data generated in step S1 is used for training. During training, hinge loss is used as the loss function to help optimize the gap between the specular detection module's predictions and the pseudo-labels. In this stage, although the pseudo-label data is not manually labeled, its sheer volume helps the model obtain good initial weights during early training. Through pseudo-label training, the model gradually acquires a certain saliency score prediction ability.

[0045] Step S4: Progressive multi-stage training – Stage 2, supervised learning based on manually labeled data.

[0046] While pseudo-labeled data can provide a large number of training samples to help with the initial training of the model, the model's predictions still require further calibration due to the inevitable noise present in the pseudo-labeled data. In the second stage, this invention introduces high-quality, manually annotated datasets, such as publicly available specular detection datasets like TVSum, YouTube HL, and QVHighlights. Similar to the first stage, this stage uses the same training process and employs the well-prepared weights obtained in the first stage as initialization, significantly reducing the need for high-quality labeled data.

[0047] At this stage, the accuracy of the highlight detection module is further improved by training on high-quality manually labeled data, ensuring the accuracy and stability of the saliency score prediction results.

[0048] After two stages of training, the resulting highlight detection model has achieved high prediction accuracy. During detection, the input video and highlight detection instructions are used to represent the scene that the user wants to highlight. For example, for the provided video, the highlighted scene, i.e., the highlight key content, is marked according to the description. The timestamps corresponding to these highlight contents are recorded and their salience scores are evaluated to complete the highlight detection task.

[0049] Step S5: Iterative self-training.

[0050] To further improve model performance, the method of this invention optionally introduces an iterative self-training strategy. In this process, the prediction results of the specular detection model after its initial two-stage training, i.e., the saliency score per frame, are normalized according to S1 to obtain new specular pseudo-labels, generating higher-quality pseudo-label data. Subsequently, these updated pseudo-labels replace the pseudo-label data used in step S3, executing stages one and two, corresponding to the training process of steps S3-S4. These steps are repeated several times, allowing the model to be continuously trained during multiple iterations until its task metrics converge on a specified validation set.

[0051] This iterative self-training method can effectively utilize massive amounts of video text data and alleviate the noise problem in the initial pseudo-labels, enabling the model to gradually improve its ability to recognize highlight segments in each iteration.

[0052] Finally, the highlight detection model scores the saliency of each frame in the video output and selects the K frames with the highest scores to form a highlight video clip.

[0053] This invention is based on a computer program and implements the highlight detection model through at least one instruction or at least one program segment. The at least one instruction or at least one program segment is executed by a data processor via remote or local invocation to perform highlight detection on video. The instructions or program code used to implement the method of this invention can be written in any combination of one or more programming languages. The instructions or program code can be executed entirely on the data processor, partially on the processor, or as a standalone software package partially on the processor and partially on a remote device, or entirely on a remote device or server.

[0054] This invention presents a video highlight detection method based on a multimodal large model of video language. It integrates multimodal fusion, automated pseudo-tag generation, and a high-efficiency detection module, offering broad practical application value in content optimization and personalized recommendation, video editing and production, social media, and education and training scenarios. By accurately capturing highlight moments in videos, it can provide users with more personalized content recommendations, enhancing the user experience. For the video editing and production industry, this invention can quickly locate and extract key segments from videos, saving significant time and improving production efficiency. The highlight detection method can also be used on social media platforms to automatically generate short highlight video clips, assisting users in quickly understanding video content. In the education and training field, it can help teachers quickly identify key points and highlight moments in courses, providing students with a more efficient learning experience.

Claims

1. A video specular detection method based on a weakly supervised multimodal large model, characterized by: Based on a large video language model, paired data of video clips and text descriptions are constructed, and a highlight detection module is configured to calculate the saliency score of video frames according to highlight detection instructions. A highlight detection model is trained through a multi-stage progressive training method for highlight detection in videos, including the following steps: S1. Using video frames, video description instructions, and corresponding subtitles as input, a visual language big data model is used to generate specular detection pseudo-label data consisting of video clips and description subtitles: the attention score between visual tokens in the video and text tokens of corresponding subtitles is calculated, and the contribution of visual tokens to core elements is quantified using the attention score. The total contribution score is obtained by accumulating the contributions of each visual token. At the frame level, the average of the total contribution scores of all visual tokens is calculated to obtain the pseudo-salience score of the frame. The frame-level pseudo-salience scores are then normalized to obtain specular pseudo-labels. S2. Construct a time-series numerically sensitive specular detection module. The video frames and specular detection instructions are used as inputs to the visual language big model. The specular detection module predicts the saliency score of the output and introduces learnable conditional embeddings. The learnable conditional embeddings are input into the visual language big model along with the specular detection instructions and video frames. The specular detection module is based on the Transformer decoder architecture. The output of the corresponding conditional embedding part of the visual language big model is used as the input part of the cross-attention of the specular detection module, that is, as the key and value of cross-attention. Through an autoregressive method, the specular detection module adaptively processes the variable-length sequence of video frames and outputs the saliency score of each frame. S3, Multi-stage progressive training stage one, weakly supervised learning based on pseudo-labels: The highlight detection model is composed of a large video language model and a highlight detection module. Weakly supervised training is performed using the highlight pseudo-label data generated in step S1. During the training process, hinge loss is used as the loss function to optimize the gap between the significance score of the highlight detection module and the highlight pseudo-label. S4. Multi-stage progressive training stage two: supervised learning based on manually labeled data: introduce manually labeled specular detection data, and supervise the specular detection module in the specular detection model trained in step S3 to improve the prediction accuracy of the specular detection module for saliency score. The specular detection model, after multi-stage progressive training, is used to perform specular detection on the input video and specular detection commands.

2. The video specular detection method based on a weakly supervised multimodal large model according to claim 1, characterized in that: The process also includes step S5: normalizing the specular detection model prediction results according to step S1 to obtain updated specular pseudo-labels; then, using these updated specular pseudo-labels to replace the pseudo-label data used in step S3, executing the training process of stage one and stage two, updating pseudo-labels and repeating the multi-stage progressive training several times, continuously training the specular detection model in multiple iterations until the specular detection model converges on the specified validation set.

3. A video specular highlight detection method based on a weakly supervised multimodal large model according to claim 1 or 2, characterized in that, in step S1, the video subtitles are first processed using a natural language processing toolkit to extract the core elements of the sentences, and the attention scores of visual tokens in the multimodal model when generating these key information are used to calculate the contribution value of each visual token to these elements: Let the set of key information extracted from the subtitles be K = {k1, k2, ..., k n Let V = {v1, v2, ..., v} be the set of visual tokens corresponding to a certain video frame. m }, then each visual token v j Generate key information k i The attention score is A(v j ,k i ), where A is the attention module computation function of the selected multimodal video language large model output layer.

4. The video specular detection method based on a weakly supervised multimodal large model according to claim 3, characterized in that: The total contribution score for visual tokens is: At the frame level, the pseudo-significance score of a frame is: The normalization process for frame-level pseudo-saliency scores is as follows: S' frame This represents the spurious significance score after normalization.

5. A video specular detection method based on a weakly supervised multimodal large model according to claim 1 or 2, characterized in that: The highlight detection model scores the saliency of each frame in the video output and selects the K frames with the highest scores to form a highlight video clip.

6. A video specular detection method based on a weakly supervised multimodal large model according to claim 1 or 2, characterized in that: The highlight detection model is implemented by at least one instruction or at least one program segment, wherein the at least one instruction or at least one program segment is executed by the data processor via remote or local invocation to perform highlight detection on the video.

Images