Lens label extraction method and device based on full-modal understanding and medium

Abstract

The invention provides a full-modal understanding-based lens label extraction method and device and a medium, and relates to the technical field of video processing, and the method comprises the steps: segmenting an original video into a plurality of segment units with complete semanteme, and labeling each segment with a mirror image attribute label to form a labeling data set; performing supervised training by using the annotation data set based on the visual language model; the visual language model adopts a Qwen 2.5-VL (Visual Language) model; based on a supervised fine tuning model obtained through supervised training, optimizing the visual language model by adopting a GRPO algorithm; and performing preliminary annotation on the to-be-processed video based on the optimized visual language model, and iteratively updating the visual language model by manually correcting wrong annotations. In a complex video scene, the method provided by the invention is adopted to extract the shot label, so that the extraction precision and the extraction efficiency can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a shot tag extraction method, device, and medium based on omnimodal understanding. Background Art

[0002] In the fields of content analysis and intelligent creation, the automated extraction of shot labels is a key technology for achieving structured video understanding, content retrieval, and intelligent editing. Traditional methods rely on manual annotation or rule-based classification strategies to assign labels to video clips through preset keyword libraries or visual feature matching. While these methods can handle simple scenarios, they often suffer from reduced label accuracy when faced with complex spatiotemporal semantics, multimodal interactions (such as visual-text-action collaborative analysis), or dynamic scene switching due to insufficient feature expression capabilities or a lack of contextual awareness. For example, the coupling relationship between the subject's action and the background environment in dynamic shots, and the continuity constraints of cross-frame semantics, are difficult to effectively model using single-modal analysis or static rules.

[0003] In recent years, with the development of large multimodal models (such as the Qwen-VL series), deep learning-based shot understanding technologies have gradually replaced traditional methods. By jointly modeling video spatiotemporal features and text semantics, these technologies significantly improve the semantic consistency and fine-grained expression capabilities of label generation. However, existing solutions still face three core challenges: 1. High data labeling costs: The construction of high-quality multimodal training data relies on large-scale manual labeling, and the labeling system must balance semantic breadth and granularity, resulting in long data preparation cycles and high costs. 2. Limited model generalization: Models in supervised learning frameworks tend to overfit the distribution characteristics of training data, and perform unstable when faced with long-tail labels, rare shot types, or cross-domain scenarios. 3. Bottlenecks in inference and training efficiency: The multi-model integrated information fusion and inference process involves high-dimensional tensor operations. Combined with the temporal continuity requirements of video sequences, this places higher demands on computing resources (such as video memory and computing power), limiting the feasibility of real-time applications.

[0004] To address the above issues, one can try to introduce reinforcement learning (such as the GRPO algorithm) to optimize the generation strategy, or reduce the reliance on purely manually labeled data through progressive learning. However, such methods still have shortcomings in reward function design, multi-stage iterative stability, and optimization of human-machine collaboration processes. For example, a single-dimensional reward signal makes it difficult to fully measure the semantic integrity and format standardization of the generated labels, and the cumulative error effect of the multi-stage training process may lead to fluctuations in model performance. In addition, the architectural design of core modules such as video boundary detection and multimodal feature alignment has not yet formed a unified paradigm. A systematic solution is urgently needed to ensure label accuracy while lowering the technical threshold for engineering implementation. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art, namely, insufficient multimodal feature fusion, low label generation accuracy, and poor model iteration efficiency.

[0006] To this end, the present invention provides a shot label extraction method based on omnimodal understanding.

[0007] A second aspect of the present invention provides a computer device.

[0008] A third aspect of the present invention provides a computer-readable storage medium.

[0009] The shot label extraction method based on omnimodal understanding proposed in the present invention includes: The original video is divided into multiple semantically complete segment units, and each segment is labeled with a mirror attribute label to form a labeled dataset; Based on the visual language model, supervised training is performed using a labeled dataset; the visual language model adopts the Qwen2.5-VL model; Based on the supervised fine-tuning model obtained through supervised training, the visual language model is optimized using the GRPO algorithm; The video to be processed is preliminarily labeled based on the optimized visual language model, and the visual language model is iteratively updated by manually correcting erroneous labels.

[0010] The shot tag extraction method based on omnimodal understanding according to the above technical solution of the present invention may also have the following additional technical features: In the above technical solution, the original video is divided into multiple semantically complete segment units, and each segment is labeled with a mirror attribute label to form a labeled dataset, including: Use the video shot cut detection model to predict the boundary entry and exit points of the video; Split the video into video clips of different scenes according to the entry and exit points of the video clips; Define a shot label classification system and build a shot label prompt word template library; The image attribute labels are annotated for each fragment to form an annotation dataset.

[0011] In the above technical solution, the method of using the video shot cut detection model to predict the boundary entry and exit points of the video includes: A spatiotemporal feature extraction structure is used to parse video content frame by frame and generate a spatiotemporal feature matrix; the spatiotemporal feature extraction structure includes a cascaded three-dimensional convolutional neural network and a self-attention mechanism component; A dual analysis mechanism is used to obtain a first similarity vector and a second similarity vector of adjacent video frames; wherein the first similarity vector is generated by comparing the similarity of the spatiotemporal features of the adjacent video frames, and the second similarity vector is generated by performing a statistical analysis based on the color distribution features of the adjacent video frames; Based on the multi-path output network, a confidence probability value of the current frame as a video boundary is outputted at least according to the first similarity vector and the second similarity vector.

[0012] In the above technical solution, the supervised training using the labeled dataset based on the visual language model includes: Preprocess the video clips and corresponding text prompts to extract text features and video features; Multimodal feature fusion encoding combines video features with text features to construct multimodal input features; the multimodal input features are input into the visual language model to generate label prediction results; Based on the label prediction results generated by the visual language model, the cross entropy loss function is used to update the parameters.

[0013] In the above technical solution, the cross entropy loss function is used to update the parameters, including:

[0014] in, represents the cross entropy loss function; B represents the number of samples; represents the input vector, Represents the input vector of the i-th sample, R is a real number identifier, and D represents the feature dimension; represents the output layer weight matrix; V represents the vocabulary size; T represents the transpose matrix operation; Represents the bias vector, corresponding to the bias item of each category The original score output after linear transformation is , Represents the unnormalized output of the i-th sample for the j-th category; represents the true label of the i-th sample.

[0015] In the above technical solution, the visual language model is optimized using the GRPO algorithm based on the supervised fine-tuning model obtained through supervised training, including: Load the supervised fine-tuned basic visual language model; Preprocess the video clips and corresponding text prompts to extract text features and video features; Combine video features with text features to construct multimodal input features; input the multimodal input features into the supervised fine-tuned basic visual language model to generate label prediction results; Based on the reward function set by the GRPO algorithm, the basic visual language model that completes supervised fine-tuning is further optimized through advantage estimation and strategy update in the GRPO algorithm.

[0016] In the above technical solution, the reward function set by the GRPO algorithm includes an accuracy reward and a format reward for the set label dimension; The accuracy reward includes the F1 value of comparing the content generated by the visual language model with the true value using the BERT encoder as a semantic reward; wherein the F1 value is the harmonic mean of precision and recall; The format reward includes defining a format constraint reward function and verifying the integrity of the output label field; The set label dimensions include at least one of picture description, core elements, background scenes and visual elements.

[0017] In the above technical solution, the video to be processed is preliminarily annotated based on the optimized visual language model, and the visual language model is iteratively updated by manually correcting erroneous annotations, including: Use the video shot cut detection model to predict the boundary entry and exit points of the video to be processed; Divide the video to be processed into video clips of different scenes according to the entry and exit points of the video clips; Based on the visual language model optimized by the GRPO algorithm, the video clips are coarse-grainedly labeled; Manually verify the labeling results and correct labels whose confidence level is lower than the threshold; The corrected data is added to the training set, and the model is re-trained with the newly formed training set and optimized based on the GRPO algorithm until the model converges.

[0018] The present invention also provides a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is loaded and executed by the processor, the shot label extraction method based on omnimodal understanding as described in any one of the above technical solutions is implemented.

[0019] The present invention further provides a computer-readable storage medium storing a program, which, when loaded by a processor, implements the shot label extraction method based on omnimodal understanding as described in any one of the above technical solutions.

[0020] In summary, due to the adoption of the above technical features, the beneficial effects of the present invention are: This method, by integrating a multi-stage collaborative mechanism involving video boundary detection, supervised fine-tuning, reinforcement learning, and progressive learning, addresses the technical shortcomings of traditional methods, such as poor label generation consistency in complex scenes, high manual annotation costs, and weak model generalization. Using this method for shot label extraction in complex video scenes significantly improves both accuracy and efficiency.

[0021] Additional aspects and advantages of the invention will become apparent from the description which follows, or may be learned by practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which: Figure 1 is a flowchart of a shot label extraction method based on omnimodal understanding according to an embodiment of the present invention; Figure 2 1 is a schematic diagram of video labeling in a shot label extraction method based on omnimodal understanding according to an embodiment of the present invention; Figure 3 3 is a schematic diagram of shot label extraction in a shot label extraction method based on omnimodal understanding according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0024] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0025] Refer to the following Figures 1 to 3 The following describes a shot tag extraction method, device, and medium based on omnimodal understanding according to some embodiments of the present invention.

[0026] Some embodiments of the present application provide a shot tag extraction method based on omnimodal understanding.

[0027] like Figure 1 As shown, the first embodiment of the present invention proposes a shot label extraction method based on omnimodal understanding, including the following steps S1-S4.

[0028] S1. Divide the original video into multiple semantically complete segment units, and annotate each segment with a mirror attribute label to form a labeled dataset.

[0029] Specifically, in step S1, the original video is divided into semantically complete segment units using a video shot segmentation method, and each segment is labeled with a composite mirror attribute label. The labeled dataset formed by the video segments and corresponding labels can be used for subsequent model training.

[0030] In some embodiments, step S1 includes the following steps S11 - S14 .

[0031] S11. Use the video shot switching detection model to predict the boundary entry and exit points of the video.

[0032] In a specific embodiment, the video shot cut detection model includes a triple architecture, consisting of a first subsystem, a second computing unit, and a third decision module.

[0033] The first subsystem employs a spatiotemporal feature extraction structure to parse video content frame by frame and generate a spatiotemporal feature matrix. This structure comprises a cascaded three-dimensional convolutional neural network and a self-attention mechanism. The second computational unit employs a dual analysis mechanism to obtain a first similarity vector and a second similarity vector for adjacent video frames. The first similarity vector is generated by comparing the spatiotemporal feature similarities of adjacent video frames, and the second similarity vector is generated by performing a statistical analysis based on color distribution characteristics on adjacent video frames.

[0034] Based on a multi-path output network (at least three stacked fully connected layers), the system fuses multi-dimensional features based on at least the first and second similarity vectors to output a confidence probability value for the current frame as a video boundary. Video frames that meet the confidence probability threshold are determined to be video boundaries, i.e., entry and exit points.

[0035] S12. Divide the video into video segments of different scenes according to entry and exit points of the video segments.

[0036] S13. Define a shot label classification system and construct a shot label prompt word template library. The shot label classification system includes multiple dimensions such as visual features, scene attributes, and shot motion. The shot label prompt word template library is a domain-adapted text prompt word library used to guide the model to generate standardized label descriptions.

[0037] S14, label each segment with a mirror attribute label to form a labeling dataset. The labeling result is as follows Figure 2 shown.

[0038] S2. Supervised training (SFT) is performed using a labeled dataset based on a visual language model. The Qwen2.5-VL model is used as the visual language model. Specifically, the Qwen2.5-VL model is an open-source multimodal visual language model that deeply integrates image, video, and text information to achieve cross-modal understanding, reasoning, and interaction capabilities. The details of the Qwen2.5-VL model are well known to those skilled in the art and will not be elaborated here.

[0039] In some embodiments, step S2 includes the following steps S21-SS23.

[0040] S21. Preprocess the video clips and corresponding text prompts to extract text features and video features. Specifically, use a text feature extractor (such as BERT) to extract text features. Encode the video clips through a visual encoder (such as Transformer) to generate video features. S22, multimodal feature fusion encoding, combines video features with text features to construct multimodal input features; inputs the multimodal input features into the visual language model to generate label prediction results.

[0041] S23. Based on the label prediction results generated by the visual language model, the cross entropy loss function is used to update the parameters.

[0042] Specifically, the cross entropy loss function is used to update the parameters, including:

[0043] in, represents the cross entropy loss function; B represents the number of samples; represents the input vector, Represents the input vector of the i-th sample, R is a real number identifier, and D represents the feature dimension; represents the output layer weight matrix; V represents the vocabulary size; T represents the transpose matrix operation; Represents the bias vector, corresponding to the bias item of each category The original score output after linear transformation is , Represents the unnormalized output of the i-th sample for the j-th category; represents the true label of the i-th sample.

[0044] S3. Based on the supervised fine-tuning model obtained through supervised training, the GRPO algorithm is used to optimize the visual language model.

[0045] In some embodiments, step S3 includes the following steps S31-S33.

[0046] S31. Load the basic visual language model after supervised fine-tuning, that is, the visual language model trained in step S2.

[0047] S32 , pre-processing the video clip and the corresponding text prompt to extract text features and video features; the specific pre-processing process is the same as step S21 .

[0048] S33. Combine the video features with the text features to construct a multimodal input feature; input the multimodal input feature into the basic visual language model that has completed supervised fine-tuning to generate a label prediction result; the specific construction process of the multimodal input feature is the same as step S22.

[0049] S34. Based on the reward function set by the GRPO algorithm, the basic visual language model that has completed supervised fine-tuning is further optimized through advantage estimation and strategy update in the GRPO algorithm.

[0050] It is understandable that both advantage estimation and strategy update can follow the methods in the existing GRPO algorithm. The present disclosure mainly configures the reward function in the GRPO algorithm. In some embodiments, the reward function set by the GRPO algorithm includes an accuracy reward and a format reward for the set label dimension; the accuracy reward includes using the BERT encoder to compare the F1 value of the content generated by the visual language model with the true value (GT) as a semantic reward; wherein the F1 value is an indicator to measure the semantic similarity between the generated content and the real content, usually the harmonic mean of the precision and recall rate. The format reward includes defining a format constraint reward function to verify the integrity of the output label field. The set label dimension includes at least one of the picture description, core elements, background scene and visual elements.

[0051] Specifically, the GRPO algorithm reward function in S34 uses the BERT encoder to compare the F1 value between the content generated by the visual language model and the true value (GT) in the "screen description", "core elements", "background (scene)" and "visual elements" label dimensions as the reward value, and defines a format reward function to constrain all label predictions to contain specified fields.

[0052] S4. Perform preliminary annotations on the video to be processed based on the optimized visual language model, manually correct incorrect annotations, and iteratively update the visual language model.

[0053] In some embodiments, step S4 includes the following steps S41-S45.

[0054] S41, using the video shot switching detection model to predict the boundary entry and exit points of the video to be processed; the specific process is the same as the above step S11.

[0055] S42: Segment the video to be processed into video segments of different scenes according to the entry and exit points of the video segments.

[0056] S43. Based on the visual language model optimized by the GRPO algorithm, the video clips are coarse-grainedly labeled; that is, the video clips are preliminarily labeled using the optimized model output by step S3.

[0057] S44. Manually verify the labeling results and correct the labels whose confidence is lower than the threshold.

[0058] S45. Add the corrected data to the training set, and use the newly formed training set to re-perform supervised training and GRPO algorithm-based optimization on the model. That is, use the newly formed training set to repeatedly perform supervised fine-tuning in step S2 and reinforcement learning in step S3 until the model converges. The convergence condition can be set according to the actual situation (computing power and accuracy requirements).

[0059] like Figure 3 As shown, in the above embodiment, based on the triple architecture of video boundary detection, combined with spatiotemporal feature extraction, inter-frame similarity analysis, and multi-path decision-making, the accuracy of shot segmentation is improved. Through text-video feature alignment and large language model decoding, the semantic consistency of label generation is enhanced. A multi-dimensional reward function (semantic F1 value + format constraints) is introduced to optimize model generation quality. Through the closed-loop process of "machine labeling-manual correction-model update", labor costs are reduced and model generalization capabilities are improved. The above solution forms a complete technical chain, from video segmentation to label generation to model iteration, all centered on full-modal understanding and efficient learning, solving the core problems of low label extraction accuracy and poor efficiency in complex video scenarios in existing technologies.

[0060] Other embodiments of the present invention further provide a computer device comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is loaded and executed by the processor, the shot label extraction method based on omnimodal understanding as described in any of the above embodiments is implemented.

[0061] Still other embodiments of the present invention provide a computer-readable storage medium storing a program, which, when loaded by a processor, implements the shot tag extraction method based on omnimodal understanding as described in any of the above embodiments.

[0062] In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples.

[0063] Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A shot label extraction method based on omnimodal understanding, characterized in that: include: The original video is divided into multiple semantically complete segment units, and each segment is labeled with a mirror attribute label to form a labeled dataset; Based on the visual language model, supervised training is performed using a labeled dataset; the visual language model adopts the Qwen2.5-VL model; Based on the supervised fine-tuning model obtained through supervised training, the visual language model is optimized using the GRPO algorithm; The video to be processed is preliminarily labeled based on the optimized visual language model, and the visual language model is iteratively updated by manually correcting erroneous labels.

2. The shot label extraction method based on omnimodal understanding according to claim 1, characterized in that: The original video is divided into multiple semantically complete segment units, and each segment is labeled with a mirror attribute label to form a labeled dataset, including: Use the video shot cut detection model to predict the boundary entry and exit points of the video; Split the video into video clips of different scenes according to the entry and exit points of the video clips; Define a shot label classification system and build a shot label prompt word template library; The image attribute labels are annotated for each fragment to form an annotation dataset.

3. The shot label extraction method based on omnimodal understanding according to claim 2, characterized in that: The method of using the video shot cut detection model to predict the boundary entry and exit points of the video includes: A spatiotemporal feature extraction structure is used to parse video content frame by frame and generate a spatiotemporal feature matrix; the spatiotemporal feature extraction structure includes a cascaded three-dimensional convolutional neural network and a self-attention mechanism component; A dual analysis mechanism is used to obtain a first similarity vector and a second similarity vector of adjacent video frames; wherein the first similarity vector is generated by comparing the similarity of the spatiotemporal features of the adjacent video frames, and the second similarity vector is generated by performing a statistical analysis based on the color distribution features of the adjacent video frames; Based on the multi-path output network, a confidence probability value of the current frame as a video boundary is outputted at least according to the first similarity vector and the second similarity vector.

4. The shot tag extraction method based on omnimodal understanding according to claim 1, characterized in that: The visual language model is based on supervised training using a labeled dataset, including: Preprocess the video clips and corresponding text prompts to extract text features and video features; Multimodal feature fusion encoding combines video features with text features to construct multimodal input features; the multimodal input features are input into the visual language model to generate label prediction results; Based on the label prediction results generated by the visual language model, the cross entropy loss function is used to update the parameters.

5. The shot label extraction method based on omnimodal understanding according to claim 4, characterized in that: The cross entropy loss function is used to update the parameters, including: in, represents the cross entropy loss function; B represents the number of samples; represents the input vector, Represents the input vector of the i-th sample, R is a real number identifier, and D represents the feature dimension; represents the output layer weight matrix; V represents the vocabulary size; T represents the transpose matrix operation; Represents the bias vector, corresponding to the bias item of each category The original score output after linear transformation is , Represents the unnormalized output of the i-th sample for the j-th category; represents the true label of the i-th sample.

6. The shot label extraction method based on omnimodal understanding according to claim 1, characterized in that: The supervised fine-tuning model obtained through supervised training is used to optimize the visual language model using the GRPO algorithm, including: Load the supervised fine-tuned basic visual language model; Preprocess the video clips and corresponding text prompts to extract text features and video features; Combine video features with text features to construct multimodal input features; input the multimodal input features into the supervised fine-tuned basic visual language model to generate label prediction results; Based on the reward function set by the GRPO algorithm, the basic visual language model that completes supervised fine-tuning is further optimized through advantage estimation and strategy update in the GRPO algorithm.

7. The shot label extraction method based on omnimodal understanding according to claim 6, characterized in that: The reward function set by the GRPO algorithm includes an accuracy reward and a format reward for the set label dimension; The accuracy reward includes the F1 value of comparing the content generated by the visual language model with the true value using the BERT encoder as a semantic reward; wherein the F1 value is the harmonic mean of precision and recall; The format reward includes defining a format constraint reward function and verifying the integrity of the output label field; The set label dimensions include at least one of picture description, core elements, background scenes and visual elements.

8. The shot tag extraction method based on omnimodal understanding according to claim 1, characterized in that: The process of performing preliminary annotation on the video to be processed based on the optimized visual language model, manually correcting erroneous annotations, and iteratively updating the visual language model includes: Use the video shot cut detection model to predict the boundary entry and exit points of the video to be processed; Divide the video to be processed into video clips of different scenes according to the entry and exit points of the video clips; Based on the visual language model optimized by the GRPO algorithm, the video clips are coarse-grainedly labeled; Manually verify the labeling results and correct labels whose confidence level is lower than the threshold; The corrected data is added to the training set, and the model is re-trained with the newly formed training set and optimized based on the GRPO algorithm until the model converges.

9. A computer device, characterized in that: The system comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded and executed by the processor, the method for extracting shot labels based on omnimodal understanding according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that A program is stored, and when the program is loaded by a processor, the shot label extraction method based on omnimodal understanding according to any one of claims 1 to 8 is implemented.

Images