Method, device and medium for shot label extraction based on full-modal understanding

Abstract

The application provides a shot label extraction method and device based on full-modal understanding and a medium, relates to the technical field of video processing, and the method comprises the following steps: cutting an original video into a plurality of semantic complete segment units, and labeling a mirror attribute label for each segment to form a labeled data set; based on a visual language model, the labeled data set is used for supervised training; the visual language model adopts a Qwen2.5-VL model; based on the supervised fine-tuning model obtained through the supervised training, a GRPO algorithm is used to optimize the visual language model; based on the optimized visual language model, a to-be-processed video is preliminarily labeled, and the visual language model is iteratively updated through artificial correction of incorrect labeling. In a complex video scene, the shot label extraction method can effectively improve the extraction accuracy and efficiency.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and more specifically, to a method, device, and medium for extracting lens tags based on full-modal understanding. Background Technology

[0002] In the fields of content analytics and intelligent content creation, automated extraction of shot tags is a key technology for achieving structured video understanding, content retrieval, and intelligent editing. Traditional methods rely on manual annotation or rule-based classification strategies, assigning tags to video segments through pre-defined keyword libraries or visual feature matching. While these methods can handle simple scenarios, they often suffer from decreased tag accuracy when faced with complex spatiotemporal semantics, multimodal interactions (such as visual-text-action co-analysis), or dynamic scene transitions, due to insufficient feature representation capabilities or a lack of context awareness. For example, issues such as the coupling relationship between the subject's actions and the background environment in dynamic shots, and the continuity constraints of cross-frame semantics, are difficult to effectively model using single-modal analysis or static rules.

[0003] In recent years, with the development of multimodal large models (such as the Qwen-VL series), deep learning-based shot understanding techniques have gradually replaced traditional methods. These techniques significantly improve the semantic consistency and fine-grained expressive power of tag generation by jointly modeling video spatiotemporal features and text semantics. However, existing solutions still face three core challenges:

[0004] 1. High cost of data annotation: The construction of high-quality multimodal training data relies on large-scale manual annotation, and the labeling system needs to take into account both semantic breadth and fine granularity, resulting in long data preparation cycle and high cost;

[0005] 2. Limited model generalization ability: Models under supervised learning frameworks are prone to overfitting the distribution characteristics of training data and are unstable when facing long-tailed labels, rare shot types, or cross-domain scenes.

[0006] 3. Inference and training efficiency bottlenecks: The information fusion and inference process of multi-model integration involves high-dimensional tensor operations. Combined with the temporal continuity requirements of video sequences, this places higher demands on computing resources (such as video memory and computing power), limiting the feasibility of real-time applications.

[0007] To address the aforementioned issues, reinforcement learning (such as the GRPO algorithm) can be introduced to optimize the generation strategy, or progressive learning can be used to reduce reliance on purely manually labeled data. However, such methods still have shortcomings in reward function design, multi-stage iterative stability, and human-machine collaboration process optimization. For example, a single-dimensional reward signal is insufficient to comprehensively measure the semantic integrity and format standardization of generated labels, and the cumulative error effect of multi-stage training processes may lead to fluctuations in model performance. Furthermore, the architectural design of core modules such as video boundary detection and multimodal feature alignment has not yet formed a unified paradigm, and a systematic solution is urgently needed to reduce the technical threshold for engineering implementation while ensuring label accuracy. Summary of the Invention

[0008] The present invention aims to at least solve one of the technical problems existing in the prior art, such as insufficient multimodal feature fusion, low accuracy of label generation, and poor model iteration efficiency.

[0009] Therefore, this invention provides a lens tag extraction method based on full modal understanding.

[0010] A second aspect of the present invention provides a computer device.

[0011] A third aspect of the present invention provides a computer-readable storage medium.

[0012] The lens tag extraction method based on full modal understanding proposed in this invention includes:

[0013] The original video is divided into multiple semantically complete segment units, and each segment is labeled with a mirror attribute label to form a labeled dataset;

[0014] Based on a visual language model, supervised training was performed using a labeled dataset; the visual language model adopted was the Qwen2.5-VL model.

[0015] Based on the supervised fine-tuning model obtained through supervised training, the GRPO algorithm is used to optimize the visual language model;

[0016] The optimized visual language model is used to initially annotate the video to be processed, and the visual language model is iteratively updated by manually correcting erroneous annotations.

[0017] The lens tag extraction method based on full-modal understanding according to the above-described technical solution of the present invention may also have the following additional technical features:

[0018] In the above technical solution, the step of dividing the original video into multiple semantically complete segment units and labeling each segment with mirror attribute tags to form a labeled dataset includes:

[0019] Use a video camera switching detection model to predict the in-and-out points of the video boundary;

[0020] The video is divided into video segments of different scenes based on the in and out points of the video clips;

[0021] Define a lens tag classification system and build a lens tag prompt word template library;

[0022] Each fragment is labeled with a mirror attribute label to form a labeled dataset.

[0023] In the above technical solution, the step of using a video camera switching detection model to predict the boundary in / out points of the video includes:

[0024] A spatiotemporal feature extraction structure is adopted to parse video content frame by frame and generate a spatiotemporal feature matrix; the spatiotemporal feature extraction structure includes a cascaded three-dimensional convolutional neural network and a self-attention mechanism component;

[0025] A dual analysis mechanism is used to obtain the first similarity vector and the second similarity vector of adjacent video frames; the first similarity vector is generated by comparing the spatiotemporal feature similarity of adjacent video frames, and the second similarity vector is generated by performing statistical analysis based on color distribution features on adjacent video frames.

[0026] Based on the multi-path output network, the confidence probability value of the current frame as the video boundary is output based on at least the first similarity vector and the second similarity vector.

[0027] In the above technical solution, the step of using a labeled dataset for supervised training based on a visual language model includes:

[0028] Preprocess the video clips and their corresponding text prompts to extract text and video features;

[0029] Multimodal feature fusion encoding combines video and text features to construct multimodal input features; these multimodal input features are then fed into a visual language model to generate label prediction results.

[0030] The label prediction results generated by the visual language model are updated using the cross-entropy loss function.

[0031] In the above technical solution, the step of using the cross-entropy loss function for parameter updating includes:

[0032]

[0033] in, B represents the cross-entropy loss function; B represents the number of samples. Represents the input vector. Let R represent the input vector of the i-th sample, where R is a real number and D represents the feature dimension. V represents the output layer weight matrix; T represents the vocabulary size; T represents the transpose operation. This represents the bias vector, corresponding to the bias term for each category. The original score output after linear transformation, i.e. , This represents the unnormalized output of the i-th sample for the j-th class; This represents the true label of the i-th sample.

[0034] In the above technical solution, the optimization of the visual language model using the GRPO algorithm based on the supervised fine-tuning model obtained through supervised training includes:

[0035] The supervised fine-tuning of the basic visual language model has been completed.

[0036] Preprocess the video clips and their corresponding text prompts to extract text and video features;

[0037] Video features and text features are combined to construct multimodal input features; these multimodal input features are then fed into a base visual language model that has undergone supervised fine-tuning to generate label prediction results.

[0038] Based on the reward function set by the GRPO algorithm, the basic visual language model that has completed supervised fine-tuning is further optimized through advantage estimation and policy update in the GRPO algorithm.

[0039] In the above technical solution, the reward function set by the GRPO algorithm includes an accuracy reward and a format reward for the set label dimension;

[0040] The accuracy reward includes the F1 score, which is calculated by comparing the content generated by the visual language model with the ground truth using the BERT encoder, as a semantic reward; where the F1 score is the harmonic mean of precision and recall.

[0041] The format reward includes defining a format constraint reward function and verifying the integrity of the output label field;

[0042] The defined label dimensions include at least one of the following: image description, core elements, background scene, and visual elements.

[0043] In the above technical solution, the initial annotation of the video to be processed based on the optimized visual language model, and the iterative update of the visual language model by manually correcting erroneous annotations, includes:

[0044] The video camera switching detection model is used to predict the in-and-out points of the video to be processed.

[0045] The video to be processed is divided into video segments of different scenes based on the in and out points of the video segments;

[0046] Based on the visual language model optimized using the GRPO algorithm, coarse-grained labeling is performed on video clips;

[0047] Manually verify the labeling results and correct labels with confidence levels below the threshold;

[0048] The corrected data is added to the training set, and the model is retrained and optimized using the newly formed training set in a supervised manner, and then optimized using the GRPO algorithm, until the model converges.

[0049] The present invention also provides a computer device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is loaded and executed by the processor, it implements the lens tag extraction method based on full modal understanding as described in any of the above technical solutions.

[0050] The present invention also provides a computer-readable storage medium storing a program that, when loaded by a processor, implements the lens tag extraction method based on full-modal understanding as described in any of the above technical solutions.

[0051] In summary, due to the adoption of the above-mentioned technical features, the beneficial effects of the present invention are:

[0052] This invention addresses the shortcomings of traditional methods in complex scenes by integrating a multi-stage collaborative mechanism of video boundary detection, supervised fine-tuning, reinforcement learning, and progressive learning. These shortcomings include poor consistency in label generation, high manual annotation costs, and weak model generalization ability. When applied to shot label extraction in complex video scenes, this method significantly improves both accuracy and efficiency.

[0053] Additional aspects and advantages of the invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0054] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0055] Figure 1 This is a flowchart of a lens tag extraction method based on full modal understanding according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of video tag annotation in a lens tag extraction method based on full modal understanding according to an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of lens tag extraction in a lens tag extraction method based on full modal understanding according to an embodiment of the present invention. Detailed Implementation

[0058] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0059] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0060] The following reference Figures 1 to 3 This describes a lens tag extraction method, apparatus, and medium based on full modal understanding, provided according to some embodiments of the present invention.

[0061] Some embodiments of this application provide a lens tag extraction method based on full modal understanding.

[0062] like Figure 1 As shown, the first embodiment of the present invention proposes a lens tag extraction method based on full modality understanding, including the following steps S1-S4.

[0063] S1. Divide the original video into multiple semantically complete segment units, and label each segment with a mirror attribute label to form a labeled dataset.

[0064] Specifically, in step S1, the original video is divided into semantically complete segment units using a video shot segmentation method, and each segment is labeled with a composite mirror attribute label. The labeled dataset formed by the video segments and their corresponding labels can be used for subsequent model training.

[0065] In some embodiments, step S1 includes the following steps S11-S14.

[0066] S11. Use a video camera switching detection model to predict the in-and-out points of the video boundary.

[0067] In one specific embodiment, the video shot switching detection model includes a triple architecture, consisting of a first subsystem, a second computing unit, and a third decision module.

[0068] The first subsystem employs a spatiotemporal feature extraction structure to parse video content frame by frame and generate a spatiotemporal feature matrix. This structure includes a cascaded three-dimensional convolutional neural network and a self-attention mechanism. The second computational unit uses a dual analysis mechanism to obtain a first similarity vector and a second similarity vector for adjacent video frames. Specifically, the first similarity vector is generated by comparing the spatiotemporal feature similarity of adjacent video frames, and the second similarity vector is generated by performing statistical analysis based on color distribution features on adjacent video frames.

[0069] Based on a multi-path output network (at least three stacked fully connected layers), the current frame is used as a video boundary by fusing multi-dimensional features based on at least the first and second similarity vectors. Video frames that meet the confidence probability threshold are determined as video boundaries, i.e., the in-and-out points of the video boundary.

[0070] S12. Divide the video into video segments of different scenes based on the in and out points of the video segments.

[0071] S13. Define a shot label classification system and construct a shot label cue word template library. The shot label classification system includes multiple dimensions such as visual features, scene attributes, and shot motion. The shot label cue word template library is a domain-adapted text cue word library used to guide the model in generating standardized label descriptions.

[0072] S14. Label each segment with mirror attribute labels to form a labeled dataset. The labeling results are as follows: Figure 2 As shown.

[0073] S2. Based on the visual language model, supervised training (SFT) is performed using a labeled dataset; the visual language model adopts the Qwen2.5-VL model. Specifically, the Qwen2.5-VL model is an open-source multimodal visual language model that achieves cross-modal understanding, reasoning, and interaction capabilities by deeply fusing image, video, and text information. The specific content of the Qwen2.5-VL model is well known to those skilled in the art and will not be elaborated here.

[0074] In some embodiments, step S2 includes steps S21-SS23.

[0075] S21. Preprocess the video clips and corresponding text prompts to extract text features and video features; specifically, use a text feature extractor (such as BERT) to extract text features; and use a visual encoder (such as Transformer) to encode the video clips to generate video features.

[0076] S22. Multimodal feature fusion encoding combines video features with text features to construct multimodal input features; the multimodal input features are then input into a visual language model to generate label prediction results.

[0077] S23. The label prediction results generated by the visual language model are updated using the cross-entropy loss function.

[0078] Specifically, the parameter update using the cross-entropy loss function includes:

[0079]

[0080] in, B represents the cross-entropy loss function; B represents the number of samples. Represents the input vector. Let R represent the input vector of the i-th sample, where R is a real number and D represents the feature dimension. V represents the output layer weight matrix; T represents the vocabulary size; T represents the transpose operation. This represents the bias vector, corresponding to the bias term for each category. The original score output after linear transformation, i.e. , This represents the unnormalized output of the i-th sample for the j-th class; This represents the true label of the i-th sample.

[0081] S3. Based on the supervised fine-tuning model obtained through supervised training, the visual language model is optimized using the GRPO algorithm.

[0082] In some embodiments, step S3 includes the following steps S31-S33.

[0083] S31. Load the completed supervised fine-tuning base visual language model, that is, the visual language model trained in step S2.

[0084] S32. Preprocess the video clips and corresponding text prompts to extract text features and video features; the specific preprocessing process is the same as step S21.

[0085] S33. Combine video features with text features to construct multimodal input features; input the multimodal input features into the basic visual language model that has completed supervised fine-tuning to generate label prediction results; the specific construction process of multimodal input features is the same as step S22.

[0086] S34. Based on the reward function set by the GRPO algorithm, the basic visual language model that has completed supervised fine-tuning is further optimized through advantage estimation and policy update in the GRPO algorithm.

[0087] It is understandable that advantage estimation and policy update can follow the methods used in the existing GRPO algorithm. This disclosure mainly focuses on configuring the reward function in the GRPO algorithm. In some embodiments, the reward function set by the GRPO algorithm includes accuracy rewards and format rewards for a set label dimension; the accuracy reward includes the F1 score of comparing the content generated by the visual language model with the ground truth (GT) using the BERT encoder as a semantic reward; wherein, the F1 score is an indicator that measures the semantic similarity between the generated content and the real content, and is usually the harmonic mean of precision and recall. The format reward includes defining a format constraint reward function to verify the integrity of the output label fields. The set label dimension includes at least one of image description, core elements, background scene, and visual elements.

[0088] Specifically, in S34, the GRPO algorithm reward function uses the BERT encoder to compare the F1 score between the content generated by the visual language model and the ground truth (GT) in the label dimensions of "image description", "core elements", "background (scene)" and "visual elements" as the reward value, and defines a format reward function to constrain all label predictions to include specified fields.

[0089] S4. Based on the optimized visual language model, perform initial annotation on the video to be processed, and iteratively update the visual language model by manually correcting erroneous annotations.

[0090] In some embodiments, step S4 includes steps S41-S45.

[0091] S41. Use the video lens switching detection model to predict the boundary in and out points of the video to be processed; the specific process is the same as step S11 above.

[0092] S42. Divide the video to be processed into video segments of different scenes according to the in and out points of the video segments.

[0093] S43. Based on the visual language model optimized using the GRPO algorithm, perform coarse-grained labeling on the video segments; that is, use the optimized model output from step S3 to perform preliminary labeling on the video segments.

[0094] S44. Manually verify the labeling results and correct labels with confidence levels below the threshold.

[0095] S45. Add the corrected data to the training set, and use the newly formed training set to retrain the model in a supervised manner and optimize it based on the GRPO algorithm. That is, use the newly formed training set to repeat the supervised fine-tuning in step S2 and the reinforcement learning in step S3 until the model converges. The convergence condition can be set according to the actual situation (computing power and accuracy requirements).

[0096] like Figure 3As shown, in the above embodiments, based on a triple architecture for video boundary detection, combined with spatiotemporal feature extraction, inter-frame similarity analysis, and multi-path decision-making, the accuracy of shot segmentation is improved; semantic consistency of tag generation is enhanced through text-video feature alignment and large language model decoding; a multi-dimensional reward function (semantic F1 score + format constraints) is introduced to optimize model generation quality; and a closed-loop process of "machine annotation - manual correction - model update" reduces manual costs and improves model generalization ability. The above solution forms a complete technology chain, from video segmentation to tag generation to model iteration, all revolving around full-modal understanding and efficient learning, solving the core problems of low accuracy and poor efficiency in tag extraction in complex video scenarios in existing technologies.

[0097] Other embodiments of the present invention also provide a computer device including a processor and a memory, wherein the memory stores a computer program that, when loaded and executed by the processor, implements the lens tag extraction method based on full modal understanding as described in any of the above embodiments.

[0098] Some embodiments of the present invention provide a computer-readable storage medium storing a program that, when loaded by a processor, implements the lens tag extraction method based on full-modal understanding as described in any of the above embodiments.

[0099] In this specification, the illustrative expressions of the terms used do not necessarily refer to the same embodiments or examples. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0100] Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this invention shall be included within the scope of protection of this invention.

Claims

1. A shot label extraction method based on full-modal understanding, characterized in that, The application relates to a method for automatically labeling video mirror attributes, and belongs to the technical field of video processing. The original video is cut into a plurality of semantic complete segment units, and a mirror attribute label is labeled for each segment to form a labeled data set; A visual language model is trained in a supervised manner based on the labeled data set; the visual language model adopts a Qwen2.5-VL model; The visual language model is optimized by using a GRPO algorithm based on a supervised fine-tuning model obtained through supervised training; The visual language model is optimized by using a GRPO algorithm based on a supervised fine-tuning model obtained through supervised training; The visual language model is optimized by using a GRPO algorithm based on a supervised fine-tuning model obtained through supervised training; The original video is cut into a plurality of semantic complete segment units, and a mirror attribute label is labeled for each segment to form a labeled data set, comprising: The boundary entry and exit points of the video are predicted by using a video shot change detection model; The video is divided into video segments of different scenes according to the entry and exit points of the video segments; A shot label classification system is defined, and a shot label prompt word template library is constructed; A mirror attribute label is labeled for each segment to form a labeled data set; The boundary entry and exit points of the video are predicted by using a video shot change detection model, comprising: A space-time feature extraction structure is adopted to analyze the video content frame by frame and generate a space-time feature matrix; the space-time feature extraction structure comprises a cascaded three-dimensional convolutional neural network and a self-attention mechanism component; A double analysis mechanism is adopted to obtain a first similarity vector and a second similarity vector of adjacent video frames; wherein the first similarity vector is generated by comparing the space-time feature similarity of adjacent video frames, and the second similarity vector is generated by statistically analyzing the color distribution features of adjacent video frames; 2.The shot label extraction method based on full-modal understanding according to claim 1, wherein, Based on a multi-path output network, the confidence probability value of the current frame as the video boundary is output at least according to the first similarity vector and the second similarity vector. The visual language model is trained in a supervised manner based on the labeled data set, comprising: The video segments and corresponding text prompts are preprocessed to extract text features and video features; Multi-modal feature fusion coding is used to combine the video features and the text features to construct multi-modal input features; the multi-modal input features are input into the visual language model to generate a label prediction result; 3. The shot label extraction method based on full-modal understanding according to claim 2, characterized in that, Based on the label prediction result generated by the visual language model, a cross-entropy loss function is used for parameter updating. wherein, represents the cross-entropy loss function; B represents the number of samples; represents the input vector, represents the input vector of the i-th sample, R is a real number identifier, and D represents the feature dimension; represents the output layer weight matrix; V represents the size of the vocabulary; and T represents the transpose matrix operation; represents the bias vector, corresponding to the bias term of each category the original score output after linear transformation, i.e. , represents the unnormalized output of the i-th sample to the j-th category; represents the true label of the i-th sample. 4.The shot label extraction method based on full-modal understanding according to claim 1, wherein, The visual language model is trained in a supervised manner based on the labeled data set, comprising: The supervised fine-tuning model is loaded into the basic visual language model; The video segments and corresponding text prompts are preprocessed to extract text features and video features; The video features and the text features are combined to construct multi-modal input features; the multi-modal input features are input into the basic visual language model to generate a label prediction result; Based on the reward function set by the GRPO algorithm, the basic visual language model subjected to the supervision fine-tuning is further optimized through advantage estimation and strategy updating in the GRPO algorithm.

5. The shot label extraction method based on full-modal understanding according to claim 4, characterized in that, The reward function set by the GRPO algorithm comprises an accuracy reward and a format reward for a set label dimension. The accuracy reward includes F1 value of content generated by the BERT encoder contrastive visual language model and the true value as a semantic reward; wherein the F1 value is the harmonic mean of precision and recall; The format reward includes defining a format constraint reward function to verify the integrity of the output label field; The set label dimension includes at least one of picture description, core element, background scene and visual element.

6. The shot label extraction method based on full-modal understanding according to claim 1, characterized in that, The preliminary labeling of the to-be-processed video based on the optimized visual language model, and the iterative updating of the visual language model through artificial correction of the wrong labeling, comprises: Using a video shot change detection model to predict the boundary in-out points of the to-be-processed video; According to the in-out points of the video clips, the to-be-processed video is segmented into video clips of different scenes; Based on the visual language model optimized by the GRPO algorithm, the video clips are labeled with coarse-grained labels; Artificially checking the labeling results and correcting labels with a confidence level below a threshold; The corrected data is added to the training set, and the newly formed training set is used to retrain the model with supervision and optimization based on the GRPO algorithm until the model converges.

7. A computer device, comprising: A processor and a memory are included, and the memory stores a computer program which, when loaded and executed by the processor, implements the full-modal understanding based shot label extraction method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, A program is stored, which, when loaded by a processor, implements the full-modal understanding based shot label extraction method according to any one of claims 1 to 6.

Images