Supervision data generation method based on video semantic structured analysis

The video score is calculated through video semantic structured analysis and scene graph analyzer, and the key video is selected for annotation, which solves the problems of scarce supervision data and high manual annotation cost in video document generation, and realizes efficient video document generation model training.

CN119048964AActive Publication Date: 2024-11-29HARBIN INSTITUTE OF TECHNOLOGY (SHENZHEN) (INSTITUTE OF SCIENCE AND TECHNOLOGY INNOVATION HARBIN INSTITUTE OF TECHNOLOGY SHENZHEN) +1

Patent Information

Application Number
CN202411268855.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-29
Estimated Expiration
2044-09-11

AI Technical Summary

Technical Problem

Due to the complexity of video content and the diversity of natural language labels in the video manuscript generation task, supervised data is scarce, traditional data augmentation methods are not effective, manual labeling is expensive, and active learning is limited in the application of open tasks, especially collective The effects of outliers are not adequately processed.

Method used

Using a supervised data generation method based on video semantic structured analysis, the learnability, diversity and uncertainty scores of videos are calculated through a large vision-language model and scene graph analyzer, and the key videos are selected for manual annotation, so as to gradually improve the videos. The effect of the document generation model.

Benefits of technology

It effectively reduces the cost of manual labeling, improves model performance, significantly improves the quality and efficiency of video document generation, reduces the impact of collective outliers, and realizes high-quality supervised data generation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048964A_ABST
    Figure CN119048964A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of computer vision and machine learning, and discloses a video semantic structured analysis-based supervision data generation method, which comprises the following steps of initializing, constructing an annotation set and an unannotated set, training a video manuscript generation model, preparing a large vision-language model, and generating a video manuscript. Determining an active learning selection stage number and a budget selected by each stage; the videos in the unlabeled set are sampled to obtain frames, image description is generated through a large vision-language model, and meanwhile a manuscript is generated through a video manuscript generation model; converting the image description and the manuscript in the text form into a scene graph by referring to SPICE; according to the method, the learnability score of the unlabeled video is calculated by performing internal and external measurement on the sample, the method follows a pool-based active learning process, and the learnability, diversity and uncertainty are effectively utilized to perform supervision data labeling on the most valuable video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and machine learning, and specifically relates to a supervised data generation method based on video semantic structured analysis. Background Art

[0002] Video manuscript generation, as a cutting-edge technology, aims to deeply analyze video content and automatically convert it into text-based manuscripts or subtitles. It shows great potential in the fields of AI-assisted content creation, news reporting automation, and media convergence. This technology is leading the innovation of information dissemination by significantly improving work efficiency. However, the core challenge of video manuscript generation is that it is highly dependent on large-scale, high-quality supervisory data, which is usually expressed as the correspondence between the video and the corresponding natural language text. Unfortunately, due to the complexity of video content and the diversity of natural language labels, directly available supervisory data is extremely scarce, especially when building specific fields such as news writing systems. This problem is more prominent.

[0003] Given the complex generation process of video content and the flexibility of natural language labels, traditional data augmentation or synthesis methods are unable to cope with this task. Currently, pseudo-labeling technology provides a possible solution, that is, predicting the content of unlabeled videos through models and using them as training labels, but its effectiveness is limited by the model's ability to understand complex video content. Over-reliance on such methods may lead to instability in the training process and a decline in model performance. Therefore, high-quality supervised data mainly relies on manual labeling, which is not only time-consuming and labor-intensive, but also costly.

[0004] In order to alleviate this dilemma, active learning strategies have gradually come into people's view. In the case of limited data annotation resources, active learning is committed to intelligently selecting the most valuable samples from unlabeled datasets for manual annotation, in order to achieve the best model performance improvement with the least annotation cost. This strategy has achieved remarkable results in closed tasks such as image classification and object detection. Its core lies in guiding sample selection by evaluating the uncertainty or diversity of samples. However, in the open and complex task of video manuscript generation, the application of active learning is still in the exploratory stage.

[0005] It is worth noting that while uncertainty and diversity have been widely studied in active learning, potential issues in the human annotation process, especially the presence of collective outliers, are often overlooked. The impact of collective outliers, a group of data points that deviate significantly from the norm or expected behavior of the dataset, is particularly significant in open-ended tasks, including video transcript generation. These outliers may arise from inconsistencies between human annotators, specifically in the form of differences in abstraction levels and description granularity. This inconsistency not only increases the complexity of the dataset, but also may pose a threat to the effectiveness of active learning.

[0006] To address this problem, the present invention proposes a supervised data generation method based on video semantic structured parsing to solve the above problem. Summary of the invention

[0007] The purpose of the present invention is to provide a supervised data generation method based on video semantic structured parsing to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a method for generating supervised data based on video semantic structured analysis, comprising the following steps: S1: Initialization, constructing annotated and unannotated sets, training a video manuscript generation model, preparing a large visual-language model, determining the number of active learning selection stages and the budget for each stage; S2: Sample the videos in the unlabeled set to obtain frames, generate image descriptions through a large visual-language model, and generate text using a video text generation model; S3: Referring to SPICE, the textual image description and manuscript are converted into scene graphs; S4: Compute the learnability score of the unlabeled videos by performing internal and external metrics on the samples.

[0009] S5: Calculate the diversity score of the unlabeled videos based on the frequency of samples in the entire dataset.

[0010] S6: Calculate the uncertainty score of the unlabeled video based on the reliability of the sample; S7: comprehensively obtain the overall score of the unlabeled videos, select and label some unlabeled videos that improve the effect of the video manuscript generation model, integrate them to obtain a new supervision data set, and update the video manuscript generation model; S8: Repeat S2 to S7 until the model achieves the desired effect, the annotation budget is exhausted, or the maximum number of active learning selection stages is reached.

[0011] Preferably, the S1 specifically comprises the steps of: S11: Select the existing supervised data to form a label set, expressed as ,Depend on Text description of video sequences and their human annotations composition; S12: Unmarked The video is represented as , ; S13: Use the supervised data L to train an initial video script and generate a model ; S14: Prepare the large-scale visual-language model BLIP2; S15: Determine the number of stages for active learning selection based on the annotation budget and actual situation and the budget chosen at each stage .

[0012] Preferably, S2 specifically includes the following steps: S21: From Sampling evenly in frame, and the Frame is called ; S22: Apply BLIP2 to all sample frames to obtain a set of ,in yes Image description on; S23: Generate models from video transcripts get The manuscripts on the list are sorted and the manuscript with the highest confidence is denoted as .

[0013] Preferably, S3 specifically includes the following steps: S31: Given an image description or manuscript , use or refer to the Stanford scene graph parser to obtain a scene graph consisting of objects, attributes and relationships or ; in, and By object ,property and relationship composition; is from The collection of all scene graphs generated, , and .

[0014] Preferably, the S4 specifically includes the following steps: S41; Pay attention to abstract inconsistencies, calculate internal metrics, and use the scene graph generated by BLIP2 frame by frame Measuring the internal prediction consistency of each video, the specific formula is as follows: ; in To calculate an object, attribute, or relationship Number of times it appears in , and Capture The number of unique objects and attributes in , intuitively speaking, Formula 1 prefers In other words, The higher, The more consistent the prediction is relative to other frames. In addition, Deliberately Ignoring Relationships , because this relationship is not very reliable in current large-scale vision-language models such as BLIP2; S42: Pay attention to the inconsistency of granularity of human annotations at different granularities, calculate external metric scores, and convert unlabeled samples In the active learning selection stage and The absolute distance between the SPICEs on , the specific formula is as follows: ; in Yes all The minimum and maximum normalization function on samples.

[0015] Preferably, the S5 specifically includes the following steps: S51: Calculate diversity score by re-weighting TF for longer descriptions. Indicated as containing The collection of scene graphs with the highest confidence predictions , the image description of BLIP2 prediction for the full dataset is represented as , ,in Defined as: ,in is a binary indicator function.

[0016] Preferably, the S6 specifically includes the following steps: S61: Calculate the uncertainty score. The specific formula is as follows: ; in, Scene graph for computing video transcripts generated by the model predicting the transcript and scene graphs for image descriptions predicted by large vision-language models Shared objects, properties, and relationships, where Used as human annotation, To measure the highest confidence.

[0017] Preferably, the S7 specifically includes the following steps: S71: Calculation Sample Overall score on , The calculation formula is as follows: ; in , , is a hyperparameter; S72: In the active learning algorithm Stage, based on overall score Will The unlabeled videos are ranked, with the lower Indicates that the amount of information is greater, depending on the budget selected at this stage choose The lowest batch of unlabeled samples ; S73: Feed samples to humans for annotation ; S74: Use Update labeled and unlabeled sets; S75: Use the newer Retrain the video transcript generation model .

[0018] Preferably, SPICE is a graph-based semantic representation method used to evaluate the quality of machine-generated image descriptions in image annotation tasks.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention (2) The present invention BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of active learning based on pool in the present invention; Figure 2 A flow chart of a scene graph parser used in the present invention; Figure 3 A schematic diagram of the active learning process performed by the present invention; Figure 4 This is a diagram showing the effect of the present invention on the MSVD public data set;

[0021] Figure 5 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0023] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0025] See also Figure 1-Figure 5 As shown, the present invention provides the following technical solutions: Implementation Method 1 A supervised data generation method based on video semantic structured analysis, characterized in that it includes the following steps: S1: Initialization, constructing annotated and unannotated sets, training a video manuscript generation model, preparing a large visual-language model, determining the number of active learning selection stages and the budget for each stage; S2: Sample the videos in the unlabeled set to obtain frames, generate image descriptions through a large visual-language model, and generate text using a video text generation model; S3: Convert textual image descriptions and manuscripts into scene graphs based on SPICE, a graph-based semantic representation method used to evaluate the quality of machine-generated image descriptions in image annotation tasks. S4: Compute the learnability score of the unlabeled videos by performing internal and external metrics on the samples.

[0026] S5: Calculate the diversity score of the unlabeled videos based on the frequency of samples in the entire dataset.

[0027] S6: Calculate the uncertainty score of the unlabeled video based on the reliability of the sample; S7: comprehensively obtain the overall score of the unlabeled videos, select and label some unlabeled videos that improve the effect of the video manuscript generation model, integrate them to obtain a new supervision data set, and update the video manuscript generation model; S8: Repeat S2 to S7 until the model achieves the desired effect, the annotation budget is exhausted, or the maximum number of active learning selection stages is reached.

[0028] In addition, in the present invention, regarding the above S1, S1 specifically includes the steps of: S11: Select the existing supervised data to form a label set, expressed as ,Depend on Text description of video sequences and their human annotations composition; S12: Unmarked The video is represented as , ; S13: Use the supervised data L to train an initial video script and generate a model ; S14: Prepare the large-scale visual-language model BLIP2; S15: Determine the number of stages for active learning selection based on the annotation budget and actual situation and the budget chosen at each stage .

[0029] In addition, in the present invention, regarding the above S2, S2 specifically includes the following steps: S21: From Sampling evenly in frame, and the Frame is called ; S22: Apply BLIP2 to all sample frames to obtain a set of ,in yes Image description on; S23: Generate models from video transcripts get The manuscripts on the list are sorted and the manuscript with the highest confidence is denoted as .

[0030] In addition, in the present invention, regarding the above S3, S3 specifically includes the following steps: S31: Given an image description or manuscript , use or refer to the Stanford scene graph parser to obtain a scene graph consisting of objects, attributes and relationships or ; in, and By object ,property and relationship composition; is from The collection of all scene graphs generated, , and .

[0031] In addition, in the present invention, regarding the above S4, S4 specifically includes the following steps: S41; Pay attention to abstract inconsistencies, calculate internal metrics, and use the scene graph generated by BLIP2 frame by frame Measuring the internal prediction consistency of each video, the specific formula is as follows: ; in To calculate an object, attribute, or relationship Number of times it appears in , and Capture The number of unique objects and attributes in , intuitively speaking, Formula 1 prefers In other words, The higher, The more consistent the prediction is relative to other frames. In addition, Deliberately Ignoring Relationships , because this relationship is not very reliable in current large-scale vision-language models such as BLIP2; S42: Pay attention to the granularity inconsistency of human annotations at different granularities and calculate external metric scores. Image description and video transcript generation model predicted by large-scale vision-language model based on images The predicted manuscripts tend to capture the most different aspects of and Specifically, the granularity inconsistency can be measured by the SPICE score between the two predictions, where a lower score indicates a higher inconsistency. Relying solely on SPICE gains may not be high, so focusing on each The time-varying variation of SPICE to simulate the expected variation in granularity consistency. In the active learning selection stage and The absolute distance between the SPICEs on , the specific formula is as follows: ; in Yes all The minimum and maximum normalized function on samples. Preferring samples with moderate variation, large and small variations may be associated with less informative and harder-to-learn samples, respectively.

[0032] In addition, in the present invention, regarding the above S5, S5 specifically includes the following steps: S51: Calculate diversity score. Diversity is preferred where the content is beyond the current video transcript generation model. Specifically, we use the concept of Term Frequency-Inverse Document Frequency (TF-IDF) to measure the importance of video content. Longer descriptions tend to co-occur more with more diverse videos, so we re-weight TF for longer descriptions. Indicated as containing The collection of scene graphs with the highest confidence predictions Similarly, the image description of BLIP2 predictions for the full dataset is represented as Mathematically, ,in Defined as: in is a binary indicator function if If effective is equal to 1. Mathematically speaking, It only focuses on the content outside the current video manuscript generation model. At the same time, it emphasizes unique samples with important content. Overall, higher express The diversity is greater.

[0033] In addition, in the present invention, regarding the above S6, S6 specifically includes the following steps: S61: Calculate the uncertainty score. The specific formula is as follows: ; Specifically, Compute the scene graph of the manuscript predicted by the video manuscript generation model and scene graphs for image descriptions predicted by large vision-language models Shared objects, properties, and relationships, where Used as human annotation to measure the highest confidence How good is the prediction of Reflects More certainty.

[0034] In addition, in the present invention, regarding the above S7, S7 specifically includes the following steps: S71: Calculation Sample Overall score on , The calculation formula is as follows: ; in , , is a hyperparameter; S72: In the active learning algorithm Stage, based on overall score Will The unlabeled videos are ranked, with the lower Indicates that the amount of information is greater, depending on the budget selected at this stage choose The lowest batch of unlabeled samples ; S73: Feed samples to humans for annotation ; S74: Use Update labeled and unlabeled sets; S75: Use the newer Retrain the video transcript generation model .

[0035] Furthermore, in order to assist personnel in understanding the present invention, the following embodiments are provided: Implementation Method 2 like Figure 1 As shown in the figure, pool-based active learning is a classic paradigm in active learning. Pool-based active learning assumes that there is a large-scale unlabeled data pool and a small-scale labeled data set. The initial model is trained on this small-scale labeled data set. This data set can usually only provide limited information, which may cause the model to be inaccurate when facing unlabeled data, but as the iteration proceeds, it will gradually become more complex and accurate. Each time, the algorithm will actively select several of the most valuable samples from the unlabeled data pool for labeling according to the active learning strategy. The selected samples will be sent to manual annotators for labeling. The labeled results will be added to the labeled data set. Combine the newly labeled data with the previously labeled data and retrain the model. This step allows the model to gradually learn the features of more samples, thereby improving the overall performance. Considering the limited labeling budget and expensive labeling process, general active learning methods do not spend all the budget at once, but rather as follows Figure 1As shown in the figure, a part of the budget is spent each time, and the selection-labeling process is continuously repeated. By gradually selecting more valuable unlabeled samples and labeling them, the effect of the model is gradually improved until the entire budget is exhausted.

[0036] like Figure 2 As shown, the workflow of the rule-based scene graph parser used is as follows: A1: Given a document to be parsed, a probabilistic context-free grammar (PCFG) dependency parser is used to convert it into a dependency parse tree.

[0037] A2: According to the parsing dependency tree, a semantic graph is obtained by simplifying quantitative modifiers and parsing pronouns.

[0038] A3: Parse the semantic graph through language rules, extract stemmed objects, relations and attributes, and form a scene graph.

[0039] like Figure 3 As shown, the specific description of the active learning process performed by the present invention is as follows: Due to the lack of access to human annotations in active learning, it is impossible to directly identify collective outliers. The present invention turns to the main attributes or inconsistencies of collective outliers as a breakthrough, and estimates the abstract and granular inconsistencies in unlabeled data. Specifically, the video frames of unlabeled videos are estimated by existing large-scale image-based visual-language models (which are more mature than video-based models at present), and the image descriptions of each frame with consistent granularity are obtained to approximate human annotations. By converting the description results into semantic scene graphs, different description results can be compared, which is important because learnability, uncertainty and diversity are all measured based on scene graphs. The present invention not only measures the consistency between the description results of each frame in the video based on the image description results of the large-scale visual-language model on the video frame, but also measures the consistency between the manuscripts generated by the video manuscript generation model and the description results of each frame externally. The internal measurement captures the abstract consistency, and the external measurement relies on the granular consistency. The two are combined to measure the learnability of the sample, or the possibility that this sample belongs to the collective outlier. The present invention also measures the diversity and uncertainty of samples according to the frequency of samples in the entire data set (using TF-IDF technology to find samples that have not been learned and are far away from other samples) and reliability (the consistency between the scene graph of the manuscript predicted by the video manuscript generation model and the scene graph of the image description predicted by the large visual-language model), and finally sorts the samples according to the overall score of weighted fusion, selects the most valuable samples according to the sorting, and annotates them, then combines them with the annotated data, and updates the model. The present invention can select reliable, diverse but learnable samples, and achieve a good trade-off between accuracy and manpower.

[0040] To test the effectiveness of the proposed method, comparative experiments were conducted on the public dataset MSVD and two models SwinBERT and CoCap, using five evaluation indicators: BLEU4, METEOR, ROUGE_L, CIDEr, and SPICE. To ensure more reliable results, all active learning methods were run three times. Figure 4 As shown in Figure 1, which reports the average performance and the difference, there are several interesting observations. First, it is worth noting that our method consistently outperforms all other baselines in all five evaluation metrics for both models, highlighting the advantages of our method. As expected, random sampling is usually the second best choice, consistent with findings from other open-ended tasks. More importantly, our method uses less than 25% of the labeled samples to surpass the results of fully training with 100% labeled data using SwinBERT as shown by the dashed horizontal line on 4 evaluation metrics. For example, when measured by BLEU4 and ROUGE-L, our method only uses about 10% and 15% of the labeled data to achieve full results. For CIDEr, our method achieves 103.65% of full results with 25% labeled samples. Even with CoCap, our method can almost always achieve more than 95% of full performance with 25% of the annotations. This observation supports our method's concerns about the negative impact of collective outliers during training. Clearly, our method effectively reduces the impact of collective outliers, leading to significant performance improvements for supervised data generation methods.

[0041] In addition, in the present invention, the supervised data generation method mainly follows the pool-based active learning paradigm, performs structured analysis on video semantics, and annotates unlabeled videos based on the results, which is used to provide rich and high-quality supervised data for video manuscript generation tasks, and promote the video manuscript generation method to achieve better results. The video semantic structured analysis involves large-scale visual-language models (image-based Large Vision-Language Models, LVLM) and scene graphs. By reasoning about the video through a large-scale visual-language model and generating a scene graph based on semantics, the video can be analyzed and discriminated from three perspectives: learnability, uncertainty, and diversity through the scene graph.

[0042] Implementation Method 3 In this specific embodiment, the method of the present invention is applied to generate a supervised dataset of sports event videos (such as football matches). First, a portion of videos with detailed game commentary manuscripts are selected from the existing sports event video library as the initial annotation set. Then, videos are sampled from the unannotated sports event video library, and the image description of each frame is generated using the BLIP2 visual-language model, and a preliminary game commentary manuscript is generated through the video manuscript generation model. By calculating internal metrics (such as consistency between frames within the video) and external metrics (such as changes in SPICE scores between image descriptions and manuscripts), as well as diversity and uncertainty scores, the video with the largest amount of information is selected for manual annotation. When annotating, focus on the description of key game events (such as goals, fouls) to improve the quality and diversity of the dataset. With the iterative training of the model, the quality of the generated manuscripts is gradually improved, and finally a high-quality sports event video supervision dataset is formed.

[0043] Implementation Method 4 The method of the present invention is also applicable to the generation of supervised data for online educational videos (such as course explanations and experimental demonstrations). First, videos with detailed teaching notes are collected from the educational platform as a labeled set. Then, the video frames are sampled from the unlabeled video library, the visual-language model is used to describe the video frames, and a preliminary teaching manuscript is generated through the video manuscript generation model. When calculating the learnability, diversity, and uncertainty scores, special attention is paid to the coherence and completeness of the teaching content. Videos that can cover a wide range of knowledge points and have a large difference between the predicted manuscript and the image description are selected for manual annotation. When annotating, pay attention to the accuracy of the knowledge points and the clarity of the teaching logic to improve the teaching value of the data set. Through multiple iterations, a supervised dataset suitable for educational video understanding is generated.

[0044] Implementation Method 5 In the field of news video reporting, the method of the present invention can be used to construct a supervised dataset containing key information such as news events, places, and people. First, obtain videos with detailed news report manuscripts from news websites or TV stations as annotation sets. Then, sample from the unreported news video library, use the visual-language model to generate image descriptions of video frames, and generate preliminary news report manuscripts through the video manuscript generation model. When calculating various indicators, special attention is paid to the accuracy and timeliness of news events. Videos that contain important news events and whose predicted manuscripts are highly consistent with the image descriptions are selected for manual annotation. When annotating, pay attention to the completeness and accuracy of news elements to improve the news value of the dataset. Through multiple iterations, a high-quality news video report supervision dataset is constructed, which helps to improve the effects of tasks such as automatic summarization and classification of news videos.

[0045] Implementation Method 6 In the field of nature documentaries, the method of the present invention can help generate a supervised dataset containing rich biodiversity and ecological information. First, videos containing detailed commentaries are selected from the existing nature documentary library as the initial annotation set. These commentaries usually describe the habits, ecological environment, etc. of animals and plants. Then, samples are taken from the unlabeled nature documentary video library, and large-scale visual-language models such as BLIP2 are used to generate image descriptions for video frames, and preliminary commentary manuscripts are generated through the video manuscript generation model. When calculating the learnability, diversity, and uncertainty scores, special attention is paid to the recognition accuracy of biological species, the completeness of the ecological environment description, and the consistency between the predicted manuscript and the image description. Videos containing rare species, unique ecological environments, and prediction results with high uncertainty are selected for manual annotation to ensure the uniqueness and richness of the dataset. Through multiple iterations, a high-quality nature documentary video supervision dataset is generated, which will help promote the progress of biodiversity research and ecological protection.

[0046] Implementation Method 7 In the advertising creative industry, the method of the present invention can be applied to construct a supervised dataset containing a variety of advertising creative elements. First, an advertising video containing detailed creative copy and visual elements is selected from the advertising database as a labeling set. Then, a sample is taken from the unpublished advertising video library, and an image description is generated for the advertising video frame using a visual-language model, and a preliminary creative copy is generated through a video manuscript generation model. When calculating various indicators, attention is paid to the novelty of the advertising creativity, the targeting of the target audience, and the correlation between the predicted copy and the image description. Those advertising videos with unique creativity, clear target audiences, and high uncertainty in prediction results are selected for manual labeling to enrich the creative diversity and practicality of the dataset. Through multiple iterations, a high-quality advertising video creative supervision dataset is generated, which provides strong support for the automatic generation and optimization of advertising creativity.

[0047] Implementation Method 8 In the field of film promotion and marketing, the method of the present invention can be used for data enhancement and annotation optimization of movie trailers. First, trailers with detailed plot overviews and character introductions are collected from the movie database as an annotation set. Then, for movies that are about to be released or are in production, samples are taken from unreleased trailer materials, the video frames are described using a visual-language model, and a preliminary trailer copy is generated through a video manuscript generation model. When calculating various indicators, special attention is paid to the coherence of the plot, the shaping of the characters, and the matching degree between the trailer copy and the visual content. Those trailer materials that can attract the audience's attention, show the core selling points of the movie, and have a high diversity of prediction results are selected for manual annotation to enhance the attractiveness and representativeness of the data set. Through multiple iterations, a high-quality movie trailer supervision data set is generated to provide more accurate and effective data support for movie promotion and marketing.

[0048] In addition, to achieve the above-mentioned purpose, an embodiment of the present invention further proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the supervision data generation method based on video semantic structured parsing of an embodiment of the present invention.

[0049] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable vehicles (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] The embodiments of the present invention are described with reference to flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0051] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0052] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0053] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "" and / or "" indicate that either one of the two can be selected, or both can be selected. Moreover, the terms "" include "" "" contain "" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "" include a..."" does not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

[0055] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A supervised data generation method based on video semantic structured analysis, characterized by: The steps include: S1: Initialization, constructing annotated and unannotated sets, training a video manuscript generation model, preparing a large visual-language model, determining the number of active learning selection stages and the budget for each stage; S2: Sample the videos in the unlabeled set to obtain frames, generate image descriptions through a large visual-language model, and generate text using a video text generation model; S3: Referring to SPICE, the textual image description and manuscript are converted into scene graphs; S4: Calculate the learnability score of unlabeled videos by performing internal and external metrics on the samples; S5: Calculate the diversity score of the unlabeled video based on the frequency of the sample in the entire dataset; S6: Calculate the uncertainty score of the unlabeled video based on the reliability of the sample; S7: comprehensively obtain the overall score of the unlabeled videos, select and label some unlabeled videos that improve the effect of the video manuscript generation model, integrate them to obtain a new supervision data set, and update the video manuscript generation model; S8: Repeat S2 to S7 until the model achieves the desired effect, the annotation budget is exhausted, or the maximum number of active learning selection stages is reached.

2. The method for generating supervised data based on video semantic structured analysis according to claim 1, characterized in that: The S1 specifically comprises the steps of: S11: Select the existing supervised data to form a label set, expressed as ,Depend on Text description of video sequences and their human annotations composition; S12: Unmarked The video is represented as , ; S13: Use the supervised data L to train an initial video script and generate a model ; S14: Prepare the large-scale visual-language model BLIP2; S15: Determine the number of stages for active learning selection based on the annotation budget and actual situation and the budget chosen at each stage .

3. The method for generating supervised data based on video semantic structured analysis according to claim 1, characterized in that: The S2 specifically includes the following steps: S21: From Sampling evenly in frame, and the Frame is called ; S22: Apply BLIP2 to all sample frames to obtain a set of ,in yes Image description on; S23: Generate models from video transcripts get The manuscripts on the list are sorted and the manuscript with the highest confidence is denoted as .

4. The method for generating supervised data based on video semantic structured analysis according to claim 1, characterized in that: The S3 specifically includes the following steps: S31: Given an image description or manuscript , use or refer to the Stanford scene graph parser to obtain a scene graph consisting of objects, attributes and relationships or ; in, and By object ,property and relationship composition; is from The collection of all scene graphs generated, , and .

5. The method for generating supervised data based on video semantic structured analysis according to claim 1, characterized in that: The S4 specifically includes the following steps: S41; Pay attention to abstract inconsistencies, calculate internal metrics, and use the scene graph generated by BLIP2 frame by frame Measuring the internal prediction consistency of each video, the specific formula is as follows: ; in To calculate an object, attribute, or relationship Number of times it appears in , and Capture The number of unique objects and attributes in ; S42: Pay attention to the inconsistency of granularity of human annotations at different granularities, calculate external metric scores, and convert unlabeled samples In the active learning selection stage and The absolute distance between the SPICEs on , the specific formula is as follows: ; in Yes all The minimum and maximum normalization function on samples.

6. The method for generating supervised data based on video semantic structured analysis according to claim 1, characterized in that: The S5 specifically includes the following steps: S51: Calculate diversity score by re-weighting TF for longer descriptions. Indicated as containing The collection of scene graphs with the highest confidence predictions , the image description of BLIP2 prediction for the full dataset is represented as , ,in Defined as: ,in is a binary indicator function.

7. The method for generating supervised data based on video semantic structured analysis according to claim 1, characterized in that: The S6 specifically includes the following steps: S61: Calculate the uncertainty score. The specific formula is as follows: ; in, Scene graph for computing video transcripts generated by the model predicting the transcript and scene graphs for image descriptions predicted by large vision-language models Shared objects, properties, and relationships, where Used as human annotation, To measure the highest confidence.

8. The method for generating supervised data based on video semantic structured analysis according to claim 1, characterized in that: The S7 specifically includes the following steps: S71: Calculation Sample Overall score on , The calculation formula is as follows: ; in , , is a hyperparameter; S72: In the active learning algorithm Stage, based on overall score Will The unlabeled videos are ranked, with the lower Indicates that the amount of information is greater, depending on the budget selected at this stage choose The lowest batch of unlabeled samples ; S73: Feed samples to humans for annotation ; S74: Use Update labeled and unlabeled sets; S75: Use the newer Retrain the video transcript generation model .

9. The method for generating supervised data based on video semantic structured analysis according to claim 1, characterized in that: SPICE is a graph-based semantic representation method used to evaluate the quality of machine-generated image descriptions in image annotation tasks.

Citation Information

Patent Citations

  • Optical character recognition method in patent text scene

    CN110674777A

  • Value chain network control tower and enterprise management platform

    CN115699050A

  • Visual references for refining information provided in response to multi-modal queries

    CN118551064A

  • METHOD AND SYSTEM FOR SEMANTIC APPEARANCE TRANSFER USING SPLICING ViT FEATURES

    WO2023126914A2

Cited By

  • Method for automatically generating monitoring video language data set with controllable subject and behavior deviation

    CN120670746A

  • Automatic generation method for language datasets of surveillance videos with controllable deviations in subject and behavior

    CN120670746B

  • Graph-based information-driven active learning method and system

    CN120877000A