Automatic labeling method and device based on multi-modal large model

Abstract

The application discloses a kind of automatic labeling method and device based on multimodal large model, wherein the method comprises: inputting to be labeled video into trained multimodal large model, output multiple unstructured descriptive results;The multimodal large model is trained in advance by historical video and unstructured descriptive information;Confidence evaluation is carried out on each unstructured descriptive result, and confidence level is output;Confidence level includes first level and other levels;The confidence of first level is higher than the confidence of other levels;The application scenario of to-be-labeled video is identified using multiple unstructured descriptive results, and the template file of application scenario is generated;The template file is used to generate structured labeling file using unstructured descriptive result;Unstructured descriptive result with confidence level as first level is filled into the template file of application scenario, and the labeling file of to-be-labeled video is output.The application can improve labeling accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an automatic annotation method and apparatus based on a multimodal large model. Background Technology

[0002] With the rapid development of artificial intelligence, computer vision, and multimedia big data technologies, video data is experiencing explosive growth. Video is essentially high-dimensional, continuous, and semantically sparse unstructured data, making it difficult for artificial intelligence models to directly understand and utilize. Therefore, video annotation has become an indispensable foundational technology in many fields, including intelligent video analysis, content retrieval, autonomous driving, security monitoring, medical imaging, and industrial quality inspection.

[0003] Early video annotation primarily relied on manual work, with annotators adding category labels, bounding boxes, and action descriptions frame by frame or segment by segment. While this method offered high accuracy, it was costly, inefficient, and ill-suited for processing massive amounts of video data. To improve automation, existing technologies have introduced AI-based automatic / semi-automatic annotation methods. For example, by acquiring visual, textual, and audio features of video frames, multimodal Transformer architectures and time-series encoding modules are used to model event trends and achieve annotation. However, the accuracy of automatic / semi-automatic annotation methods still needs improvement, they lack versatility, and most automatic annotation processes are end-to-end outputs, making it impossible to evaluate intermediate processes and leading to errors. Summary of the Invention

[0004] This invention provides an automatic annotation method based on a multimodal large model to improve annotation accuracy. The method includes:

[0005] The video to be labeled is input into a pre-trained multimodal large model, which outputs multiple unstructured descriptive results. The multimodal large model is pre-trained using historical videos and unstructured descriptive information. The unstructured descriptive results include descriptive information about the content of the video to be labeled.

[0006] For each unstructured descriptive result, a confidence level is assessed, and the confidence level is output. The confidence level includes a first level and other levels; the confidence level of the first level is higher than that of the other levels.

[0007] The application scenarios of the video to be labeled are identified using multiple unstructured descriptive results, and template files for the application scenarios are generated; the template files are used to generate structured annotation files using the unstructured descriptive results.

[0008] The unstructured descriptive results with a confidence level of 1 are used to fill the template file of the application scenario, and the annotation file of the video to be annotated is output.

[0009] In one embodiment, the multimodal large model includes a visual encoder and a large language model;

[0010] The visual encoder is used to: divide each frame of a video into fixed-size image blocks, encode the position of the image blocks to obtain visual features; extract the audio track to generate audio features; and fuse the visual features and audio features through a cross-modal attention mechanism to obtain fused features.

[0011] Large language models are used to generate multiple unstructured descriptive results using fused features.

[0012] In one embodiment, the multiple unstructured descriptive results include a video global summary, a sequence of unit events, and scene information;

[0013] For each unstructured descriptive result, a confidence level is assessed, and the confidence level is output, including:

[0014] For video global summaries: the confidence level is determined by calculating perplexity and assessing semantic integrity.

[0015] For unit event sequences: the confidence level is determined by calculating the perplexity, evaluating the event development logic, and judging the continuity of event time;

[0016] For scene information: the confidence level is determined by evaluating the confidence of entity recognition and performing logical verification in the multimodal large model processing.

[0017] In one embodiment, the other levels include a second level and a third level; the confidence level of the second level is higher than that of the third level.

[0018] After assessing the confidence level for each unstructured descriptive result and outputting the confidence level, the following steps are also included:

[0019] Unstructured descriptive results with a confidence level of 2 are re-input into the multimodal large model for processing; unstructured descriptive results with a confidence level of 3 are sent to the manual review queue.

[0020] In one embodiment, multiple unstructured descriptive results are used to identify the application scenario of the video to be labeled, and a template file for the application scenario is generated, including:

[0021] We utilize large language models to collect video annotation example information from multiple application scenarios;

[0022] Entities are extracted and their frequency of occurrence is counted by using natural language processing and video annotation example information from multiple application scenarios.

[0023] Generate template files for application scenarios by using entities and their frequency of occurrence.

[0024] In one embodiment, after filling the template file of the application scenario with unstructured descriptive results of confidence level 1 and outputting the annotation file of the video to be annotated, the method further includes:

[0025] Logical verification and conflict resolution are performed on the annotation files of the videos to be annotated to obtain the corrected annotation files of the videos to be annotated.

[0026] This invention also provides an automatic annotation device based on a multimodal large model to improve annotation accuracy. The device includes:

[0027] The multimodal large model processing module is used to input the video to be labeled into a trained multimodal large model and output multiple unstructured descriptive results; the multimodal large model is pre-trained on historical videos and unstructured descriptive information; the unstructured descriptive results include descriptive information about the content of the video to be labeled;

[0028] The confidence level sensing module is used to evaluate the confidence level of each unstructured descriptive result and output the confidence level. The confidence level includes a first level and other levels. The confidence level of the first level is higher than that of the other levels.

[0029] A scene-adaptive template engine is used to identify the application scenario of the video to be labeled using multiple unstructured descriptive results, and generate a template file for the application scenario; the template file is used to generate a structured annotation file using the unstructured descriptive results.

[0030] The annotation file output module is used to fill the template file of the application scenario with unstructured descriptive results with a confidence level of 1, and output the annotation file of the video to be annotated.

[0031] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described automatic annotation method based on a multimodal large model.

[0032] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described automatic annotation method based on a multimodal large model.

[0033] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described automatic annotation method based on a multimodal large model.

[0034] In this embodiment of the invention, unstructured descriptive results are output from the original video. High-confidence unstructured descriptive results are retained through confidence evaluation. A template file corresponding to the application scenario is established, and the unstructured descriptive results are filled into the template file of the application scenario, outputting the annotation file for the video to be annotated. This embodiment of the invention can adapt to new scenarios and solves the problem of unstable output from pure end-to-end models, improving the accuracy and efficiency of video annotation. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0036] Figure 1 This is a flowchart illustrating the automatic annotation method based on a multimodal large model in an embodiment of the present invention;

[0037] Figure 2 This is a specific example diagram of the automatic annotation method based on a multimodal large model in an embodiment of the present invention;

[0038] Figure 3 This is another specific example of the automatic annotation method based on a multimodal large model in the embodiments of the present invention;

[0039] Figure 4 This is a schematic diagram of an automatic annotation device based on a multimodal large model in an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0042] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0043] With the explosive growth of video data, high-quality video annotation has become crucial for training AI models. Traditional video annotation mainly relies on manual labor, which is inefficient, costly, and prone to subjective inconsistencies. Existing automated solutions are mostly based on a single vision model, which has the following drawbacks:

[0044] Comprehension bias: Relying solely on visual information fails to grasp the audio context, such as dialogue and background noise, leading to inaccurate annotations.

[0045] Poor generalizability: Dedicated models require predefined categories and are difficult to generalize to new scenarios.

[0046] Efficiency bottleneck: It requires intensive calculations on each frame or a large number of segments of the video, resulting in huge resource consumption.

[0047] Figure 1 This is a flowchart illustrating the automatic annotation method based on a multimodal large model in an embodiment of the present invention, as shown below. Figure 1 The method includes:

[0048] Step 101: Input the video to be labeled into the trained multimodal large model and output multiple unstructured descriptive results; the multimodal large model is pre-trained on historical videos and unstructured descriptive information; the unstructured descriptive results include descriptive information of the content of the video to be labeled;

[0049] Step 102: Evaluate the confidence level of each unstructured descriptive result and output the confidence level; the confidence level includes the first level and other levels; the confidence level of the first level is higher than that of other levels;

[0050] Step 103: Identify the application scenario of the video to be labeled using multiple unstructured descriptive results, and generate a template file for the application scenario; the template file is used to generate a structured annotation file using the unstructured descriptive results;

[0051] Step 104: Fill the template file of the application scenario with the unstructured descriptive results of confidence level 1, and output the annotation file of the video to be annotated.

[0052] from Figure 1 As shown in the flowchart, the automatic annotation method based on a multimodal large model in this embodiment of the invention is divided into several stages: generating unstructured descriptions, evaluating confidence levels, generating scene templates, and outputting annotation results. The automatic annotation method based on a multimodal large model in this embodiment of the invention will be explained in detail below.

[0053] In practice, a multimodal large model is pre-constructed for video annotation. This multimodal large model is pre-trained on historical videos and unstructured descriptive information.

[0054] In one embodiment, the multimodal large model may include a visual encoder and a large language model;

[0055] The visual encoder is used to: divide each frame of a video into fixed-size image patches, encode the positions of the image patches to obtain visual features; extract audio tracks to generate audio features; and fuse visual and audio features through a cross-modal attention mechanism to obtain fused features.

[0056] Large language models are used to generate multiple unstructured descriptive results using fused features.

[0057] For example, a visual encoder breaks down each frame of a video into fixed-size image blocks, adds 3D positional encoding (temporal t + spatial h, w), and constructs multiple visual tokens. An audio encoder (including CNN + Transformer) outputs downsampled audio tokens. The visual and audio tokens are then aligned along the timeline and fused using a cross-modal attention mechanism. After fusion, global temporal modeling can be performed on the timeline. Self-attention is used to calculate attention weights between tokens at each time step and tokens at all other time steps, capturing long-distance dependencies across time steps and achieving complete modeling of the global dynamic changes in temporal data.

[0058] During implementation, the video to be labeled is input into a pre-trained multimodal large model, which outputs multiple unstructured descriptive results. These unstructured descriptive results include at least descriptive information about the content of the video to be labeled.

[0059] In one embodiment, the multiple unstructured descriptive results include, but are not limited to, video global summaries, unit event sequences, scene information, etc.

[0060] A video global summary can be a descriptive statement summarizing the overall video content. A unit event is an independent, minimal event, such as drinking water or standing up from a seat. A sequence of unit events consists of multiple unit events. Scene information is information closely related to the scene, such as a construction site, workers, shopping mall, shop, classroom, or students.

[0061] Furthermore, a confidence level assessment is performed on each unstructured descriptive result, and the confidence level is output, which may include:

[0062] For video global summaries: the confidence level is determined by calculating perplexity and assessing semantic integrity.

[0063] For unit event sequences: the confidence level is determined by calculating the perplexity, evaluating the event development logic, and judging the continuity of event time;

[0064] For scene information: the confidence level is determined by evaluating the confidence of entity recognition and performing logical verification in the multimodal large model processing.

[0065] For example, for video global summaries: calculate the average negative log probability of the entire sentence tokens; the lower the value, the more reliable the summary. Use artificial intelligence methods to assess whether the sentence contains the basic "subject-action-object" elements; if missing, it is downgraded. It can also determine the similarity between the video global summary and the video content, whether they are basically consistent.

[0066] For unit event sequences: calculate the perplexity for each event description separately and label the event confidence level. Event development logic, such as opening a door, entering, and sitting on a sofa, is considered a reasonable path; the sequence of unit events is determined through this normal logic. Event temporal continuity is judged using the timestamps of each unit event; check if the timestamps are consecutive and non-overlapping, and downgrade any unreasonable times.

[0067] Regarding scenario information: The multimodal large model includes a large language model. The reliability of the output entity is determined by the token-level log probability of the large language model. Logical verification, for example, is reasonable if it includes "classroom + student" and unreasonable if it includes "classroom + chef".

[0068] In the embodiments, the confidence levels may include a first level and other levels; the confidence level of the first level is higher than that of other levels, for example, it is divided into three levels: high, medium and low.

[0069] Figure 2 This is a specific example diagram of the automatic annotation method based on a multimodal large model in an embodiment of the present invention, such as... Figure 2 As shown, after evaluating the confidence level for each unstructured descriptive result and outputting the confidence level, the method may further include:

[0070] Step 201: Re-input the unstructured descriptive results with a confidence level of 2 into the multimodal large model for processing; send the unstructured descriptive results with a confidence level of 3 into the manual review queue.

[0071] In this example, to ensure the integrity of the information, the results with medium to low confidence levels are processed again. The results with medium confidence levels are reassessed and can be marked as "weak evidence". The results with low confidence levels can be ignored or handled manually.

[0072] In step 103, multiple unstructured descriptive results are used to identify application scenarios and generate template files for the application scenarios.

[0073] The system automatically identifies the application scenario of the video to be labeled based on descriptive results and generates a template file for that scenario. During implementation, it can also first search the template library for a template file with the same application scenario to avoid generating it again and directly matching the desired scenario.

[0074] Application scenarios include teaching, construction sites, traffic monitoring, medical surgery, sports training, and automated factories.

[0075] Template files can be generated in advance for each application scenario to prepare for formal use, and new application scenarios can be generated on the spot.

[0076] The template file is used to generate structured annotation files from unstructured descriptive results. It is a data template with fixed syntax and structure, such as a JSON template, which specifies fields, enumeration values ​​and data types.

[0077] Figure 3 This is another specific example diagram of the automatic annotation method based on a multimodal large model in the embodiments of the present invention, such as... Figure 3 As shown, multiple unstructured descriptive results are used to identify application scenarios and generate template files for these scenarios, including:

[0078] Step 301: Collect video annotation example information from multiple application scenarios using a large language model;

[0079] Step 302: Extract entities using video annotation example information from multiple application scenarios through natural language processing, and count the frequency of entity occurrence;

[0080] Step 303: Generate template files for application scenarios using entities and their frequency of occurrence.

[0081] During implementation, multiple descriptive information entries for the same application scenario are collected, high-frequency entities are extracted, and template files are generated with the help of a large language model and incorporated into the template library.

[0082] Finally, the unstructured descriptive results with a confidence level of 1 (i.e., high confidence) are filled into the template file of the application scenario, and the annotation file of the video to be annotated is output.

[0083] During implementation, based on a template file and a pre-established deterministic filling procedure, intermediate interpretable unstructured descriptive results are accurately mapped and filled to generate a final structured annotation file (such as JSON / XML) that can be seamlessly read by the annotation platform. This method solves the problems of unstable output and frequent formatting errors in pure end-to-end models by isolating the model's generalization understanding ability from the generation of data structures.

[0084] In one embodiment, after filling the template file of the application scenario with unstructured descriptive results of confidence level 1 and outputting the annotation file of the video to be annotated, the method may further include:

[0085] Logical verification and conflict resolution are performed on the annotation files of the videos to be annotated to obtain the corrected annotation files of the videos to be annotated.

[0086] Logical validation includes checks on timing, semantics, and format, as well as conflict resolution for issues such as incorrect event order or potential risks. Additionally, rule bases for different application scenarios can be pre-built, and a lightweight rule engine can be used for logical validation and conflict resolution.

[0087] This invention also provides an automatic annotation device based on a multimodal large model, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the automatic annotation method based on a multimodal large model, the implementation of this device can refer to the implementation of the automatic annotation method based on a multimodal large model; repeated details will not be elaborated further.

[0088] Figure 4 This is a schematic diagram of an automatic annotation device based on a multimodal large model in an embodiment of the present invention, such as... Figure 4 As shown, the device 400 includes:

[0089] The multimodal large model processing module 401 is used to input the video to be labeled into the trained multimodal large model and output multiple unstructured descriptive results; the multimodal large model is pre-trained on historical videos and unstructured descriptive information; the unstructured descriptive results include descriptive information of the content of the video to be labeled;

[0090] The confidence perception module 402 is used to evaluate the confidence of each unstructured descriptive result and output the confidence level; the confidence level includes a first level and other levels; the confidence of the first level is higher than that of other levels;

[0091] The scene-adaptive template engine 403 is used to identify the application scene of the video to be labeled using multiple unstructured descriptive results and generate a template file for the application scene; the template file is used to generate a structured annotation file using the unstructured descriptive results.

[0092] The annotation file output module 404 is used to fill the template file of the application scenario with unstructured descriptive results with a confidence level of first level, and output the annotation file of the video to be annotated.

[0093] In one embodiment, the multimodal large model includes a visual encoder and a large language model;

[0094] The visual encoder is used to: divide each frame of a video into fixed-size image blocks, encode the position of the image blocks to obtain visual features; extract the audio track to generate audio features; and fuse the visual features and audio features through a cross-modal attention mechanism to obtain fused features.

[0095] Large language models are used to generate multiple unstructured descriptive results using fused features.

[0096] In one embodiment, the multiple unstructured descriptive results include a video global summary, a sequence of unit events, and scene information;

[0097] The confidence perception module is specifically used for:

[0098] For video global summaries: the confidence level is determined by calculating perplexity and assessing semantic integrity.

[0099] For unit event sequences: the confidence level is determined by calculating the perplexity, evaluating the event development logic, and judging the continuity of event time;

[0100] For scene information: the confidence level is determined by evaluating the confidence of entity recognition and performing logical verification in the multimodal large model processing.

[0101] In one embodiment, the other levels include a second level and a third level; the confidence level of the second level is higher than the confidence level of the third level; the device 400 further includes:

[0102] The confidence level arbitration module is used to evaluate the confidence level of each unstructured descriptive result in the confidence level perception module, output the confidence level, and then re-input the unstructured descriptive results with the second confidence level into the multimodal large model for processing; and send the unstructured descriptive results with the third confidence level to the manual review queue.

[0103] In one embodiment, the scene-adaptive template engine is specifically used for:

[0104] We utilize large language models to collect video annotation example information from multiple application scenarios;

[0105] Entities are extracted and their frequency of occurrence is counted by using natural language processing and video annotation example information from multiple application scenarios.

[0106] Generate template files for application scenarios by using entities and their frequency of occurrence.

[0107] In one embodiment, the device 400 further includes:

[0108] The verification module is used to fill the template file of the application scenario with unstructured descriptive results with high confidence level after the annotation file output module outputs the annotation file of the video to be annotated, and then perform logical verification and conflict resolution on the annotation file of the video to be annotated to obtain the corrected annotation file of the video to be annotated.

[0109] In this embodiment, a confidence perception module and a scene adaptive template engine are plugged into the backend of the multimodal large model processing module. This fully combines the powerful generalization and understanding capabilities of the large model with the rigor and determinism of specific algorithms, transforming the model's "fuzzy understanding" into "precise output" that can be directly used downstream. This completely solves the problem of unstable output of pre-annotation results while significantly reducing the cost of manual annotation.

[0110] Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention, such as... Figure 5 As shown, this embodiment of the invention also provides a computer device 500, including a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the above-mentioned automatic annotation method based on multimodal large models.

[0111] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described automatic annotation method based on a multimodal large model.

[0112] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described automatic annotation method based on a multimodal large model.

[0113] The embodiments of the present invention have the following technical advantages:

[0114] High accuracy: It integrates multimodal information and combines confidence assessment and scene adaptation to improve annotation accuracy.

[0115] High efficiency: The active learning mechanism reduces manual intervention by more than 70%, and the overall annotation speed is increased several times.

[0116] Low cost: Reduces reliance on a large number of professional annotation personnel, thus lowering project costs.

[0117] High generalization ability: The powerful generalization ability of the multimodal large model enables it to handle various video types such as news, sports, and variety shows without having to retrain the model for each domain.

[0118] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0120] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0122] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An automatic annotation method based on multimodal large models, characterized in that, include: The video to be labeled is input into a pre-trained multimodal large model, which outputs multiple unstructured descriptive results; The multimodal large model is pre-trained on historical videos and unstructured descriptive information; the unstructured descriptive results include descriptive information about the content of the video to be labeled. For each unstructured descriptive result, assess its confidence level and output the confidence level. Confidence levels include a first level and other levels; the confidence level of the first level is higher than that of the other levels. The application scenarios of the video to be labeled are identified using multiple unstructured descriptive results, and template files for the application scenarios are generated; the template files are used to generate structured annotation files using the unstructured descriptive results. The unstructured descriptive results with a confidence level of 1 are used to fill the template file of the application scenario, and the annotation file of the video to be annotated is output. The multiple unstructured descriptive results include video global summary, unit event sequence, and scene information; For each unstructured descriptive result, a confidence level is assessed, and the confidence level is output, including: For video global summaries: the confidence level is determined by calculating perplexity and assessing semantic integrity. For unit event sequences: the confidence level is determined by calculating the perplexity, evaluating the event development logic, and judging the continuity of event time; For scene information: the confidence level is determined by evaluating the confidence of entity recognition and performing logical verification in the multimodal large model processing.

2. The method as described in claim 1, characterized in that, The multimodal large model includes a visual encoder and a large language model; Visual encoders are used to: divide each frame of a video into fixed-size image blocks, encode the positions of the image blocks to obtain visual features; and extract audio tracks to generate audio features. Visual and audio features are fused using a cross-modal attention mechanism to obtain fused features; Large language models are used to generate multiple unstructured descriptive results using fused features.

3. The method as described in claim 1, characterized in that, The other levels include the second level and the third level; the confidence level of the second level is higher than that of the third level. After assessing the confidence level for each unstructured descriptive result and outputting the confidence level, the following steps are also included: Unstructured descriptive results with a confidence level of 2 are re-input into the multimodal large model for processing; Unstructured descriptive results with a confidence level of 3 are sent to the manual review queue.

4. The method as described in claim 1, characterized in that, The application scenarios of the videos to be labeled are identified using multiple unstructured descriptive results, and template files for these application scenarios are generated, including: We utilize large language models to collect video annotation example information from multiple application scenarios; Entities are extracted and their frequency of occurrence is counted by using natural language processing and video annotation example information from multiple application scenarios. Generate template files for application scenarios by using entities and their frequency of occurrence.

5. The method as described in claim 1, characterized in that, After populating the template file of the application scenario with unstructured descriptive results of confidence level 1, and outputting the annotation file of the video to be annotated, it also includes: Logical verification and conflict resolution are performed on the annotation files of the videos to be annotated to obtain the corrected annotation files of the videos to be annotated.

6. An automatic annotation device based on a multimodal large model, characterized in that, include: The multimodal large model processing module is used to input the video to be labeled into the trained multimodal large model and output multiple unstructured descriptive results; The multimodal large model is pre-trained on historical videos and unstructured descriptive information; the unstructured descriptive results include descriptive information about the content of the video to be labeled. The confidence-aware module is used to evaluate the confidence level of each unstructured descriptive result and output the confidence level. Confidence levels include a first level and other levels; the confidence level of the first level is higher than that of the other levels. A scene-adaptive template engine is used to identify the application scenario of the video to be labeled using multiple unstructured descriptive results, and generate a template file for the application scenario; the template file is used to generate a structured annotation file using the unstructured descriptive results. The annotation file output module is used to populate the template file of the application scenario with unstructured descriptive results of confidence level 1, and output the annotation file of the video to be annotated; The multiple unstructured descriptive results include video global summary, unit event sequence, and scene information; The confidence perception module is specifically used for: For video global summaries: the confidence level is determined by calculating perplexity and assessing semantic integrity. For unit event sequences: the confidence level is determined by calculating the perplexity, evaluating the event development logic, and judging the continuity of event time; For scene information: the confidence level is determined by evaluating the confidence of entity recognition and performing logical verification in the multimodal large model processing.

7. The apparatus as claimed in claim 6, characterized in that, The multimodal large model includes a visual encoder and a large language model; Visual encoders are used to: divide each frame of a video into fixed-size image blocks, encode the positions of the image blocks to obtain visual features; and extract audio tracks to generate audio features. Visual and audio features are fused using a cross-modal attention mechanism to obtain fused features; Large language models are used to generate multiple unstructured descriptive results using fused features.

8. The apparatus as claimed in claim 6, characterized in that, The other levels include the second level and the third level; the confidence level of the second level is higher than that of the third level. Also includes: The confidence level arbitration module is used to evaluate the confidence level of each unstructured descriptive result in the confidence level perception module, output the confidence level, and then re-input the unstructured descriptive results with the second confidence level into the multimodal large model for processing; and send the unstructured descriptive results with the third confidence level to the manual review queue.

9. The apparatus as claimed in claim 6, characterized in that, The scene-adaptive template engine is specifically used for: We utilize large language models to collect video annotation example information from multiple application scenarios; Entities are extracted and their frequency of occurrence is counted by using natural language processing and video annotation example information from multiple application scenarios. Generate template files for application scenarios by using entities and their frequency of occurrence.

10. The apparatus as claimed in claim 6, characterized in that, Also includes: The verification module is used to fill the template file of the application scenario with unstructured descriptive results of confidence level 1 after the annotation file output module outputs the annotation file of the video to be annotated, and then perform logical verification and conflict resolution on the annotation file of the video to be annotated to obtain the corrected annotation file of the video to be annotated.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.

Images