A video data automatic segmentation and labeling method, system, device and medium
By combining dynamic adaptive frame extraction and multimodal large models, the high cost and low efficiency of video data segmentation and annotation in embodied intelligent robot training are solved, generating a structured dataset that conforms to the task logic and supports embodied intelligent robot training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANFU JIANGXI LAB
- Filing Date
- 2026-04-09
- Publication Date
- 2026-06-09
AI Technical Summary
In the training of embodied intelligent robots, existing technologies suffer from high costs, low efficiency, and poor consistency in video data segmentation and annotation. Traditional visual algorithms cannot understand task logic, existing patents rely on insufficient text and audio analysis, fixed frame rate processing is prone to missing key information, and multi-model integration consumes a lot of computing resources.
A dynamic adaptive frame extraction strategy combined with a multimodal large model is adopted to perform video semantic understanding based on image features. The segments are divided by semantic logical association and secondary fine annotation is performed to generate a structured dataset.
It achieves efficient, low-cost, and accurate video data segmentation and annotation, and the output data meets the requirements of embodied intelligence tasks, supports large-scale production, reduces labor costs, and improves data quality and consistency.
Smart Images
Figure CN122176713A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of training data processing technology for embodied intelligent robots, and in particular to a method, system, device, and medium for automatic segmentation and annotation of video data. Background Technology
[0002] Imitation learning training for embodied intelligent robots requires a large amount of standardized video data. This necessitates segmenting long sequences of task videos into independent video segments based on subdivided tasks and execution steps. Simultaneously, multi-dimensional annotations must be performed on each segment, including information on the scene, actions, skills, and interactive objects, to construct a dataset of instruction-action pairs suitable for robot learning. Currently, video data segmentation and annotation are primarily achieved through manual operation, traditional visual algorithms, and existing patented technologies. All these methods have limitations in meeting the training requirements of embodied intelligent robots.
[0003] Manual annotation requires a significant investment of manpower and time to complete the time segmentation and semantic annotation of long videos, which is costly and has extremely low processing efficiency, making it unsuitable for large-scale production of massive video data. At the same time, different annotators have subjective differences in their judgment of action boundaries and skill granularity, resulting in poor consistency of annotation results and the generation of noise in the data, which directly interferes with the training and convergence effect of the robot strategy model.
[0004] Traditional video segmentation algorithms rely on visual features such as image pixel differences and optical flow changes to achieve processing. They can only identify visual abrupt changes such as scene transitions and cannot understand the task logic and action semantics of video content. For continuous and complex action transition sequences in embodied intelligence scenarios, they cannot complete accurate segmentation that conforms to the task logic, and the segmentation results have a very low degree of matching with the actual needs of robot learning.
[0005] Chinese invention patent CN120340016B discloses a method and system for generating structured tags for long videos. The method includes the following steps: acquiring a long video file, processing the long video file to generate text content with timestamps; obtaining segmentation suggestions based on the obtained text content through LLM analysis; performing decision analysis on the obtained segmentation suggestions to generate final segments; generating three-level tags for the final segments, and dynamically fusing the three-level tags to generate structured tags for the long video. However, this patent only analyzes the text recognition and audio information of the video, without utilizing the core image visual features of the video, resulting in a lack of semantic understanding dimension and insufficient accuracy and rationality in video segmentation.
[0006] Invention patent CN120411861B discloses a method and system for structured decomposition and analysis of video content based on temporal segmentation. First, the video and its metadata are acquired. A first artificial intelligence model integrating spatiotemporal attention and sliding window mechanisms is used for temporal segmentation to generate time segments. Then, FFmpeg is used to cut the video into several video segments. A second artificial intelligence model integrating multimodal fusion and timestamp embedding is used to extract text descriptions for each segment, achieving semantic alignment between actions and text. Simultaneously, FFmpeg is used to extract keyframes to generate image groups, and a third artificial intelligence model combining object detection and cross-frame consistency constraints performs semantic analysis to generate action tags. Finally, the time segments, keyframes, text descriptions, and action tags are integrated to generate a complete structured output.
[0007] However, CN120411861B uses a fixed frame rate for video processing and segmentation, which cannot dynamically adjust the processing density according to changes in the action within the video. Key action segments with high information density are prone to information omission. Furthermore, this type of patent uses a complex processing flow that integrates multiple models, resulting in high computational resource consumption and limited overall adaptability and practicality.
[0008] To address the various pain points of existing technologies, there is an urgent need to develop an automatic video data segmentation and annotation technology that combines semantic understanding, dynamic adaptation, high efficiency, and low cost. Summary of the Invention
[0009] To address the aforementioned issues, this invention proposes a method, system, device, and medium for automatic video data segmentation and annotation, which can provide highly efficient, low-cost, and highly consistent annotation data support for embodied intelligent model training.
[0010] The technical solution adopted in this invention is as follows: A method for automatic segmentation and annotation of video data includes: The original long video is dynamically extracted to obtain key frames, and each key frame is described in a multi-dimensional structure based on preset prompt words and preset multimodal large model. Based on the semantic logical association aggregation of continuous frame structured description, video semantic segments are divided and the start and end timestamps of the video semantic segments are determined. The original video is physically segmented based on start and end timestamps, and the segmented segments are then annotated again. The annotation information is then integrated to output a structured dataset.
[0011] Furthermore, the method of dynamically extracting keyframes from the original long video includes: using equal-interval frame extraction as the basic strategy, setting a frame extraction benchmark based on annotation accuracy requirements and computational budget, automatically encrypting the frame extraction frequency when drastic changes in action are detected in the video, and achieving key extraction of high-information-density segments by dynamically adjusting the frame extraction density.
[0012] Furthermore, the step of performing multi-dimensional structured description of each keyframe based on preset prompt words and preset multimodal large model includes: guiding the preset multimodal large model to perform multi-dimensional structured description of each keyframe through preset prompt words. The description content includes environmental background, robot state, target object position and current action. The description results are stored in a structured format and directly parsed and output based on the image features of the keyframe.
[0013] Furthermore, the step of dividing video semantic segments based on the semantic logic association aggregation of the structured description of continuous frames and determining the start and end timestamps of the video semantic segments includes: inputting the structured text description of continuous frames into a preset multimodal large model, determining the task boundary by comparing the changes in object state and the switching of action targets in the previous and subsequent frames, aggregating semantically similar frames into semantic groups, generating a preliminary task logic description for each group, and finally determining the start and end timestamps of the corresponding semantic segments.
[0014] Furthermore, the physical segmentation of the original video based on start and end timestamps includes: performing a segmentation operation on the original video according to the start and end frame timestamps corresponding to the semantic group, separating the long video into several independent subdivided task video segments.
[0015] Furthermore, the secondary annotation of the segmented segments includes: performing frame extraction processing on the segmented subdivided task video segments again, and using refined prompt words to guide the preset multimodal large model to perform attribute annotation. The annotation information includes scene, execution action, required skills, involved items, and item interaction status.
[0016] Furthermore, the integrated annotation information output structured dataset includes: integrating annotation information to form a structured dataset, wherein the structured dataset includes video storage path, segment start and end timestamps, task logic description, action classification, and item list, and the structured dataset can be directly connected to the embodied intelligent robot training framework.
[0017] An automatic video data segmentation and annotation system, comprising: The multi-dimensional structured description module is configured to dynamically extract key frames from the original long video and perform multi-dimensional structured descriptions on each key frame based on preset prompts and preset multimodal large models. The video semantic segmentation module is configured to segment video semantic segments based on the semantic logical association aggregation of the structured description of consecutive frames, and to determine the start and end timestamps of the video semantic segments. The structured dataset output module is configured to physically segment the original video based on start and end timestamps, perform secondary annotation on the segmented segments, and integrate the annotation information to output a structured dataset.
[0018] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the automatic segmentation and annotation method for video data.
[0019] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for automatically segmenting and labeling video data.
[0020] The beneficial effects of this invention are as follows: 1. Dynamic Adaptive Frame Processing and Direct Use of Image Features, Balancing Efficiency and Information Integrity. This invention employs a dynamic adaptive frame extraction strategy. Using basic frame extraction as a benchmark, the frame extraction density is adjusted in real-time based on the magnitude of action changes. This allows for the extraction of key content from high-information-density segments. Simultaneously, guided by prompts, a multimodal large model directly completes frame-level structured descriptions based on image visual features, overcoming the shortcomings of existing patents that rely solely on text and audio processing with fixed frame rates. This approach avoids the omission of key action information, effectively balances computational resource consumption, and produces standardized and consistent descriptive information. It eliminates subjective differences from manual annotation at the source, providing complete and accurate foundational data for subsequent processing.
[0021] 2. Semantic logic-driven segmentation, aligning with the patterns of embodied intelligence tasks. This invention abandons the limitations of traditional visual mutation detection, using the semantic logic association of consecutive frames as the core to determine task boundaries. Through large-scale model analysis of changes in object states and action targets between frames, semantic grouping and segmentation are completed. The segmentation results perfectly match the execution logic of embodied intelligence tasks. Compared to existing patents that cannot understand task semantics and suffer from complex and redundant processes, the segmentation method of this invention is more suitable for robot learning needs, with precise and reasonable segmentation, significantly improving the matching degree between data and training scenarios.
[0022] 3. Hierarchical and refined annotation and standardized output enable cost reduction, efficiency improvement, and scalable application. This invention adopts a two-tiered, refined annotation architecture, which improves multi-dimensional information such as scenes, actions, skills, items, and interaction states while controlling computational costs, significantly enhancing annotation accuracy and comprehensiveness. Finally, the data is integrated to form a standardized dataset that can be directly interfaced with embodied intelligent robot training frameworks without additional format adaptation. Fully automated processing enables uninterrupted batch operations, significantly reducing labor costs and shortening data processing cycles, while ensuring consistent annotation logic across batches and stable, controllable data quality, supporting the large-scale production and rapid application of massive amounts of video data. Attached Figure Description
[0023] Figure 1 This is a flowchart of the automatic video data segmentation and annotation method of Embodiment 1 of the present invention. Detailed Implementation
[0024] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments are now described. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0025] Example 1 like Figure 1 As shown, this embodiment provides a method for automatic segmentation and annotation of video data, including: The original long video is dynamically extracted to obtain key frames, and each key frame is described in a multi-dimensional structure based on preset prompt words and preset multimodal large model. Based on the semantic logical association aggregation of continuous frame structured description, video semantic segments are divided and the start and end timestamps of the video semantic segments are determined. The original video is physically segmented based on start and end timestamps, and the segmented segments are then annotated again. The annotation information is then integrated to output a structured dataset.
[0026] It should be noted that this progressive processing flow can achieve end-to-end automated conversion of unstructured video to structured data, replacing manual and traditional algorithm processing modes, and ensuring the efficiency and standardization of data processing.
[0027] Preferably, keyframes are obtained by dynamically extracting frames from the original long video, including: using equal-interval frame extraction as the basic strategy, setting a frame extraction benchmark based on annotation accuracy requirements and computational budget, automatically encrypting the frame extraction frequency when drastic changes in action are detected in the video, and achieving key extraction of high-information-density segments by dynamically adjusting the frame extraction density.
[0028] Specifically, the original long video is first subjected to basic frame extraction at equal intervals. Then, the frame extraction baseline parameters are determined based on the actual annotation accuracy requirements and available computing resources. The amplitude of motion changes in the video is monitored in real time. When drastic motion changes are detected, the frame extraction frequency is automatically increased, and the frame extraction density is dynamically adjusted to focus on extracting key frames of high information density segments.
[0029] It should be noted that dynamic adaptive frame extraction can balance computational resource consumption with the effectiveness of key information extraction, avoid the omission of key action information caused by fixed frame extraction, and provide complete basic data for subsequent semantic analysis.
[0030] Preferably, the keyframes are described in a multi-dimensional structure based on preset prompt words and a preset multimodal large model, including: guiding the preset multimodal large model to perform multi-dimensional structured descriptions on each keyframe through preset prompt words. The description content includes environmental background, robot state, target object position and current action. The description results are stored in a structured format and directly parsed and output based on the image features of the keyframes.
[0031] Specifically, preset prompts and keyframes are synchronously input into a multimodal large model. The model is guided by the prompts to analyze the keyframe images, extract information related to the environmental background, robot limb state, target object position, and current action. The analysis results are stored in a structured form. The entire analysis process is directly based on the image features of the keyframes.
[0032] It should be noted that frame-level structured description based on prompt words can make full use of image visual features, output standardized and uniform descriptive information, eliminate subjective differences in manual annotation, and provide an accurate basis for subsequent semantic aggregation and segmentation.
[0033] Preferably, the semantic logical association aggregation of the structured description of continuous frames is used to divide the video semantic segments and determine the start and end timestamps of the video semantic segments. This includes: inputting the structured text description of continuous frames into a preset multimodal large model; determining the task boundary by comparing the changes in object state and the switching of action targets in the previous and next frames; aggregating semantically similar frames into semantic groups; generating a preliminary task logical description for each group; and finally determining the start and end timestamps of the corresponding semantic segments.
[0034] Specifically, the structured text descriptions generated from consecutive frames are input into the multimodal large model. The model compares the changes in object state and the transformation information of action targets in the previous and subsequent frames one by one, and determines the boundary nodes of the task accordingly. Frames with consistent semantic logic are classified into the same semantic group, and a corresponding task logic description is generated for each group. Then, the start and end timestamps of the semantic segments are determined according to the frame sequence corresponding to the group.
[0035] It should be noted that segmenting based on semantic logic breaks through the limitations of traditional visual mutation detection, aligns with the logical rules of embodied intelligence tasks, and greatly improves the rationality and adaptability of segment segmentation.
[0036] Preferably, the original video is physically segmented based on the start and end timestamps, including: performing a segmentation operation on the original video according to the start and end frame timestamps corresponding to the semantic group, and separating the long video into several independent subdivided task video segments.
[0037] Specifically, the start and end frame timestamp parameters corresponding to the semantic group are retrieved, and the original high-definition video is precisely segmented according to these timestamp parameters, breaking the complete long video into multiple independent sub-task video segments.
[0038] It should be noted that precise physical segmentation based on semantic timestamps can yield video segments that conform to the task logic, avoiding the generation of invalid segments and facilitating subsequent targeted annotation and robot model training.
[0039] Preferably, the segmented segments are further annotated, including: performing frame extraction on the segmented sub-task video segments again, and using refined prompts to guide the preset multimodal large model to perform attribute annotation. The annotation information includes scene, execution action, required skills, involved items, and item interaction status.
[0040] Specifically, the segmented video segments are first subjected to secondary frame extraction. Then, the refined prompts and the frame extraction results are input into the multimodal large model to guide the model to perform attribute annotation on the segments and fully extract relevant information such as scene type, execution action, required skills, interactive items and item interaction status.
[0041] It should be noted that secondary refined annotation can further improve the accuracy and comprehensiveness of the annotation information, meeting the high-dimensional and high-precision requirements of embodied intelligent model training for annotation data.
[0042] Preferably, the structured dataset is output by integrating the annotation information, including: integrating the annotation information to form a structured dataset, which includes video storage path, segment start and end timestamps, task logic description, action classification, and item list. The structured dataset can be directly connected to the embodied intelligent robot training framework.
[0043] Specifically, video segmentation information and secondary fine annotation information are integrated and summarized to construct a dataset that includes video storage path, segment start and end timestamps, task logic description, action classification, and item list. The dataset is then adjusted to a standard format so that it can be directly adapted to the embodied intelligent robot training framework.
[0044] It should be noted that standardized structured datasets can be directly integrated into the training process, shortening the cycle from data processing to model training, reducing data adaptation costs, and improving data utilization efficiency.
[0045] Accordingly, this embodiment also provides an automatic video data segmentation and annotation system, including a multi-dimensional structured description module, a video semantic segmentation module, and a structured dataset output module. The multi-dimensional structured description module is configured to dynamically extract keyframes from the original long video, perform multi-dimensional structured descriptions on each keyframe based on preset prompts and a preset multimodal large model. The video semantic segmentation module is configured to aggregate and segment video semantic segments based on the semantic logical associations of the structured descriptions of consecutive frames, and determine the start and end timestamps of the video semantic segments. The structured dataset output module is configured to physically segment the original video based on the start and end timestamps, perform secondary annotation on the segmented segments, and integrate the annotation information to output a structured dataset.
[0046] Specifically, the video data automatic segmentation and annotation system completes the extraction of original video frames and description of key frames through a multi-dimensional structured description module, completes the segmentation of semantic segments and determination of timestamps through a video semantic segmentation module, and completes video segmentation, secondary annotation and dataset output through a structured dataset output module. All modules work together to execute the entire process of automatic video segmentation and annotation.
[0047] It should be noted that modular system design enables independent operation and collaborative cooperation of various functions, improves the stability and scalability of system operation, and facilitates iterative optimization of functions according to scenario requirements.
[0048] Example 2 This embodiment provides a method for automatic segmentation and annotation of video data, including: 1. Dynamic Adaptive Frame Extraction Mechanism: Based on annotation accuracy requirements and computational budget, the original long video is subjected to equally spaced frame extraction. Adaptive frame extraction strategies are supported, automatically increasing the frame extraction frequency when drastic changes in motion are detected. By extracting frames and describing keyframes, the multimodal large model only needs to support the image modality, and high-information-density segments can be extracted with a variable frame extraction frequency.
[0049] 2. Frame-level Structured Description Based on Cue Word Engineering: This method guides a multimodal large-scale model to perform multi-dimensional descriptions of each extracted frame image using cue words. Outputs include, but are not limited to: environmental background, robot state, target object position, and current suspected action. Description results are stored in structured formats such as JSON. This method leverages the multimodal capabilities of a large-scale model to generate relevant text based on image content, directly utilizing image features.
[0050] 3. Semantic-driven video segmentation engine: The structured text descriptions of consecutive frames are input into a large model (multimodal or plain text model). The model automatically identifies task boundaries by comparing the logical correlation between the descriptions of consecutive frames (such as changes in object state or switching of action targets), aggregates frames with similar semantics into groups, and generates a preliminary logical description for each group (such as "moved above the object").
[0051] 4. Physical segmentation and fragment generation: Based on the start and end frame timestamps of semantic grouping, the original high-definition video is physically segmented to generate multiple task fragments.
[0052] 5. Secondary Refined Attribute Annotation: For each segmented video clip, frames are extracted again, and more detailed prompts guide the large model to output refined annotations. Specifically, the annotation information includes: scene, action, required skills, involved objects and their interaction states. This step ensures that the annotation results are highly concise and accurate at the clip level.
[0053] 6. Structured Dataset Output: Integrates all information and outputs a standard dataset format containing fields such as video path, start and end timestamps, task description, action classification, and item list, which can be directly connected to the robot training framework.
[0054] Preferably, the multimodal large model can be selected from models such as Qwen VL and GPT-4o, depending on the requirements.
[0055] Preferably, the frame extraction logic can incorporate optical flow or key point detection as a pre-screening process to further optimize frame extraction efficiency.
[0056] Preferably, the annotation results can be stored in MongoDB or directly integrated into the HDF5 dataset file.
[0057] Specifically, the automatic video data segmentation and annotation method of this embodiment can be implemented by the following steps: 1. Obtain a 30-second video to be processed (such as a robot making coffee), and use a frame extraction module (such as uniform frame extraction) to extract 60 images from the video.
[0058] 2. Input each image and prompt word into the large model in sequence, and the large model will output the image description.
[0059] 3. Using the 60 image descriptions obtained in the previous step, concatenate these descriptions to construct prompt words, input them into the large model, and the large model outputs grouped messages.
[0060] 4. Based on the output of the previous large model, the original video is segmented according to the start and end positions (start_idx, end_idx) to obtain several segments of related tasks (such as the robot picking up a cup, operating a coffee machine, etc.).
[0061] 5. Process each segment obtained in the previous step by performing frame extraction. For example, if the segment of the robot holding up the cup lasts for 10 seconds, use uniform frame extraction to extract 20 images. Input these images and prompts into the large model, and the large model will output detailed annotations.
[0062] 6. This yields the video segmentation and a description of each segment. A program is then written to convert the structured data output by the large model into the format required by the algorithm, making it ready for use.
[0063] Example 3 This embodiment is based on embodiment 1: This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the automatic video data segmentation and annotation method of Embodiment 1. The computer program can be in the form of source code, object code, executable file, or some intermediate form.
[0064] Specifically, the computer device's memory stores the computer program corresponding to the method, and the processor calls and executes the program to sequentially complete the entire process of video frame extraction, structured description, semantic segmentation, video segmentation, secondary annotation, and dataset output.
[0065] It should be noted that relying on computer equipment to implement the process can ensure the automation and uninterrupted operation of the processing, meeting the needs of batch processing of large-scale video data.
[0066] Example 4 This embodiment is based on embodiment 1: This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the automatic video data segmentation and annotation method of Embodiment 1. The computer program can be in the form of source code, object code, executable file, or some intermediate form. The storage medium includes any entity or device capable of carrying computer program code, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0067] Specifically, a computer-readable storage medium stores a computer program for the corresponding method. The processor reads and executes the program in the storage medium to complete the entire process of automatic video data segmentation and annotation.
[0068] It should be noted that computer-readable storage media enable convenient storage and portability of methods and programs, facilitating deployment and application on different computer devices and broadening the applicability of the methods.
[0069] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.
[0070] It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
Claims
1. A method for automatic segmentation and annotation of video data, used for training data processing of embodied intelligent robots, characterized in that, The method includes: The original long video is dynamically extracted to obtain key frames, and each key frame is described in a multi-dimensional structure based on preset prompt words and preset multimodal large model. Based on the semantic logical association aggregation of continuous frame structured description, video semantic segments are divided and the start and end timestamps of the video semantic segments are determined. The original video is physically segmented based on start and end timestamps, and the segmented segments are then annotated again. The annotation information is then integrated to output a structured dataset.
2. The automatic video data segmentation and annotation method according to claim 1, characterized in that, The process of dynamically extracting keyframes from the original long video includes: using equal-interval frame extraction as the basic strategy, setting a frame extraction benchmark based on annotation accuracy requirements and computational budget, automatically encrypting the frame extraction frequency when drastic changes in action are detected in the video, and achieving key extraction of high-information-density segments by dynamically adjusting the frame extraction density.
3. The automatic video data segmentation and annotation method according to claim 1, characterized in that, The process of performing multi-dimensional structured descriptions of each keyframe based on preset prompt words and a preset multimodal large model includes: guiding the preset multimodal large model to perform multi-dimensional structured descriptions of each keyframe through preset prompt words. The description content includes environmental background, robot state, target object position, and current action. The description results are stored in a structured format and directly parsed and output based on the image features of the keyframes.
4. The automatic video data segmentation and annotation method according to claim 1, characterized in that, The process of semantic logic association aggregation and segmentation of video semantic segments based on the structured description of continuous frames, and determining the start and end timestamps of the video semantic segments, includes: inputting the structured text description of continuous frames into a preset multimodal large model, determining the task boundary by comparing the changes in object state and the switching of action targets in the previous and subsequent frames, aggregating semantically similar frames into semantic groups, generating a preliminary task logic description for each group, and finally determining the start and end timestamps of the corresponding semantic segments.
5. The automatic video data segmentation and annotation method according to claim 1, characterized in that, The physical segmentation of the original video based on start and end timestamps includes: performing a segmentation operation on the original video according to the start and end frame timestamps corresponding to the semantic groups, separating the long video into several independent sub-task video segments.
6. The automatic video data segmentation and annotation method according to claim 1, characterized in that, The secondary annotation of the segmented segments includes: performing frame extraction processing on the segmented sub-task video segments again, and using refined prompt words to guide the preset multimodal large model to perform attribute annotation. The annotation information includes scene, execution action, required skills, involved items, and item interaction status.
7. The automatic video data segmentation and annotation method according to claim 1, characterized in that, The integrated annotation information output structured dataset includes: integrating annotation information to form a structured dataset, the structured dataset including video storage path, segment start and end timestamps, task logic description, action classification and item list, the structured dataset can be directly connected to the embodied intelligent robot training framework.
8. A video data automatic segmentation and annotation system for processing training data of embodied intelligent robots, characterized in that, The system includes: The multi-dimensional structured description module is configured to dynamically extract key frames from the original long video and perform multi-dimensional structured descriptions on each key frame based on preset prompts and preset multimodal large models. The video semantic segmentation module is configured to segment video semantic segments based on the semantic logical association aggregation of the structured description of consecutive frames, and to determine the start and end timestamps of the video semantic segments. The structured dataset output module is configured to physically segment the original video based on start and end timestamps, perform secondary annotation on the segmented segments, and integrate the annotation information to output a structured dataset.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the automatic video data segmentation and annotation method according to any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the automatic video data segmentation and annotation method according to any one of claims 1-7.