Embodied intelligence-oriented training data set automatic construction method and system

Abstract

The application belongs to the field of embodied intelligence, and discloses an embodied intelligence-oriented automatic training data set construction method and system, which comprises the following steps: taking internet human operation videos as data sources, obtaining video segments meeting quality conditions through video screening and preprocessing, and cutting the video segments into short video segments containing single atomic actions through action change detection; generating pseudo action labels through motion trajectory extraction and action backstepping, automatically generating imperative language instructions through a multi-modal large model, generating standard action labels through inverse kinematics mapping and kinematics constraint adaptation, and finally combining video frame sequences, language instructions and standard action labels into VLA triple data sets. The application solves the technical problems of high acquisition cost, low artificial labeling efficiency, small data scale and insufficient generalization in the construction of existing VLA training data sets.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligence, specifically relating to a method and system for automatically constructing training datasets for embodied intelligence. Background Technology

[0002] In applications such as home service, industrial assembly, and humanoid robots, embodied intelligent robots rely on large-scale visual-language-action (VLA) multimodal training datasets to construct action policy models, enabling autonomous execution of complex tasks. Currently, the main methods for constructing VLA training datasets include: data acquisition based on teleoperation or teaching of real robots, generation of synthetic data using simulation platforms, and manually annotated public video or image datasets. All of these methods provide data support for policy learning in embodied intelligent robots to some extent, and related public datasets have been widely used in the field of robot operation.

[0003] However, existing methods for constructing VLA training datasets have significant technical limitations, hindering the large-scale development of embodied intelligent robots. Real-machine teleoperation data acquisition is highly dependent on hardware and scene setup, involves complex acquisition processes, covers only a limited range of scenarios, and is difficult to scale up significantly. Synthetic data generated by simulation platforms exhibits a significant sim-to-real transfer gap, resulting in insufficient adaptability to real-world scenarios. Manual annotation methods face problems such as low annotation efficiency, high labor costs, and difficulty in ensuring annotation consistency. Furthermore, the accuracy of manual annotation is insufficient for the details of actions in continuous and complex operations, failing to meet the requirements of high-quality training data. Existing public datasets have limited overall size and insufficient coverage of action types, resulting in weak generalization ability for models trained on them, making it difficult to adapt to the actual needs of diverse scenarios and complex operational tasks in the real world. Summary of the Invention

[0004] This application addresses the technical problems of high acquisition costs, low efficiency of manual annotation, small data scale, and insufficient generalization in the construction of existing VLA training datasets, and provides an automated method and system for constructing training datasets for embodied intelligence.

[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, a method for automatically constructing training datasets for embodied intelligence includes: Collect videos of humans operating objects and filter them to obtain video clips; The video segment is preprocessed to obtain a preprocessed video segment. The preprocessed video segment is then segmented into short video segments by motion change detection. Each short video segment contains a single atomic motion. Extract the motion trajectory data of the short video clip, and based on the motion trajectory data, reverse the atomic action type of the single atomic action to generate a continuous action sequence, and encapsulate the continuous action sequence into a corresponding pseudo action tag; The short video clip is input into a multimodal large model to generate an instruction description corresponding to the short video clip, and the instruction description is formatted into an imperative language instruction. After mapping the continuous action sequence in the pseudo-action label to the robot action space, standard action labels are obtained through kinematic constraint adaptation and trajectory optimization. The video frame sequence of the short video clip, the imperative language instructions, and the standard action tags are combined into VLA triples, and the VLA triples are output in a standard format to obtain a triple dataset. Data augmentation and filtering are performed on the triplet dataset to obtain a dataset for training the VLA model of the embodied intelligent robot.

[0006] In some embodiments, the preprocessing of the video segment to obtain a preprocessed video segment includes: After sequentially performing jitter removal and redundant frame removal on the video segment, it is uniformly adjusted to a preset resolution and preset frame rate to obtain the preprocessed video segment.

[0007] In some implementations, the step of segmenting the preprocessed video segment into short video segments through motion change detection includes: The preprocessed video segment is subjected to frame-by-frame motion change detection based on the frame difference method to obtain motion change nodes. Using the action change node as the segmentation boundary, the preprocessed video segment is segmented into short video segments, and each short video segment contains only a single atomic action.

[0008] In some implementations, the step of extracting motion trajectory data from the short video clip, and based on the motion trajectory data, inversely deducing the atomic action type of the single atomic action to generate a continuous action sequence includes: The three-dimensional spatial coordinates of each joint of the human hand in the short video clip are extracted by detecting key points of the hand, and the three-dimensional motion trajectory of the hand is obtained. Six degrees of freedom pose estimation is performed on the object being manipulated in the short video clip to obtain the pose change data of the object being manipulated. The motion trajectory data is obtained by temporally correlating the three-dimensional motion trajectory of the hand with the pose change data of the manipulated object based on optical flow tracking technology. The atomic action type of the single atomic action is deduced from the motion trajectory data. The atomic action type includes grasping, moving, rotating, pressing, or placing. Based on the atomic action type and the motion trajectory data, the continuous action sequence is generated, which includes: six-degree-of-freedom pose, gripper opening and closing state, and motion speed.

[0009] In some implementations, after mapping the continuous action sequence in the pseudo-action labels to the robot action space, standard action labels are obtained through kinematic constraint adaptation and trajectory optimization, including: After mapping the continuous motion sequence to the robot's motion space using inverse kinematics, joint angle limit constraints are applied in conjunction with the target robot's kinematic parameters. The standard motion labels are then obtained through collision detection and avoidance and trajectory interpolation smoothing.

[0010] In some implementations, combining the video frame sequence of the short video clip, the imperative language instructions, and the standard action tags into a VLA triple includes: Frame extraction is performed on the short video segment to obtain the video frame sequence; The video frame sequence, the imperative language instructions, and the standard action tags are encapsulated into the VLA triple according to a preset data storage format.

[0011] Secondly, an automated training dataset construction system for embodied intelligence includes: The video acquisition and filtering module is used to acquire and filter videos of humans operating objects to obtain video clips. The temporal segmentation module is used to preprocess the video segment to obtain a preprocessed video segment, and to segment the preprocessed video segment into short video segments by motion change detection, wherein the short video segment contains a single atomic action; The motion inversion module is used to extract motion trajectory data of the short video clip, invert the atomic action type of the single atomic action based on the motion trajectory data, generate a continuous action sequence, and encapsulate the continuous action sequence into a corresponding pseudo action tag. The instruction generation module is used to input the short video clip into the multimodal large model, generate an instruction description corresponding to the short video clip, and format the instruction description into an imperative language instruction. The motion mapping module is used to map the continuous motion sequence in the pseudo-motion labels to the robot motion space, and then obtain standard motion labels through kinematic constraint adaptation and trajectory optimization. The triplet construction module is used to combine the video frame sequence of the short video clip, the imperative language instructions, and the standard action tags into VLA triplets, and output the VLA triplets in a standard format to obtain a triplet dataset. The dataset generation module is used to perform data augmentation and data filtering on the triplet dataset to obtain a dataset for training the embodied intelligent robot VLA model.

[0012] Thirdly, a computer device includes: a processor and a computer-readable storage medium; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the method for automatically constructing a training dataset for embodied intelligence as described above.

[0013] Fourthly, a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed as described in the method for automatically constructing a training dataset for embodied intelligence.

[0014] Fifthly, a computer program product comprising a computer program that, when executed by a processor, implements the method for automatically constructing a training dataset for embodied intelligence as described above.

[0015] Compared with the prior art, this application has the following beneficial effects: This application eliminates the cost of equipment purchase and scene construction by replacing real-device demonstrations with internet video in the step of collecting and filtering videos of humans operating objects. The two steps of extracting motion trajectory data from short video clips and reversing atomic action types to generate continuous action sequences, and inputting short video clips into a multimodal large model to generate corresponding instruction descriptions, are completed automatically, replacing manual frame-by-frame annotation and instruction writing, thus solving the problems of low efficiency and poor consistency of manual annotation. Using internet video as the data source, the data scale is not constrained by hardware conditions, breaking through the scale bottleneck of traditional acquisition methods. Data augmentation and filtering of the triplet dataset further expand the data diversity, and the constructed dataset covers diverse real-world scenarios and operation types, effectively improving the generalization ability of the VLA model trained on this dataset.

[0016] Furthermore, frame-by-frame motion change detection is performed on the preprocessed video segments using the frame difference method to obtain motion change nodes. These motion change nodes are then used as segmentation boundaries to divide the video segments into short video segments containing only a single atomic motion. By accurately locating motion boundaries using the frame difference method and using motion change nodes as the segmentation basis, the atomicity of each short video segment at the semantic level is guaranteed. This avoids label ambiguity caused by multiple actions being mixed into the same sample, thereby improving the semantic accuracy of subsequent pseudo-action label generation.

[0017] Furthermore, by detecting key points on the hand, the three-dimensional spatial coordinates of each joint of the human hand in the short video clip are extracted to obtain the three-dimensional motion trajectory of the hand, thus acquiring the spatial motion information of the operating subject. Six-degree-of-freedom pose estimation is performed on the operating object in the short video clip to obtain the pose change data of the operating object, supplementing the complete pose information of the operated object. Based on optical flow tracking technology, the three-dimensional motion trajectory of the hand and the pose change data of the operating object are temporally correlated to obtain motion trajectory data, establishing a temporal correspondence between the operating subject and the object. The combination of the three technical means enables the motion trajectory data to simultaneously cover both the operating subject and the operated object dimensions. Based on this, the atomic motion types and continuous motion sequences containing six-degree-of-freedom pose, gripper opening and closing state, and motion speed have the complete descriptive dimensions required for robot motion learning. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating an automated method for constructing training datasets for embodied intelligence, as provided in this application embodiment; Figure 2 This application provides an embodiment of an automated training dataset construction system architecture diagram for embodied intelligence. Figure 3 A computer device diagram provided for an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In one embodiment of this application, such as Figure 1 As shown, an automated method for constructing training datasets for embodied intelligence is provided, including: S1: Collect and filter videos of humans operating objects to obtain video clips; Specifically, videos featuring human manipulation of objects were collected from publicly available video platforms. The collection scope covered human operational behaviors in various scenarios, prioritizing videos with clear first-person and third-person perspectives. The collected videos were filtered according to scene type, action type, and shooting angle. Videos with multiple people performing cluttered operations, blurry images, obvious watermarks, and discontinuous actions were filtered out. Videos with a single operator, continuous actions, unobstructed views, and no watermarks were retained, resulting in video clips that met preset quality criteria, which served as the basic material for subsequent data processing.

[0022] S2, preprocess the video segment to obtain a preprocessed video segment, and divide the preprocessed video segment into short video segments by motion change detection, wherein the short video segment contains a single atomic action; Specifically, the video segments are sequentially subjected to de-shaking and redundant frame removal processes to eliminate image jitter and redundant frames caused by differences in shooting equipment. All video segments are then uniformly adjusted to a preset resolution and frame rate to ensure stable video footage and consistent timing, resulting in pre-processed video segments. In one specific embodiment, all video segments are uniformly adjusted to 1080P resolution and 30fps frame rate.

[0023] Subsequently, frame-by-frame motion change detection is performed on the preprocessed video segment using the frame difference method: adjacent frames of the preprocessed video segment are sequentially grayscaled, the pixel difference between adjacent frames is calculated to obtain a difference image, the difference image is thresholded, and the number of changed pixels is counted. When the number of changed pixels exceeds a preset threshold, the current frame is determined to be a motion change node. Using the motion change node as the segmentation boundary, the preprocessed video segment is segmented into short video segments. Each short video segment contains only a single atomic action, and the duration of each short video segment is controlled between 3 and 8 seconds. For example, the continuous operation of opening the refrigerator, taking out an egg, and placing it on the counter is segmented into three independent short video segments: opening the refrigerator, taking out an egg, and placing the egg on the counter.

[0024] S3, extract the motion trajectory data of the short video segment, reverse the atomic action type of the single atomic action based on the motion trajectory data, generate a continuous action sequence, and encapsulate the continuous action sequence into a corresponding pseudo action tag; Specifically, the three-dimensional spatial coordinates of each joint of the human hand in the short video clip are extracted by hand keypoint detection to obtain the three-dimensional motion trajectory of the hand; at the same time, the pose of the object being manipulated in the short video clip is estimated with six degrees of freedom to obtain the pose change data of the object at each time step; the correspondence between the three-dimensional motion trajectory of the hand and the pose change data of the object being manipulated in the time dimension is calculated based on optical flow tracking technology to realize the temporal correlation between the two and obtain the motion trajectory data.

[0025] Based on the spatial relationship and movement trend between the hand and the manipulated object in the motion trajectory data, the atomic action type of the single atomic action is deduced. The atomic action type includes grasping, moving, rotating, pressing, or placing. Based on the atomic action type and the motion trajectory data, the six-degree-of-freedom pose, gripper opening / closing state, and movement speed corresponding to each time step are generated to form a continuous action sequence containing the above parameters. The continuous action sequence is then encapsulated into corresponding pseudo-action tags.

[0026] S4, input the short video clip into the multimodal large model, generate an instruction description corresponding to the short video clip, and format the instruction description into an imperative language instruction; Specifically, the short video clips are input into a multimodal large model. The multimodal large model understands and analyzes the visual content of the short video clips, automatically generating instruction descriptions corresponding to the operational content of the short video clips. These instruction descriptions are then formatted into command language instructions that the robot can directly recognize and execute. For example, for a short video clip of taking an egg and placing it on a table, the generated command language instruction would be: "Take the egg out of the refrigerator and place it on the table."

[0027] S5, after mapping the continuous action sequence in the pseudo-action label to the robot action space, the standard action label is obtained through kinematic constraint adaptation and trajectory optimization. Specifically, the continuous motion sequence in the pseudo-motion label is mapped to the robot joint space or end effector space through inverse kinematics; combined with the kinematic parameters of the target robot, the mapping result is adapted by joint angle limit constraints to ensure that the joint angle at each time step is within the physical motion range of the target robot; collision detection and avoidance processing is performed on the mapped trajectory to eliminate motion segments with collision risk in the trajectory; and then trajectory interpolation and smoothing processing is performed to eliminate singular trajectories and redundant actions introduced by the irregularity of human actions, finally obtaining standard motion labels that the target robot can directly learn and execute.

[0028] S6, combine the video frame sequence of the short video clip, the imperative language instructions, and the standard action tags into VLA triples, and output the VLA triples in a standard format to obtain a triple dataset; Specifically, the short video clips are frame-extracted to obtain a video frame sequence arranged in chronological order; the video frame sequence, the imperative language instructions, and the standard action tags are encapsulated into VLA triples according to a preset data storage format; after data deduplication and quality verification of all VLA triples, the dataset is divided according to the ratio of training set, validation set, and test set, and the triple dataset is output. In one specific embodiment, the dataset is divided into training set, validation set, and test set in an 8:1:1 ratio.

[0029] S7, perform data augmentation and data filtering on the triplet dataset to obtain a dataset for training the embodied intelligent robot VLA model.

[0030] Specifically, random cropping, color jittering, and perspective synthesis are used to perform data augmentation on the triplet dataset to expand the diversity of the data in the visual dimension; the triplet dataset is filtered to remove unqualified data containing invalid actions, low-confidence labels, and noisy instructions, and the final output is a dataset that can be directly used for training the embodied intelligent robot VLA model.

[0031] In one embodiment of this application, such as Figure 2 As shown, an automated training dataset construction system for embodied intelligence is provided, including: The video acquisition and filtering module is used to acquire and filter videos of humans operating objects to obtain video clips. The temporal segmentation module is used to preprocess the video segment to obtain a preprocessed video segment, and to segment the preprocessed video segment into short video segments by motion change detection, wherein the short video segment contains a single atomic action; The motion inversion module is used to extract motion trajectory data of the short video clip, invert the atomic action type of the single atomic action based on the motion trajectory data, generate a continuous action sequence, and encapsulate the continuous action sequence into a corresponding pseudo action tag. The instruction generation module is used to input the short video clip into the multimodal large model, generate an instruction description corresponding to the short video clip, and format the instruction description into an imperative language instruction. The motion mapping module is used to map the continuous motion sequence in the pseudo-motion labels to the robot motion space, and then obtain standard motion labels through kinematic constraint adaptation and trajectory optimization. The triplet construction module is used to combine the video frame sequence of the short video clip, the imperative language instructions, and the standard action tags into VLA triplets, and output the VLA triplets in a standard format to obtain a triplet dataset. The dataset generation module is used to perform data augmentation and data filtering on the triplet dataset to obtain a dataset for training the embodied intelligent robot VLA model.

[0032] For specific limitations regarding the automated construction system for training datasets oriented towards embodied intelligence, please refer to the limitations of the automated construction method for training datasets oriented towards embodied intelligence described above. The corresponding technical effects can be obtained equivalently and will not be repeated here. Each module in the aforementioned automated construction system for training datasets oriented towards embodied intelligence can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0033] Figure 3 An internal structural diagram of a computer device is shown in one embodiment. This computer device may specifically be a terminal or a server. Figure 3As shown, the computer device includes a processor, memory, network interface, display, camera, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an automated method for constructing training datasets for embodied intelligence. The display screen can be an LCD screen or an e-ink display. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0034] As will be understood by those skilled in the art, computer equipment Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. Specific computing devices may include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0035] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0036] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0037] In summary, the embodiments of this application provide an automated construction method, system, computer device, and storage medium for embodied intelligence training datasets. Using internet videos of human operations as the original data source, the system achieves automated construction of low-cost, scalable, and high-quality VLA training datasets through a fully automated processing pipeline that includes video acquisition and filtering, preprocessing and temporal segmentation, action inversion and pseudo-action tag generation, language instruction generation, action space mapping, VLA triple construction, and data augmentation and filtering. Compared with existing technologies, this application does not rely on real robot equipment for data collection and extensive manual annotation. It automatically generates pseudo-action labels through motion trajectory extraction and action inverse inference, and automatically generates imperative language instructions through a multimodal large model, significantly reducing the cost of dataset construction and improving annotation consistency. Using massive amounts of video data from the Internet as the data source, the data scale is not constrained by hardware conditions and can be expanded as needed. Through inverse kinematics mapping and kinematic constraint adaptation, human actions are transformed into standard action labels that robots can directly execute, ensuring the adaptability of training data to the target robot platform. The constructed dataset covers diverse real-world scenarios and operation types, effectively improving the generalization ability of the embodied intelligent robot VLA model, and is suitable for policy training of various embodied intelligent robots such as robotic arms, mobile robots, and humanoid robots.

[0038] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts of the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0039] The above-described embodiments are merely preferred embodiments of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A method for automatically constructing training datasets for embodied intelligence, characterized in that, include: Collect videos of humans operating objects and filter them to obtain video clips; The video segment is preprocessed to obtain a preprocessed video segment. The preprocessed video segment is then segmented into short video segments by motion change detection. Each short video segment contains a single atomic motion. Extract the motion trajectory data of the short video clip, and based on the motion trajectory data, reverse the atomic action type of the single atomic action to generate a continuous action sequence, and encapsulate the continuous action sequence into a corresponding pseudo action tag; The short video clip is input into a multimodal large model to generate an instruction description corresponding to the short video clip, and the instruction description is formatted into an imperative language instruction. After mapping the continuous action sequence in the pseudo-action label to the robot action space, standard action labels are obtained through kinematic constraint adaptation and trajectory optimization. The video frame sequence of the short video clip, the imperative language instructions, and the standard action tags are combined into VLA triples, and the VLA triples are output in a standard format to obtain a triple dataset. Data augmentation and filtering are performed on the triplet dataset to obtain a dataset for training the VLA model of the embodied intelligent robot.

2. The method for automatically constructing a training dataset for embodied intelligence according to claim 1, characterized in that, The preprocessing of the video segment to obtain the preprocessed video segment includes: After sequentially performing jitter removal and redundant frame removal on the video segment, it is uniformly adjusted to a preset resolution and preset frame rate to obtain the preprocessed video segment.

3. The method for automatically constructing a training dataset for embodied intelligence according to claim 1, characterized in that, The step of segmenting the preprocessed video clip into short video segments through motion change detection includes: The preprocessed video segment is subjected to frame-by-frame motion change detection based on the frame difference method to obtain motion change nodes. Using the action change node as the segmentation boundary, the preprocessed video segment is segmented into short video segments, and each short video segment contains only a single atomic action.

4. The method for automatically constructing a training dataset for embodied intelligence according to claim 1, characterized in that, The process of extracting motion trajectory data from the short video clips, and then, based on the motion trajectory data, deducing the atomic action type of the single atomic action to generate a continuous action sequence, includes: The three-dimensional spatial coordinates of each joint of the human hand in the short video clip are extracted by detecting key points of the hand, and the three-dimensional motion trajectory of the hand is obtained. Six degrees of freedom pose estimation is performed on the object being manipulated in the short video clip to obtain the pose change data of the object being manipulated. The motion trajectory data is obtained by temporally correlating the three-dimensional motion trajectory of the hand with the pose change data of the manipulated object based on optical flow tracking technology. The atomic action type of the single atomic action is deduced from the motion trajectory data. The atomic action type includes grasping, moving, rotating, pressing, or placing. Based on the atomic action type and the motion trajectory data, the continuous action sequence is generated, which includes: six-degree-of-freedom pose, gripper opening and closing state, and motion speed.

5. The method for automatically constructing a training dataset for embodied intelligence according to claim 1, characterized in that, After mapping the continuous action sequence in the pseudo-action labels to the robot action space, standard action labels are obtained through kinematic constraint adaptation and trajectory optimization, including: After mapping the continuous motion sequence to the robot's motion space using inverse kinematics, joint angle limit constraints are applied in conjunction with the target robot's kinematic parameters. The standard motion labels are then obtained through collision detection and avoidance and trajectory interpolation smoothing.

6. The method for automatically constructing a training dataset for embodied intelligence according to claim 1, characterized in that, The step of combining the video frame sequence of the short video clip, the imperative language instructions, and the standard action tags into a VLA triple includes: Frame extraction is performed on the short video segment to obtain the video frame sequence; The video frame sequence, the imperative language instructions, and the standard action tags are encapsulated into the VLA triple according to a preset data storage format.

7. An automated training dataset construction system for embodied intelligence, characterized in that, include: The video acquisition and filtering module is used to acquire and filter videos of humans operating objects to obtain video clips. The temporal segmentation module is used to preprocess the video segment to obtain a preprocessed video segment, and to segment the preprocessed video segment into short video segments by motion change detection, wherein the short video segment contains a single atomic action; The motion inversion module is used to extract motion trajectory data of the short video clip, invert the atomic action type of the single atomic action based on the motion trajectory data, generate a continuous action sequence, and encapsulate the continuous action sequence into a corresponding pseudo action tag. The instruction generation module is used to input the short video clip into the multimodal large model, generate an instruction description corresponding to the short video clip, and format the instruction description into an imperative language instruction. The motion mapping module is used to map the continuous motion sequence in the pseudo-motion labels to the robot motion space, and then obtain standard motion labels through kinematic constraint adaptation and trajectory optimization. The triplet construction module is used to combine the video frame sequence of the short video clip, the imperative language instructions, and the standard action tags into VLA triplets, and output the VLA triplets in a standard format to obtain a triplet dataset. The dataset generation module is used to perform data augmentation and data filtering on the triplet dataset to obtain a dataset for training the embodied intelligent robot VLA model.

8. A computer device, characterized in that, include: Processor and computer-readable storage media; A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements a method for automatically constructing a training dataset for embodied intelligence as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1 to 6, an automated method for constructing training datasets for embodied intelligence.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements an automated method for constructing training datasets for embodied intelligence as described in any one of claims 1 to 6.

Images