Robot body data labeling method and device based on artificial intelligence large model

Abstract

The invention provides a robot body data labeling method and device based on an artificial intelligence large model, and the method comprises the steps: obtaining the multi-modal original data of a robot during task execution, and carrying out the standardization processing of the original data, and obtaining the standardized data; performing multi-modal semantic understanding and analysis on the standardized data based on an artificial intelligence large model, and reversely generating a structured task description; and generating a structured labeling result according to the structured task description, and storing the structured labeling result into a labeling database. According to the method and the device provided by the invention, data format conversion and data enhancement are performed on the robot body data through the large model, so that the consistency of the data annotation source data file and the annotation result is realized. A unified semantic understanding standard is provided, subjective deviation of manual annotation is eliminated, semantic and time sequence synchronization of language, vision and action is guaranteed through an end-to-end multi-mode alignment mechanism, the labor cost is greatly reduced through an automatic process, and large-scale data production is supported.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for labeling embodied data of robots based on a large artificial intelligence model. Background Technology

[0002] Embodied intelligence emphasizes that intelligent agents (such as robots) learn and perform tasks in interaction with the physical environment. Its core relies on high-quality, structured multimodal data, including visual perception, language commands, and action execution. In the process of building such systems, data annotation becomes a key link.

[0003] Because embodied tasks typically involve complex sequences of actions, different annotators (or operators) may exhibit significant behavioral deviations when performing the same task due to personal habits, differences in understanding, or operational styles. For example, the way objects are grasped, the movement path, or the rhythm of actions may differ, leading to a lack of comparability and generalization in annotation results. Furthermore, embodied intelligence relies on the semantic alignment between language instructions, visual observations, and action sequences. However, these three modalities differ significantly in terms of expression level, temporal scale, and level of abstraction: language is discrete and abstract, vision is continuous and high-dimensional, while actions are temporal and physically constrained. This "semantic gap" makes accurately aligning instructions with corresponding action fragments, or establishing joint representations across modalities, extremely challenging. The annotation process must simultaneously ensure precise synchronization of all three in terms of semantics and temporality, placing higher demands on annotation tools, processes, and evaluation metrics.

[0004] Due to the issues of label consistency and multimodal alignment in the existing technologies, traditional manual or semi-automatic labeling methods suffer from low efficiency, high cost, and poor scalability, making it difficult to meet the needs of embodied intelligence for large-scale, structured, and time-series aligned data. Therefore, there is an urgent need for a robot embodied data labeling method to solve these problems. Summary of the Invention

[0005] This invention provides a method and apparatus for robot embodied data annotation based on a large-scale artificial intelligence model, addressing the issues of annotation consistency and high difficulty in multimodal alignment in existing technologies, and achieving consistency between source data files and annotation results. It provides a unified semantic understanding standard, eliminates subjective bias in manual annotation, and ensures semantic and temporal synchronization among language, vision, and action through an end-to-end multimodal alignment mechanism.

[0006] This invention provides a method for labeling embodied robot data based on a large-scale artificial intelligence model, comprising:

[0007] Acquire multimodal raw data of the robot when performing tasks, and perform standardization processing on the raw data to obtain standardized data;

[0008] Based on a large artificial intelligence model, the standardized data is subjected to multimodal semantic understanding and parsing to generate a structured task description.

[0009] Based on the structured task description, structured annotation results are generated and stored in the annotation database.

[0010] According to the present invention, a robot embodied data annotation method based on an artificial intelligence large model is provided. This method performs multimodal semantic understanding and parsing on the standardized data based on the artificial intelligence large model, and reverse-generates a structured task description, including:

[0011] Based on the large-scale artificial intelligence model, the visual data in the standardized data is analyzed to obtain corresponding visual events and action segments; based on the large-scale artificial intelligence model, the language data in the standardized data is parsed to obtain language instructions.

[0012] According to the present invention, a robot embodied data annotation method based on a large artificial intelligence model generates structured annotation results based on the structured task description, including:

[0013] The language instructions, visual events, and action segments are aligned on the timeline to generate fine-grained structured annotation results.

[0014] According to the present invention, a robot embodied data annotation method based on a large artificial intelligence model further includes, before generating structured annotation results based on the structured task description:

[0015] Based on a large artificial intelligence model, scarce task data is generated. Multimodal semantic understanding and parsing are performed on the scarce task data to generate a structured task description.

[0016] According to the robot embodied data annotation method based on a large artificial intelligence model provided by the present invention, after generating the structured annotation results, it further includes:

[0017] The structured annotation results are quantitatively evaluated, and erroneous samples are selected to construct an error analysis dashboard.

[0018] According to the present invention, a method for labeling embodied robot data based on a large artificial intelligence model is provided, which includes quantitatively evaluating the structured labeling results, including:

[0019] By comparing and learning and cross-modal attention alignment verification, the alignment confidence score corresponding to the structured annotation result is generated.

[0020] According to the present invention, a robot embodied data annotation method based on a large artificial intelligence model further includes, after quantitatively evaluating the structured annotation results:

[0021] Based on the alignment confidence score, samples with low confidence are archived to form an error dataset, and the large artificial intelligence model is fine-tuned and updated based on the error dataset.

[0022] The present invention also provides a robot embodied data annotation device based on a large artificial intelligence model, comprising:

[0023] The data synchronization module is used to acquire multimodal raw data of the robot when performing tasks, and to standardize the raw data to obtain standardized data.

[0024] The data parsing module is used to perform multimodal semantic understanding and parsing on the standardized data based on the large artificial intelligence model, and to generate a structured task description in reverse.

[0025] The annotation generation module is used to generate structured annotation results based on the structured task description and store the structured annotation results in the annotation database.

[0026] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robot embodied data annotation method based on the artificial intelligence large model as described above.

[0027] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot embodied data annotation method based on an artificial intelligence large model as described above.

[0028] This invention provides a method and apparatus for labeling embodied robot data based on a large-scale artificial intelligence model. The method acquires multimodal raw data from a robot performing a task, standardizes the raw data to obtain standardized data, and then performs multimodal semantic understanding and parsing on the standardized data using a large-scale artificial intelligence model to generate a structured task description. Based on the structured task description, structured annotation results are generated and stored in an annotation database. The method and apparatus of this invention perform data format conversion and data augmentation on embodied robot data using a large-scale model, achieving consistency between the source data file and the annotation results. It provides a unified semantic understanding standard, eliminates subjective bias in manual annotation, ensures semantic and temporal synchronization between language, vision, and action through an end-to-end multimodal alignment mechanism, and significantly reduces labor costs through automated processes, supporting large-scale data production. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating the robot embodied data annotation method based on a large artificial intelligence model provided by the present invention.

[0031] Figure 2 This is a schematic diagram of an optional Prompt in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the process of constructing an end-to-end inference pipeline in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the quality assessment and iterative continuous learning process in an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of the structure of the robot embodied data annotation device based on a large artificial intelligence model provided by the present invention;

[0035] Figure 6 A schematic diagram of the workflow of an optional robot embodied data annotation device based on a large artificial intelligence model in an embodiment of the present invention;

[0036] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0038] The following is combined with Figures 1-4 This invention describes a robot embodied data annotation method based on a large artificial intelligence model.

[0039] like Figure 1 As shown, the robot embodied data annotation method based on a large artificial intelligence model provided by this invention includes the following steps:

[0040] S1. Obtain the multimodal raw data of the robot when performing the task, and standardize the raw data to obtain standardized data.

[0041] Specifically, robots are deployed to perform tasks in diverse environments, simultaneously recording multimodal raw data. Currently, industry data collection methods employ a hybrid approach combining real-world and simulation environments. Data collectors construct virtual mirror images of the real environment using simulation platforms, simultaneously running strategies within them, or remotely controlling robots to perform tasks and collect data using remote control devices such as VR controllers, joysticks, exoskeletons, and haptic gloves. The collected data includes, but is not limited to:

[0042] (1) Multi-view RGB / depth video (RGB-D video);

[0043] (2) Robot state (including but not limited to joint angles, velocity, torque, etc.);

[0044] (3) Action commands, voice commands, eye movements, and gestures (control signals);

[0045] (4) Task context (e.g., cooking, tidying up the room);

[0046] (5) Position, orientation, and contact force of 3D objects;

[0047] (6) Noise-free sensor data (can simulate RGB-D (color + depth), LiDAR (Light Detection and Ranging), IMU (Inertial Measurement Unit). These noises will reduce the data quality in real sensors, but can be completely eliminated in the simulation environment to obtain "idealized" measurement data);

[0048] (7) Any combination of viewpoint, lighting, and material.

[0049] By synchronously recording the aforementioned multimodal raw data, the limitations of traditional single-environment data acquisition are overcome. Richer training samples are obtained through hybrid acquisition, providing a data foundation for subsequent cross-modal alignment. Furthermore, extreme conditions can be simulated through the simulation environment, enhancing the robustness of the model.

[0050] After collecting the multimodal raw data, the multimodal raw data is standardized to obtain standardized data, including but not limited to timestamp alignment, noise reduction, and conversion to a unified format.

[0051] S2. Based on a large artificial intelligence model, perform multimodal semantic understanding and parsing of standardized data, and reverse generate structured task descriptions.

[0052] Large models generally refer to artificial intelligence models with a huge number of parameters, extensive training data, and powerful generalization and reasoning capabilities. In embodied intelligence and multimodal tasks, large models mainly include visual language models (VLMs) and large language models (LLMs). They can understand complex semantics, generate coherent content, and achieve knowledge transfer and collaborative reasoning in cross-modal scenarios. They are the core technological foundation for realizing automatic data annotation, task understanding, and decision generation for robots.

[0053] In this step, the visual data in the standardized data is analyzed using a large-scale artificial intelligence model to obtain corresponding visual events and action segments. The language data in the standardized data is then parsed using the same model to obtain language instructions. Specifically, the Visual Model (VLM) is used to perform scene-based understanding (e.g., object recognition, state changes, action intent recognition) on the visual sequence portion of the standardized data obtained in step S1, and the Language Model (LLM) is used to parse the language data portion of the standardized data obtained in step S1 to generate a structured task description.

[0054] In an optional embodiment of the present invention, the LLaVA-1.6 visual language model and the Qwen3-235B large language model can be used to deploy inference services on a server to automatically perform semantic parsing on the dataset embodied by the data robot, and then decompose and reverse generate labeled tasks. The deployment method uses an 8-card deployment of Huawei Ascend 910B series GPUs, enabling AWQ 4-bit quantization to reduce memory usage. Quantization technology makes it possible to deploy large models on edge devices, reducing inference costs. Simultaneously installed middleware tools include Docker+Kubernetes for service orchestration, Prometheus for monitoring QPS / latency, and FastAPI. Containerized deployment ensures high availability and scalability of the service.

[0055] S3. Based on the structured task description, generate structured annotation results and store the structured annotation results in the annotation database.

[0056] In this step, language instructions, visual events, and action segments are aligned on the timeline, generating fine-grained structured annotation results. Specifically, based on joint inference using large models (VLM, LLM), language instructions, visual events, and action segments are aligned on the timeline through attention mechanisms or temporal matching algorithms. Fine-grained annotations are then automatically generated (including but not limited to task phase planning, task parsing and agent division of labor, action semantic labels, object interaction relationships, state transitions, etc.), generating structured annotation result files and storing them in an annotation database. This enables source data association and version management for the corresponding annotation results. After aligning language instructions, visual events, and action segments on the timeline, cross-modal event alignment queries are supported (e.g., whether the "grasp" action falls within the time window of the "grab the cup" language segment).

[0057] In an optional embodiment of the present invention, the structured annotation results can be generated using the following technical details:

[0058] (1) Key sampling is performed on the source video streams or video files of the collected dataset. Keyframes are extracted based on the rate of change of action and visual differences, with 10-50 frames retained for each task to reduce the VLLM inference load. Keyframe sampling significantly improves annotation efficiency and greatly reduces computational costs. Language instructions are cleaned to remove ASR noise, and the instruction format is standardized to "Please execute: XXX". Standardizing the instruction format can improve the model's generalization ability.

[0059] (2) Establish a prompt design as a fixed version, send key frames in batches to LLaVA-1.6, obtain the semantic description of each frame, and then use rules or small models to map the description to predefined action labels, aggregate continuous frames and identify the start and end times of actions.

[0060] (3) Task decomposition and reverse annotation generation are performed using the Qwen3-235B model. Instructions are constructed and action event sequences and object lists are output. An optional Prompt design is as follows: Figure 2 As shown, the Prompt is designed with a structured task decomposition approach, and finally calls the Qwen3-235B API to generate structured JSON. The structured JSON output ensures the parsability and reusability of the annotation results.

[0061] (4) Utilizing a CI / CD system, employing service-oriented and automated pipelines, and using Kubeflow Pipelines for task orchestration, an end-to-end inference pipeline is constructed, with the process as follows: Figure 3 As shown, the automated pipeline enables minute-level annotation generation, significantly improving efficiency compared to manual annotation, greatly reducing labor costs, and supporting large-scale data production.

[0062] (5) Unified Schema (data schema).

[0063] The unified schema establishes a standardized data structure for multimodal data (RGB-D video, LiDAR point clouds, IMU sensor data, voice commands, etc.). Through predefined field names, data types, value ranges, and constraints, it ensures that data from different sources and batches maintains a consistent format. For example, all action labels use the same naming rules (e.g., "grasp", "move", "place"), and all timestamps use a unified UTC format, avoiding data parsing errors caused by format differences. In robot embodied data, data from different modalities needs precise alignment on the timeline. The unified schema ensures accurate correspondence between visual, language, and action data by defining shared timestamp fields and time-series indexes. For example, the grasping action in a visual frame and the voice command "Please grab the cup" can be aligned within the same time window, providing high-quality cross-modal supervision signals for model training. Furthermore, the standardized schema enables annotation tools to automatically identify and process data fields, reducing the workload of manual configuration and format conversion. Annotators do not need to concern themselves with the underlying data format; they only need to annotate according to the predefined labeling system, and the annotation results are automatically stored in a unified format. Meanwhile, the unified schema supports batch processing and automated pipelines, significantly improving annotation efficiency.

[0064] The following is an optional schema example in an embodiment of the present invention:

[0065] "task_id":"place_cup_on_table", / / "Task ID": "Place the cup on the table"

[0066] "task_description":"Please put the red cup on the table." / / "Task description": "Please put the red cup on the table."

[0067]

[0068]

[0069] Because embodied tasks are extremely unevenly distributed in the real world, a large number of tasks belong to low-frequency or rare scenarios (such as "repairing home appliances with specific tools" or "performing delicate operations in a confined space"). These long-tail tasks are difficult to sample sufficiently through conventional data collection, leading to poor model performance when faced with novel or marginal situations. How to efficiently collect, synthesize, or transfer labeled data to cover long-tail tasks has become a key bottleneck in improving the robustness and generalization ability of the system. To address the problem of insufficient coverage of long-tail tasks, in one optional embodiment of this invention, before generating structured labeled results, the method further includes: generating scarce task data based on a large artificial intelligence model, performing multimodal semantic understanding and parsing on the scarce task data, and reverse-generating structured task descriptions. Specifically, large models (VLM, LLM) are used to perform semantic expansion and scene reconstruction on low-frequency or rare scenario tasks, generating high-quality synthetic samples through instruction rewriting, action interpolation, and vision-action inversion to expand the long-tail distribution.

[0070] Instruction rewriting refers to using the text generation capabilities of Large Language Models (LLMs) to semantically expand and diversify the expression of the original instructions, generating new instructions with the same semantics but different forms of expression.

[0071] Action interpolation refers to the process of mathematically interpolating real action trajectories to generate smooth and continuous new action sequences, thereby achieving geometric expansion of action data.

[0072] Visual-motion inversion refers to the process of analyzing visual observations and inferring corresponding action commands in reverse, thus achieving a reverse mapping from vision to action.

[0073] Taking the "pouring coffee" task in a kitchen scenario as an example, the scenario description is that the robot needs to complete the "pouring coffee" task, but there are only a few relevant samples in the training data (let's say 10). Assuming the original instruction is "Please pour coffee", the process of expanding the long-tail distribution is as follows:

[0074] Step 1: Instruction Rewrite Extension

[0075] Original instruction: "Please pour me coffee."

[0076] Generate variants:

[0077] (1) "Pour the coffee into the cup";

[0078] (2) "Pick up the coffee pot and pour coffee into the cup";

[0079] (3) "Please pour the coffee";

[0080] (4) "Pour the coffee from the pot into the cup";

[0081] (5)……

[0082] Generate 20 different expressions, covering different language habits.

[0083] Step 2: Action Interpolation Generation

[0084] Keyframes were extracted from the actual "pouring coffee" action, and interpolation was performed on key actions such as grabbing, moving, pouring, and placing to generate 50 smooth coffee-pouring action variations. The angle, speed, and trajectory of the coffee were adjusted to increase the diversity of the actions.

[0085] Step 3: Visual-Motion Inversion

[0086] RGB-D video of the coffee-pouring process is captured, and VLM is used to analyze each frame to deduce the corresponding motion commands, generating a motion sequence of "grab coffee pot" → "move to cup position" → "tilt to pour coffee" → "put coffee pot back", and establishing time stamp alignment between visual frames and motion commands.

[0087] Step 4: Data Quality Assessment

[0088] By filtering out unreasonable actions using a small model discriminator and manually sampling 10% of the generated data to ensure the physical rationality of the actions, 200 high-quality "pouring coffee" training samples were finally obtained.

[0089] Using the above method, the "pouring coffee" task, which originally had only 10 samples, was expanded to 200 samples, significantly improving the performance of long-tail tasks.

[0090] To comprehensively measure annotation quality from multiple dimensions, in an optional embodiment of the present invention, after generating structured annotation results, the method further includes quantitatively evaluating the structured annotation results and screening out erroneous samples to construct an error analysis dashboard. Specifically, alignment confidence scores corresponding to the structured annotation results can be generated through comparative learning and cross-modal attention alignment verification. Preferably, the evaluation indicators can be designed according to three categories: annotation accuracy, cross-modal alignment accuracy, and task coverage. Quantitative evaluation can be performed using methods such as manual sampling, rule verification, and small model discriminators, based on multimodal timestamp embedding alignment metrics, task type distribution statistics, and long-tail task recall. A comprehensive multi-dimensional evaluation system for measuring annotation quality avoids bias from a single indicator. The evaluation of long-tail task recall ensures the model's generalization ability to rare scenarios, and the multimodal alignment metric verifies the spatiotemporal consistency of visual, linguistic, and action data. The following example uses a robot performing a "grab a cup" task to verify the temporal alignment between the visual grasping action and the voice command "Please grab the cup":

[0091] (1) Extract visual feature sequences from RGB-D video and extract text features from voice commands.

[0092] (2) Map visual and text features to a unified timeline and calculate the similarity of each timestamp.

[0093] (3) Calculate the visual attention matrix to the text and find the alignment relationship between the "grab" action and the "grab" word.

[0094] (4) Calculate the maximum attention weight of the alignment pair and obtain a confidence score of 0.85.

[0095] (5) Since the confidence level is >0.8 (preset threshold), it is determined to be a high-quality alignment, and the alignment timestamp and confidence score are recorded.

[0096] Through comparative learning and cross-modal attention validation, the system can automatically identify and quantify the alignment quality of cross-modal events, providing a reliable alignment confidence index for subsequent data evaluation and model optimization, and significantly improving the accuracy and reliability of multimodal data annotation.

[0097] After quantitatively evaluating the structured annotation results, structural attribution is performed on the error samples found in the evaluation to construct an error analysis dashboard. This dashboard-style display facilitates team collaboration and issue tracking. The error analysis dashboard supports filtering samples by error type, task category, and model version, and the exported results are used for fine-tuning of large models. The following is an attribution classification table for one example in this embodiment of the invention:

[0098] Table 1 Attribution Classification Table

[0099]

[0100] To achieve adaptive closed-loop optimization of the model and continuously improve annotation quality, in an optional embodiment of the present invention, after quantitatively evaluating the structured annotation results, the method further includes: archiving low-confidence samples based on alignment confidence scores to form an error dataset, and fine-tuning and updating the large-scale artificial intelligence model based on the error dataset. Specifically, as shown... Figure 4 As shown, an "error dataset" is formed by automatically archiving low-confidence or manually corrected samples. The VLM is fine-tuned using LoRA on the image description pairs for error correction to enhance object state and action intent recognition; the LLM is fine-tuned using SFT (Supervised Fine-Tuning) to optimize the task decomposition logic. The fine-tuned model is then updated and deployed to the inference service. Preferably, the adaptive closed-loop optimization of the model also includes model version management and A / B testing to ensure improved accuracy / alignment precision without catastrophic forgetting. Furthermore, an automatic triggering mechanism can be set. For example, if the error rate of a certain type exceeds a preset threshold for five consecutive days (e.g., F1 < 0.6 for long-tail tasks), the fine-tuning task is automatically triggered. After fine-tuning, the new model is deployed to the inference service and enters the next annotation-evaluation cycle.

[0101] The following is combined with Figure 5 The present invention will be described in detail with reference to a specific embodiment of the robot embodied data annotation method based on a large artificial intelligence model.

[0102] Scene description:

[0103] The robot performs the "making coffee" task in the kitchen environment, which requires a complete process from "taking the coffee cup - pouring coffee - adding sugar - stirring - serving to the user".

[0104] Data collection phase:

[0105] Simultaneous acquisition of multi-view RGB-D video, robot joint trajectories, end effector poses, and the voice command "Please make a cup of coffee" in a real kitchen and simulation environment. A modal data synchronizer is used to time-align heterogeneous data such as visual, motion, and language data.

[0106] Automatic annotation process:

[0107] 1. Keyframe Sampling: Extract 30 keyframes from a 10-minute video, covering key action nodes such as "reaching for the cup - grabbing - moving - pouring coffee - adding sugar - stirring - delivering". Collect 3D object position (coffee cup, coffee pot, sugar bowl, spoon) and contact force data.

[0108] 2. Extract semantic information from images using the VLM visual semantic parser: LLaVA-1.6 identifies objects and actions in each frame, such as "hand approaching coffee cup", "grabbing coffee cup", and "pouring coffee liquid".

[0109] 3. Transform natural language instructions into structured task descriptions through the LLM instruction structuring engine: Qwen3-235B decomposes "making coffee" into a structured task description, including objectives, constraints, and sub-task sequences.

[0110] 4. Input semantic information and structured task description into the cross-modal alignment inference engine for alignment inference. Achieve accurate alignment of language, vision and action in semantics and temporality through attention mechanism or temporal matching algorithm. Generate standardized annotation results through automatic annotation generator.

[0111] Evaluation and optimization:

[0112] Quality verification was conducted through consistency checks and feedback loops. The annotation results were validated using a rule engine or small model. The evaluation revealed that the annotation accuracy for the "pouring coffee" action was low because the flow of coffee liquid was difficult to accurately identify. The error analysis dashboard showed that this problem mainly occurred in scenarios with changing lighting. An automatic triggering mechanism collected low-confidence samples and performed VLM fine-tuning to enhance the liquid flow recognition capability. After two rounds of fine-tuning, the annotation accuracy improved from 75% to 92%. When the task being performed is scarce, the long-tail data augmentation module can also be driven to generate new instruction variants based on LLM, combine VLM and physical simulation to generate reasonable action sequences, synthesize scarce task data, and form a closed-loop optimization mechanism to comprehensively improve the consistency, coverage, and automation level of the annotation.

[0113] In summary, the robot embodied data annotation method based on a large AI model provided by this invention solves the problems of low annotation efficiency and inability to achieve cross-modal collaborative annotation caused by traditional single annotation. Furthermore, it addresses the issue of data annotation consistency by performing data format conversion and data augmentation on the robot embodied data through a large model, achieving consistency between the source data file and the annotation results. It provides a unified semantic understanding standard, eliminating subjective bias in manual annotation and meeting the data annotation requirements of interactivity, temporality, multimodality, and task orientation required for intelligent robot embodied execution of tasks. It implements an end-to-end multimodal alignment mechanism to ensure semantic and temporal synchronization among language, vision, and action. Moreover, it leverages the generalization capability of the large model to cover long-tail tasks, overcoming the data scarcity bottleneck and solving the problem of insufficient coverage of long-tail tasks.

[0114] Based on the same inventive concept, the present invention also provides a robot embodied data annotation device based on a large artificial intelligence model. The robot embodied data annotation device based on a large artificial intelligence model provided by the present invention will be described below. The robot embodied data annotation device based on a large artificial intelligence model described below and the robot embodied data annotation method based on a large artificial intelligence model described above can be referred to and corresponded to each other.

[0115] like Figure 6 As shown, the robot embodied data annotation device based on artificial intelligence large model provided by the present invention includes a data synchronization module 61, a data parsing module 62, and an annotation generation module 63.

[0116] Data synchronization module 61 is used to acquire multimodal raw data of the robot when performing tasks, and to standardize the raw data to obtain standardized data;

[0117] Data parsing module 62 is used to perform multimodal semantic understanding and parsing on the standardized data based on the artificial intelligence big data model, and to generate a structured task description in reverse.

[0118] The annotation generation module 63 is used to generate structured annotation results based on the structured task description and store the structured annotation results in the annotation database.

[0119] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute the robot embodied data annotation method based on the artificial intelligence large model provided by the above methods, which includes:

[0120] Acquire multimodal raw data of the robot when performing tasks, and perform standardization processing on the raw data to obtain standardized data;

[0121] Based on a large artificial intelligence model, the standardized data is subjected to multimodal semantic understanding and parsing to generate a structured task description.

[0122] Based on the structured task description, structured annotation results are generated and stored in the annotation database.

[0123] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the robot embodied data annotation method based on the artificial intelligence large model provided by the above methods, the method comprising:

[0125] Acquire multimodal raw data of the robot when performing tasks, and perform standardization processing on the raw data to obtain standardized data;

[0126] Based on a large artificial intelligence model, the standardized data is subjected to multimodal semantic understanding and parsing to generate a structured task description.

[0127] Based on the structured task description, structured annotation results are generated and stored in the annotation database.

[0128] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot embodied data annotation method based on the artificial intelligence large model provided by the above methods, the method comprising:

[0129] Acquire multimodal raw data of the robot when performing tasks, and perform standardization processing on the raw data to obtain standardized data;

[0130] Based on a large artificial intelligence model, the standardized data is subjected to multimodal semantic understanding and parsing to generate a structured task description.

[0131] Based on the structured task description, structured annotation results are generated and stored in the annotation database.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for robot embodiment data annotation based on an artificial intelligence large model, characterized in that, include: Acquire multimodal raw data of the robot when performing tasks, and perform standardization processing on the raw data to obtain standardized data; Based on a large artificial intelligence model, the standardized data is subjected to multimodal semantic understanding and parsing to generate a structured task description. Based on the structured task description, structured annotation results are generated and stored in the annotation database.

2. The robot embodied data annotation method based on a large artificial intelligence model according to claim 1, characterized in that, Based on a large-scale artificial intelligence model, multimodal semantic understanding and parsing are performed on the standardized data to generate a structured task description, including: Based on the large-scale artificial intelligence model, the visual data in the standardized data is analyzed to obtain corresponding visual events and action segments; based on the large-scale artificial intelligence model, the language data in the standardized data is parsed to obtain language instructions.

3. The robot embodied data annotation method based on a large artificial intelligence model according to claim 2, characterized in that, Based on the structured task description, structured annotation results are generated, including: The language instructions, visual events, and action segments are aligned on the timeline to generate fine-grained structured annotation results.

4. The robot embodied data annotation method based on a large artificial intelligence model according to claim 1, characterized in that, Before generating structured annotation results based on the structured task description, the process also includes: Based on a large artificial intelligence model, scarce task data is generated. Multimodal semantic understanding and parsing are performed on the scarce task data to generate a structured task description.

5. The robot embodied data annotation method based on a large artificial intelligence model according to claim 1, characterized in that, After generating the structured annotation results, the following is also included: The structured annotation results are quantitatively evaluated, and erroneous samples are selected to construct an error analysis dashboard.

6. The robot embodied data annotation method based on a large artificial intelligence model according to claim 5, characterized in that, The structured annotation results are quantitatively evaluated, including: By comparing and learning and cross-modal attention alignment verification, the alignment confidence score corresponding to the structured annotation result is generated.

7. The robot embodied data annotation method based on a large artificial intelligence model according to claim 6, characterized in that, After quantitatively evaluating the structured annotation results, the following steps are also included: Based on the alignment confidence score, samples with low confidence are archived to form an error dataset, and the large artificial intelligence model is fine-tuned and updated based on the error dataset.

8. A robot embodied data annotation device based on a large artificial intelligence model, characterized in that, include: The data synchronization module is used to acquire multimodal raw data of the robot when performing tasks, and to standardize the raw data to obtain standardized data. The data parsing module is used to perform multimodal semantic understanding and parsing on the standardized data based on the large artificial intelligence model, and to generate a structured task description in reverse. The annotation generation module is used to generate structured annotation results based on the structured task description and store the structured annotation results in the annotation database.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the robot embodied data annotation method based on a large artificial intelligence model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot embodied data annotation method based on a large artificial intelligence model as described in any one of claims 1 to 7.

Images