Model optimization method, computing device, and readable storage medium

Abstract

The application discloses a model optimization method, a computing device and a readable storage medium, comprising: dividing an embodied task into a plurality of execution stage subtask stages; obtaining simulation data corresponding to the subtask stages in a physical simulation environment; based on the simulation data, constructing a subtask level sample for identifying model training; and training the identification model using the subtask level sample. In this way, by dividing the embodied task into subtask stages with time sequence logic and training the identification model based on the subtask stages to construct training samples, the model can learn a fine-grained phased understanding of the task execution process, thereby improving the understanding and training efficiency of the model in the embodied intelligent scene.

Description

Technical Field

[0001] This invention relates to the field of robot application technology, and in particular to a model optimization method, computing device, and readable storage medium. Background Technology

[0002] In the fields of embodied intelligence and robotic manipulation, the operational tasks performed by robotic arms are a fundamental and crucial capability, directly impacting the efficiency and reliability of operations in scenarios such as industrial automation and logistics sorting. Currently, the determination of the execution status of operational tasks primarily relies on visual perception. Existing technologies typically employ multimodal large models for detection, analyze the acquired images, and output conclusive judgments such as "success" or "failure."

[0003] However, actual operational tasks often consist of a series of ordered actions. Existing identification methods typically focus on the final execution result, lacking the ability to identify and represent intermediate states and stages of progress during task execution.

[0004] Therefore, improving the identification and reasoning of operational processes has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a model optimization method, computing device, and storage medium that can improve the fine-grained staged understanding of the robotic arm task execution process and enhance the model's understanding and training efficiency of the task execution process in embodied intelligence scenarios.

[0006] To achieve the above objectives: In a first aspect, embodiments of this application provide a model optimization method, comprising: dividing an embodied task into multiple sub-task stages of execution; acquiring simulation data corresponding to the sub-task stages in a physical simulation environment; constructing sub-task-level samples for training a recognition model based on the simulation data; and training the recognition model using the sub-task-level samples.

[0007] In one embodiment, the embodied task is divided into multiple sub-task stages, including: dividing the embodied task into multiple sub-task stages based on a predefined sequence of path key points; the motion process between adjacent path key points constitutes a sub-task stage; in a physical simulation environment, acquiring simulation data corresponding to the sub-task stage includes: planning the motion trajectory of the robotic arm end effector in the sub-task stage based on the path key points corresponding to the sub-task stage; generating control instructions to control the robotic arm to complete the sub-task stage based on the motion trajectory; executing the control instructions in the physical simulation environment to complete the sub-task stage, and collecting image data corresponding to the sub-task stage during the execution of the control instructions; and using the image data and the identification information of the sub-task stage as at least part of the simulation data.

[0008] In one embodiment, the motion trajectory of the robotic arm end effector during the sub-task phase is planned based on the path key points corresponding to the sub-task phase. This includes: generating a connection pose trajectory of the robotic arm end effector between adjacent path key points using a smooth trajectory interpolation algorithm based on the path key points corresponding to the sub-task phase; and generating control commands to control the robotic arm to complete the sub-task phase based on the motion trajectory. This includes: converting the pose trajectory of the robotic arm end effector into control commands in the joint space of the robotic arm based on inverse kinematics solution.

[0009] In one embodiment, the simulation data includes task execution failure samples, and the failure samples carry failure type labels based on a predefined failure type system. Based on the simulation data, sub-task-level samples for identifying model training are constructed, including: for simulation data belonging to task execution failure samples, a question-answer pair is generated according to the failure type label corresponding to the sub-task stage in which the task execution failure sample is located; the question part of the question-answer pair is used to inquire about the execution status of the sub-task stage; the answer part includes the status judgment of the sub-task stage execution failure and the reason description corresponding to the failure type label; the question-answer pair, the identification information of the sub-task stage, and the corresponding image data are associated to form a sub-task-level sample.

[0010] In one embodiment, executing control commands in a physical simulation environment includes: performing real-time contact detection using a physical simulation engine based on the physical simulation environment; generating contact information in response to detecting contact between the end effector of the robotic arm and a target object; and controlling the end effector of the robotic arm to perform operations based on the contact information to execute control commands.

[0011] In one embodiment, the method further includes: during and / or after the acquisition process, detecting whether there are any abnormal situations between objects based on the contact information provided by the physical simulation engine; if an abnormal situation is detected, determining that the simulation data of the corresponding sub-task stage is abnormal, filtering the abnormal simulation data, or regenerating the corresponding control instructions to re-execute the embodied task of the sub-task stage.

[0012] In one embodiment, acquiring image data corresponding to a sub-task stage includes: simultaneously acquiring image data of the sub-task stage from multiple different camera perspectives; the method further includes: for a complete embodied task, extracting image data corresponding to each perspective in each sub-task stage; arranging image data from different camera perspectives in the same sub-task stage along a first dimension; arranging image data from the same perspective in different sub-task stages along a second dimension to form a multi-view temporal image matrix; the multi-view temporal image matrix is ​​used to construct or as part of simulation data.

[0013] In one embodiment, in a physical simulation environment, acquiring simulation data corresponding to a subtask stage includes: performing domain randomization processing on the physical simulation environment, the domain randomization processing including at least one of the following: randomly adjusting the lighting conditions and / or background of the simulation environment in the physical simulation environment; and randomly generating multiple natural language instructions with the same semantics but different expressions for the task logic of the same embodied task issued by the physical simulation environment.

[0014] Secondly, embodiments of this application provide a computing device, including: a processor and a memory storing a computer program, wherein when the processor runs the computer program, the above-described model optimization method is implemented.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned model optimization method.

[0016] The model optimization method, computing device, and storage medium provided in this application's embodiments divide the embodied task into multiple sub-task stages of execution; acquire simulation data corresponding to the sub-task stages in a physical simulation environment; construct sub-task-level samples for training the recognition model based on the simulation data; and train the recognition model using the sub-task-level samples. Thus, the recognition model optimized by the above method in this application's embodiments possesses the ability to understand the stages of the embodied task. When facing a task involving a continuous operating environment, the model can identify the current execution stage of the task based on multiple frames of input images and task instructions, and determine the task execution status of that stage. Compared with traditional methods that only output the final conclusion, the optimized model can provide recognition and reasoning of the operation process, enabling a more granular staged understanding of the task execution process. Furthermore, if a deviation is determined during task execution, it outputs state description information associated with that stage, thereby providing support for state monitoring and adaptive adjustment during task execution. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the model optimization method provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram illustrating an application scenario of the model optimization method provided in the embodiments of this application.

[0019] Figure 3 This is a schematic diagram illustrating another application scenario of the model optimization method provided in the embodiments of this application.

[0020] Figure 4 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the scope of the application.

[0022] The following is an explanation of the terms that may be involved in this embodiment: Visual Language Model (VLM) is an artificial intelligence model capable of simultaneously processing, understanding, and associating visual signals (such as images and videos) with natural language. The core technologies of VLM include, but are not limited to, at least one of the following: visual feature encoding, text semantic understanding, cross-modal alignment, and semantic reasoning and generation. The purpose of VLM is to achieve deep semantic interaction between images and language, enabling machines not only to see content but also to describe, reason, ask questions, and understand using human language, thereby supporting more comprehensible, interpretable, and interactive intelligent vision applications.

[0023] MuJoCo simulation platform: refers to a multi-joint dynamics with contact simulation engine applied to the fields of robotics, biomechanics, animation, and control. MuJoCo's core technical features include, but are not limited to, at least one of the following: efficient contact dynamics calculation, rigid body and soft body physical simulation, joint drive and constraint systems, and real-time or offline motion planning and control. The purpose of MuJoCo is to provide a highly configurable, physically accurate, and computationally efficient simulation environment, enabling researchers and engineers to model, test, and optimize the motion, interaction, and control of robots or organisms in a virtual environment. This reduces the cost and risk of experimentation and development in real systems and supports algorithm research and application verification in fields such as embodied intelligence, reinforcement learning, and robotics.

[0024] The object_sim project is an open-source repository specifically designed for providing general-purpose 3D models and physical parameters of objects in the fields of robot simulation and embodied intelligence. The core contribution of object_sim lies in its systematic collection, organization, and provision of standardized model files for various everyday objects. These files include, but are not limited to, at least one of the following: geometric mesh data, mass and inertial parameters, surface material properties (such as friction coefficient and visual texture), and predefined grasping pose information. The purpose of object_sim is to reduce the complexity and repetitive work of setting up simulation environments, enabling researchers to quickly and consistently import diverse object models into physics simulation engines such as MuJoCo. This supports the algorithm development, testing, and evaluation for tasks such as robot grasping, manipulation planning, and multi-object interaction, improving the convenience and reproducibility of simulation experiments.

[0025] See Figure 1 This application provides a model optimization method, which can be executed by a model optimization device provided in this application. The model optimization device can be implemented in software and / or hardware, such as a computer or server. For example, the model optimization method can run in a model optimization system, which can be integrated into a server or robot terminal. In this embodiment, the robot terminal is used as the execution subject of the model optimization method. The model optimization method provided in this embodiment includes: Step 11: Divide the embodied task into multiple sub-task phases of the execution phase.

[0026] Embodied tasks can refer to goal-oriented operations performed by robots or intelligent agents in a physical environment by perceiving the environment through their own sensors and interacting with the environment through actuators.

[0027] For example, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating an application scenario of the model optimization method provided in the embodiments of this application, such as... Figure 2 As shown, embodied tasks can refer to the operations performed by a robotic arm on an object in a desktop scenario, such as grasping a single object or moving an object from one location to another.

[0028] Among them, multiple execution stages can refer to a complete embodied task from beginning to end, which can be decomposed into several continuous links with clear start and end states according to the time sequence and / or operational logic. These links together constitute the execution trajectory of the task, and a segment of the execution trajectory can be considered an execution stage.

[0029] In this context, a subtask phase can be a relatively independent and definable operation unit within a corresponding task. Each subtask phase has a clear operation objective (such as moving above an object), a starting state, and an ending state, and is usually a prerequisite for the execution of subsequent phases.

[0030] It is understandable that, such as Figure 2 As shown, common object manipulation tasks in a desktop scenario can include, but are not limited to, grasping and grasping-and-place tasks. These tasks are the most basic and core capabilities in robot operation. For example, the embodied tasks here can include grasping and / or grasping-and-place tasks.

[0031] The grasping task includes tasks such as moving the target object above it, moving the grasped object down, and lifting the object. The grasping and placing task includes tasks such as moving to the target object, grasping the object, lifting and moving it to the target area, and placing the object.

[0032] For example, dividing an embodied task into multiple sub-task stages of execution phases may include: dividing the embodied task into multiple sub-task stages of execution phases based on a structured task decomposition strategy. For example, based on the logical transformation of the operational objectives and the observable changes in the system state during task execution, the continuous task execution process is deconstructed into a series of discrete stage units with clear semantics.

[0033] Specifically, the natural language instructions or formalized task descriptions of the embodied task are first parsed to extract the core operational objective sequence necessary to complete the task. This sequence constitutes the logical framework of task execution. For example, for the instruction "put the red cube into the basket," the core operational objective chain can be parsed as: locating the red cube, grasping the red cube, moving it above the basket, and placing it into the basket. Further, the expected changes in key variables in the system state space before and after execution are defined for each core operational objective. These key variables include, but are not limited to: the relative spatial relationship between the robotic arm's end effector and the target object, the opening and closing state of the gripper, and the contact state between the object and the environment. It is understood that the moment when the expected state change occurs or when the state reaches a specific threshold can be set as the observation point of the task phase. These observation points can be triggered and identified by path key points, sensor readings (contact signals), specific actions (such as gripper closing commands), or environmental events (such as the object being picked up). Then, the system state change process between two adjacent observation points is defined as a sub-task phase. Each stage is associated with a core operational objective and is given a descriptive stage label, such as the pre-fetching stage, the fetching execution stage, and the confined space movement stage. Thus, the division is based not only on spatial paths but also on the degree to which the operational intent is achieved and the system state transitions.

[0034] Step 12: In the physical simulation environment, acquire the simulation data corresponding to the sub-task stage.

[0035] Among them, the physical simulation environment can refer to a virtual three-dimensional space constructed by computer software that follows the laws of classical mechanics. This space can simulate physical phenomena such as the motion, collision, contact, friction, and gravity of objects in the real world.

[0036] For example, see also Figure 2 and Figure 3 The physical simulation environment can be a virtual experimental platform built on the MuJoCo simulation engine. This platform includes programmable robot models (such as multi-joint robotic arms and their end grippers), object models with various geometric shapes and physical properties, and a visual rendering system that simulates real lighting and background, forming a digital sandbox that can be used to perform, observe, and analyze robot operation tasks.

[0037] It should be noted that MoJoCo is a high-performance physics simulation engine commonly used in embodied intelligence, robot control and motion planning, reinforcement learning, dynamic modeling and research. For example, such as... Figure 2 and Figure 3 As shown, MoJoCo is used to build the physical simulation environment, primarily focusing on modeling indoor scenes. Exemplarily, the physical simulation environment can be constructed and laid out by layering and modularizing the robot, objects, and environment. Specifically, the robot can be defined at the robot model layer, objects at the object model layer, and the environment at the scene integration layer. Specifically, firstly, at the robot model layer, a predefined robotic arm model file is imported from a publicly available standardized model resource library. For example, an XML description file of the Panda robotic arm (a general-purpose robotic arm model with 7 degrees of freedom and a two-finger gripper) can be used; however, it can be replaced with other robotic arm models, such as the TidyBot robotic arm, depending on the task requirements. Secondly, at the object model layer, a diverse object model library is constructed by combining multiple sources. For example, standardized XML model files of common objects can be obtained from open-source projects such as object_sim. For objects not covered, their 3D mesh models are created and their physical parameters are configured using text-based or graph-based 3D generative models (such as Tencent Hunyuan 3D models). For basic geometry, their XML descriptions can be generated directly through scripts. Finally, at the scene integration layer, robot models, object models, and environmental models representing the desktop and walls are laid out in the simulation scene to form a complete physical simulation environment that can be used to perform desktop operation tasks and collect simulation data.

[0038] Simulation data refers to all digital information generated in a physical simulation environment by simulating the execution of a tangible task. It is a comprehensive record of the task's execution state in the virtual environment, including but not limited to the visual appearance of the executing entity (such as a robotic arm) and the environment, as well as low-level information describing the system's internal dynamics, interactions, and control logic. Simulation data is multimodal, accurately reproducible, and typically contains rich automated annotation information.

[0039] For example, simulation data includes, but is not limited to, visual data, state data, semantic data, and labeled data. Visual data may include viewpoint images from multiple perspectives; state data may include the state information of the robotic arm during task execution; semantic data may include task instructions, task type, and target object identification information related to the embodied task; and labeled data may include stage labels for sub-task phases and task execution result labels (such as success, failure, and specific failure types or reasons).

[0040] Step 13: Based on simulation data, construct sub-task-level samples for training the recognition model.

[0041] The recognition model can be an artificial intelligence model capable of processing multimodal inputs, including visual and linguistic inputs, and performing state recognition and reasoning during the execution of embodied tasks. For example, the recognition model can be a VLM model.

[0042] Subtask-level samples refer to training data units extracted, processed, and structurally encapsulated from simulation data, based on the execution process of a single subtask stage. Each subtask-level sample completely corresponds to a specific stage in an embodied task (such as the pre-fetching stage, the fetching execution stage, etc., as mentioned above). Subtask-level samples can include visual observation information of that stage, task semantic descriptions, and annotations and supervision signals for the execution results of that stage, which is different from traditional training samples based on the entire task or a single frame image. The core of subtask-level samples lies in the alignment of the sample granularity with the task execution stage, and the inclusion of samples aims to guide the recognition model to learn to make fine-grained judgments and attributions of the execution state of that stage.

[0043] In this step, the original, low-level simulation data is transformed into high-level, sub-task-level samples with clear learning objectives. Each sample constitutes a complete learning instance from visual scene to task instruction to reasoning objective and finally to supervised answer. A large number of such samples are used to train the recognition model, enabling the recognition model to gradually learn to automatically complete state judgment and cause explanation for any given visual input at a certain sub-task stage. This allows the recognition model to achieve more granular task understanding and diagnostic capabilities.

[0044] Step 14: Train the recognition model using sub-task level samples.

[0045] For example, training the recognition model using sub-task-level samples can include: using the aforementioned dataset of sub-task-level samples, optimizing the parameters of the recognition model through supervised learning, enabling the recognition model to learn the ability to accurately identify task execution stages and determine the execution state of those stages from multimodal inputs. The core objective of the training is to establish a precise mapping relationship between visual observation, task semantics, and state description. Here, the multimodal input can be composite data formed by structured alignment and fusion of different types of information sources, capable of collectively describing the same event or process. Here, the multimodal input is the input data prepared for the recognition model.

[0046] In summary, this embodiment divides the embodied task into multiple sub-task stages of execution; acquires simulation data corresponding to the sub-task stages in a physical simulation environment; then constructs sub-task-level samples for training the recognition model based on the simulation data; and finally trains the recognition model using the sub-task-level samples. Thus, the recognition model trained in the above way possesses the ability to understand the stages of the embodied task. When faced with a task involving a continuous operational environment, the model can identify the current execution stage of the task based on multiple frames of input images and task instructions, and determine the task execution status of that stage. Compared with traditional methods that only output the final conclusion, the optimized model can provide recognition and reasoning of the operation process, enabling a more granular staged understanding of the task execution process. Furthermore, if a deviation is detected during task execution, it outputs state description information associated with that stage, thereby providing support for state monitoring and adaptive adjustment during task execution.

[0047] In some implementations, in step 12, in a physical simulation environment, simulation data corresponding to the sub-task phase is acquired, including: Perform domain randomization on the physical simulation environment, wherein the domain randomization includes at least one of the following: The types, quantities, and positions of objects in the physical simulation environment are randomly initialized; Randomly adjust the lighting conditions and / or background of the simulation environment in the physical simulation environment; The task logic of issuing the same embodied task to the physical simulation environment randomly generates multiple natural language instructions with the same semantics but different expressions.

[0048] For example, the types, quantities, and positions of objects in a physical simulation environment are randomly initialized—a process known as scene randomization. This can include increasing the diversity of object types and positions to improve the robustness of the recognition model to environmental changes. Specifically, during initialization, task-irrelevant distracting objects are randomly introduced into the workspace, and the positions on the desktop are randomly initialized. It's important to understand that setting a three-dimensional spatial boundary on the desktop signifies that the robotic arm's end effector can reach any point within that space, while ensuring that the initialized positions of objects on the desktop are within the boundary.

[0049] For example, in randomly adjusting the lighting conditions and / or background of a simulated environment in a physical simulation environment, random adjustment of lighting conditions can improve the robustness of the recognition model under different lighting conditions and shadow changes. Specifically, random adjustment of lighting conditions may include randomly adjusting at least one of the following: the number, direction, intensity, type of light, and color temperature of the light source. The type of light can be, for example, simulating natural light and indoor lighting. Thus, by randomly adjusting the number, direction, and intensity of the light sources, natural light and indoor lighting are simulated; and by randomly switching the color temperature of the light sources (from warm light to cool light), the diversity of visual features is enhanced.

[0050] For example, in randomly adjusting the lighting conditions and / or background of a simulated environment in a physical simulation environment, random background adjustment may include random adjustment of background color and / or background texture. Random background adjustment can reduce the fitting of the recognition model to a specific background, texture, or color scheme, and improve the robustness of the visual representation. Specifically, random background adjustment may include randomly changing the color and material texture maps of the desktop, walls, and floor, while perturbing the surface texture of objects (such as changes in brightness, contrast, and reflectivity).

[0051] For example, multiple semantically identical but differently expressed natural language instructions can be randomly generated for the same embodied task issued by a physical simulation environment. This diversification of task instructions enhances the recognition model's ability to generalize its understanding of task semantics, enabling the model to correctly execute the same task logic under different languages ​​or expressions. Specifically, given a task instruction, a large language model (such as ChatGPT or DeepSeek) can be invoked to generate semantically identical but differently expressed sentences. For example, input: "Please put the red square in the basket," output: "Put the small red square in the basket," and / or, "Please help me tidy up the red square and put it in the basket," and / or, "Move the red block into the basket."

[0052] In this way, the diversity and generalization ability of simulation data are enhanced, and the robustness of the recognition model in the real environment is improved. For example, by introducing a multi-dimensional domain randomization strategy on the MoJoCo simulation platform, namely the domain randomization processing mentioned above, visual, physical and semantic factors are randomly perturbed during the construction phase and task execution in the physical simulation environment. This exposes the recognition model to diverse data distributions during the training process, thereby reducing the gap between simulation and reality and improving the recognition accuracy of the recognition model.

[0053] In some implementations, step 11, dividing the embodied task into multiple sub-task stages of execution, includes: dividing the embodied task into multiple sub-task stages based on a predefined sequence of path key points; the motion process between adjacent key points corresponds to a sub-task stage. In some implementations, step 12, in the physical simulation phase, involves acquiring simulation data corresponding to the sub-task stage, including: planning the motion trajectory of the robotic arm's end effector in the sub-task stage based on the path key points corresponding to the sub-task stage; generating control commands to control the robotic arm to complete the sub-task stage based on the motion trajectory; executing the control commands in the physical simulation environment to complete the sub-task stage, and collecting image data corresponding to the sub-task stage during the execution of the control commands; and using the image data and the identification information of the sub-task stage as at least part of the simulation data.

[0054] The predefined sequence of key path points can be understood as a set of ordered spatial path points planned before task execution, based on the operational logic and geometric constraints of the embodied task. This sequence describes the ideal spatial path framework that the robotic arm's end effector should follow to complete the task.

[0055] Understandably, path critical points are the dividing points between stages of a task, used to guide the robotic arm to complete sub-tasks such as grasping, moving, and placing. For desktop operations, path critical points are mainly set based on the object's position, the desktop height, and the grasping strategy.

[0056] Specifically, in grasping tasks, the robotic arm path typically includes the following sequence of key points: Serial Number name illustrate P0 initial point Initial position of the robotic arm gripper P1 Pre-grab point Sampling is performed within a three-dimensional spatial range above the target object to ensure safe downward exploration. P2 Catch point Positioned above the center of the object or on the contact surface, ensuring symmetrical gripping by the grippers. P3 rise point Sample from the 3D space above the target object to prevent collisions. P4 Place on top (optional) Sampling from the three-dimensional space above the placement area P5 Placement point (optional) Sampling within the desktop range boundary P6 Evacuation point (optional) After placement, lift to a safe height. For the grabbing task, the corresponding path key points include P0 ~ P3; for the grabbing and placing task, the corresponding path key points include P0 ~ P6. When the scene is randomized, the target object placement point P5 is sampled from within the boundary of the desktop (excluding the area occupied by the object).

[0057] Specifically, when setting the gripping point, it can optionally be set to a position slightly above the center of the object; for some irregular objects, gripping points can be manually marked. For example, `pos_obj` defines the position of the gripping point, and the quaternion `quat_obj` defines the orientation (direction and rotation angle) of the gripping point. In practice, samples can be taken from candidate gripping points.

[0058] { "object_id": "cup_01", "grasps": [ { "name": "center_grasp_1", "pos_obj": [0.03, 0.0, 0.04], "quat_obj": [0, 0.707, 0, 0.707], "type": "center", }, { "name": "handle_grasp_1", "pos_obj": [0.035, 0.0, 0.06], "quat_obj": [0, 0.707, 0, 0.707], "type": "handle", } ] } It should be noted that the end effector of a robotic arm is typically equipped with a gripper. The end effector of a robotic arm may include the gripper of the robotic arm.

[0059] Thus, this embodiment of the application, by deeply integrating embodied task decomposition with physical simulation execution paths based on path key points, achieves the automated generation of high-quality, structured simulation data, laying a solid data foundation for training a recognition model with fine-grained task understanding capabilities. Specifically, this embodiment of the application uses objective and quantifiable path key points as the basis for stage division, ensuring the automation, accuracy, and consistency of task decomposition; by synchronously collecting data during the execution of control commands at each stage by driving the robotic arm, the simulation data and sub-task stages are aligned in time and semantics, reducing tedious post-annotation; the data generated based on the closed-loop simulation from planning to control to execution has high physical realism, and its visual observation contains real kinematic and dynamic logic, improving the data fidelity and potential for future transfer to reality; finally, the generated data is time-series image data organized by stages and with rich action context, providing dense supervision signals for the recognition model to learn and understand the causal relationship between visual patterns and task states within a specific stage, thereby supporting the recognition model to achieve fine-grained state recognition and reasoning capabilities beyond simple classification.

[0060] In some implementations, based on the key points corresponding to the sub-task phase, the motion trajectory of the robotic arm's end effector during the sub-task phase is planned, including: Based on the path key points corresponding to the sub-task stage, a smooth trajectory interpolation algorithm is used to generate a continuous pose trajectory of the robotic arm end effector between adjacent path key points. Based on the motion trajectory, control instructions are generated to control the robotic arm to complete the sub-task phase, including: Based on inverse kinematics, the pose trajectory of the robotic arm's end effector is converted into control commands for the robotic arm's joint space.

[0061] Specifically, to avoid jitter caused by abrupt changes in the movement of the robotic arm and discontinuous acceleration, a smooth trajectory interpolation algorithm (or fifth-order polynomial) is used between adjacent key points of the path to generate continuous and smooth pose (including position and attitude) trajectories.

[0062] Specifically, the trajectory is calculated using formula (1). : (1) in, It is the starting pose of the current trajectory segment. It is the final pose. This represents the current pose of the robotic arm's end effector, and T is the execution time of this trajectory segment. s = t / T represents the normalized time variable. The p(t) generated by this method will smoothly transition from the starting point to the ending point without sudden acceleration or abrupt stops.

[0063] Simultaneously, the time T for each trajectory segment can be automatically calculated based on the Euclidean distance between two points. Specifically:

[0064] in The desired average speed at the end of the robotic arm (e.g., 0.05 m / s).

[0065] Simultaneously, within each control step, the system calculates the pose increment based on the current pose and the target pose:

[0066] The current end effector pose is read using sensors. p t The trajectory generator generates the desired pose at that moment. p d (t) Calculate the pose increment Δ p t Then, we take it a step further in the simulation environment.

[0067] Understandably, during path execution, the robotic arm's end effector must simultaneously satisfy the target position and attitude constraints.

[0068] Specifically, by combining the Jacobian matrix of the simulation model and the damped least squares (DLS) solver, the joint angle solutions of the robotic arm corresponding to each key point are calculated in real time:

[0069] in: J: Terminal Jacobian matrix; λ: Damping factor, used to improve numerical stability.

[0070] Δp: Desired pose increment, including position increment (Δx, Δy, Δz) and attitude increment (four-dimensional Δθ). Δq (Joint Space Increment): The "small angle / displacement increment" that each joint needs to adjust to make the end effector closer to the desired pose. Δq = [Δq1, Δq2, ..., Δq n ].

[0071] The target pose point of the robotic arm end effector is obtained by trajectory generation. Then, the mapping from the desired pose to the joint space control quantity is realized by inverse kinematics solution. Finally, the joint is driven so that the robotic arm end effector reaches the target point.

[0072] It should be added that joint control commands can be generated by converting the calculated joint space increment Δq into specific joint control commands.

[0073] Thus, in this embodiment, by introducing a smooth trajectory interpolation algorithm and inverse kinematics solving, the generated data is not only automatically generated but also ensures the naturalness of motion, the precision of control, and the physical rationality of the generated data, thereby improving the effectiveness and reliability of subsequent recognition model training. Specifically, the use of a smooth trajectory interpolation algorithm (such as a fifth-order polynomial) ensures that the movement of the robotic arm between adjacent key points on the path remains continuous in terms of position, velocity, and acceleration, reducing abrupt changes, jitter, or unnatural sudden stops and starts. This not only makes the robotic arm's movements in the simulation environment closer to the smooth operation of the real world but also ensures that the acquired image data is visually coherent and natural, reducing noise and misleading information that may be introduced into the recognition model due to motion distortion. Simultaneously, based on inverse kinematics-based control command generation, the high-level end-effector pose trajectory is precisely mapped to the low-level key spatial increments (control variables). This process ensures that the entire linkage system of the robotic arm tracks the desired trajectory in a coordinated, energy-efficient manner while adhering to its kinematic constraints. The resulting control commands and corresponding visual data contain rigorous kinematic logic, making the action-to-state correlation in the data more physically interpretable. Therefore, through the above method, high-quality, low-noise training materials are provided for the recognition model to learn and establish a robust mapping between specific visual motion patterns and specific task stages and states. This helps the recognition model extract more essential and generalizable spatiotemporal features, thereby improving the accuracy and robustness of the recognition model in state recognition and reasoning on real robotic systems.

[0074] It should be noted that in existing technologies, robots not only need to complete given tasks, but also need to have the ability to detect failures and learn from them. For example, if a robot drops an object or makes a movement error while grasping it, and the failure cannot be identified in time, it cannot correct the mistake and learn improvement strategies. The lack of perception and in-depth analysis of failure situations limits the robot's practical application in complex environments. Therefore, how to improve the identification and reasoning regarding failure situations during task execution is a pressing technical problem that needs to be solved.

[0075] In some implementations, the simulation data includes samples of task execution failures, and the failure samples carry failure type labels based on a predefined failure type system. In step 13, based on the simulation data, sub-training samples are constructed for training the identification model, including: For simulation data that belongs to the task execution failure sample, a question-and-answer pair is generated based on the subtask stage in which the task execution failure sample is located and the corresponding failure type label. The question part of the question-and-answer pair is used to inquire about the execution status of the subtask stage, and the answer part includes the status judgment of the subtask stage execution failure and the reason description corresponding to the failure type label. By associating question-answer pairs, subtask stage identifiers, and corresponding image data, a subtask-level sample is formed.

[0076] In this context, failure samples refer to simulation data records generated when a robot fails to achieve the expected goal at a certain sub-task stage during the execution of an embodied task in a physical simulation environment (such as failing to grasp successfully, the object falling during movement, or grasping the wrong object). Failure samples can include complete simulation data from the beginning to the end of the task stage. This means that when the robot performs a grasping task in a real environment, for example, grasping a banana and moving it from point A to point B, even if the object falls in the middle, the robotic arm's end effector will still move to point B to ensure the integrity of the data samples.

[0077] In this context, a predefined failure type system refers to the systematic classification and definition of possible failure modes during task execution based on domain knowledge during the task design phase. For example, a predefined failure type system may include major categories such as grasping failure, object slippage, and incorrect target operation, as well as subcategories under these major categories.

[0078] The failure type label can be a specific category identifier in the predefined failure type system mentioned above. It is a machine-readable, discrete symbol (such as a string or enumeration value) used to uniquely and accurately identify the type to which a failure belongs.

[0079] In this system, failure type labels are abstract classifications used for efficient indexing and annotation. The cause description is a concrete, human-oriented natural language explanation, representing the specific failure type within the context of a particular task. The two have a one-to-one or one-to-many mapping relationship; for example, one failure type label corresponds to one or more standardized cause description templates (e.g., "the gripper did not contact the object," "the object slipped due to a smooth surface"). When generating an answer, the system can select or instantiate the most appropriate description template based on the failure type label to form the final cause description.

[0080] Understandably, failure is inevitable in real-world robotic task execution. Reasons for failure include missing the target, unstable grip, and object dropping. This failure data plays a crucial role in training the recognition model. Besides successful data, considering the diverse causes of failure, common types of failed tasks include, but are not limited to, at least one of the following: If the grasping fails, the corresponding reasons may include, but are not limited to, at least one of the following: the object is too heavy, the object is smooth (low surface friction), the gripper is not closed for some reason, or the gripper's grasping position is off from the target grasping position. The cause of an object slipping can be described as follows: when the gripper moves the object to the next position, the gripper becomes loose, causing the object to slip. An incorrect target object could be identified, and the corresponding reason could include a mismatch between the target object being grabbed and the task instructions.

[0081] For example, the task instruction is to grab a red apple, but the robotic arm grabs a red cube instead, causing the task to fail.

[0082] In this context, a question-answer pair can be a supervisory signal constructed for training a visual model, consisting of a question and an answer. The question guides the recognition model to perform a specific reasoning task (such as determining the state or explaining the cause) given visual input and task context. The answer provides the standard result or truth value for that reasoning task, serving as the learning target for the recognition model.

[0083] The execution status of a subtask can refer to the binary result of whether its preset goal has been achieved when the subtask stage ends. It is usually simplified to success or failure. For a failed sample, its execution status is failure.

[0084] The status determination for failure in a subtask phase and the corresponding reason description for the failure type label can be understood as both stating the conclusion of failure and providing a specific and explainable textual explanation of why it failed. This reason description may originate from the failure type label, but it is presented in richer, more human-comprehensible natural language, linking the abstract failure category with the specific visual context of the current phase.

[0085] Specifically, the question-and-answer pairs in the subtask phase can adopt the following structure: Question section: In the current subtask, the robot's current subtask is <Current Task>. For this subtask, please first select from ["Yes", "No"] to determine whether it was successful. If you select "No", please explain why it failed.

[0086] Example: In the current subtask, the robot is trying to grab the blue cup. For this subtask, please select from ["Yes", "No"] to determine whether it was successful. If you select "No", please explain why it failed.

[0087] Answer section (success or failure + reason) <Yes / No>, <Failure Type Label> The output of the answer in the failed case is, for example: No, the gripper failed to grab the blue cup.

[0088] Furthermore, taking the VLM model as an example of the recognition model, the recognition model is trained using sub-task-level samples, as follows: A complete example of training a VLM model is as follows: { "task_instruction": "Pick up the red cube and put it into the yellow bowl." "image_matrix": I, "current_subtask": "Grab the red square", "success": false, "question": "In the current subtask, the robot is grabbing the red cube." First, select from ['Yes', 'No'] to determine whether it was successful. If you select 'No', please explain why it failed. Answer: No, the object slipped out of the gripper during the lifting process. "failure_label": "object_slip", } In general, the input to VLM is a task description (the overall task and the current subtask question) and the keyframe trajectory image of the entire task. The model predicts a linguistic description of the reasons for success and failure for each subtask. This approach can improve the spatial understanding and causal reasoning capabilities of VLM (Visual Language Model) in embodied task scenarios.

[0089] Thus, in this embodiment, by utilizing simulated failure samples with predefined failure type labels and automatically constructing guided question-and-answer pairs, efficient and large-scale training of the fine-grained failure diagnosis capability of the recognition model is achieved. Specifically, the recognition model not only learns to judge whether a failure has occurred, but also generates natural language explanations of why a failure occurred through the learning of cause descriptions, making its output more operational and providing direct basis for the debugging and autonomous recovery of the robot system. Simultaneously, the question-and-answer pairs transform discrete failure type labels into structured language tasks, providing the recognition model with richer and more guided learning objectives than simple classification labels, forcing the recognition model to deeply understand the relationship between visual phenomena and failure semantics. Secondly, the predefined failure type system ensures that the training data covers key and common failure patterns, enabling the recognition model to systematically learn these patterns rather than random and scattered failure cases, improving the comprehensiveness and robustness of the recognition model's diagnosis. Finally, by combining limited failure samples (each failure type may have only a small amount of data) with rich speech descriptions, the learning value of each failure sample is amplified, helping the recognition model to better generalize to previously unseen scenarios with subtle variations belonging to the same failure type.

[0090] For example, if the simulation data includes task execution failure samples and the failure samples carry failure type labels based on a predefined failure type system, then in step 12, obtaining the simulation data corresponding to the sub-task stage in the physical simulation environment may include: adjusting the physical simulation environment based on the failure types labeled in the predefined failure type system, thereby obtaining the simulation data corresponding to the sub-task stage based on the adjusted physical simulation environment. For example, a predefined failure type system can include major categories such as grab failure, object slippage, and erroneous target operation, as well as subcategories under these major categories. Adjusting the physical simulation environment can include, but is not limited to: Adjust the physical simulation environment used to simulate failed grasping operations; Adjust the physical simulation environment used to create the illusion of objects sliding down; Adjust the physical simulation environment used to create incorrect target objects.

[0091] Specifically, adjusting the physical simulation environment used to create a grasping failure can include at least one of the following: Increase the weight of the object; Reduce the friction on the surface of an object; When the gripper reaches the target point, the gripper is set not to close. Modify the position of a given target object.

[0092] Specifically, adjusting the physical simulation environment used to create the illusion of an object sliding can include: When the gripper moves the object, the degree of closure of the gripper is changed.

[0093] Specifically, adjusting the physical simulation environment used to create erroneous target objects can include: The robotic arm is given a gripping point for another object.

[0094] In this way, by actively and controllably adjusting the parameters of the physical simulation environment to accurately generate predefined failure samples of various types, the recognition model is provided with systematic, diverse and physically realistic failure scenario data. This enables the recognition model to fully understand the internal mechanisms and visual representations of different failure modes, thereby improving the recognition model's diagnostic accuracy, attribution ability and generalization robustness in real tasks for complex failure states.

[0095] In some implementations, executing control commands in a physical simulation environment may include: Real-time contact detection is performed using a physical simulation engine based on a physical simulation environment. Contact information is generated in response to the detection that the end effector of the robotic arm is in contact with the target object; The robotic arm end effector is controlled based on contact information to execute control commands.

[0096] Real-time contact detection based on a physical simulation engine within a physical simulation environment refers to the continuous and rapid monitoring and identification of whether physical contact has occurred between specific pairs of objects during the physical simulation process. Here, it primarily refers to whether physical contact has occurred between the end effector of the robotic arm and the target object. In practical applications, contact detection mainly refers to detecting whether physical contact has occurred between the gripper of the robotic arm and the target object.

[0097] Contact information refers to a series of physical quantities that the physics simulation engine calculates and provides to describe the contact state when it detects that the end effector of the robotic arm is in contact or colliding with a target object. For example, contact information may include at least one of the following: contact distance, contact point location, contact normal vector, contact force, and contact identifier. The contact distance is used to determine the tightness of the contact, and the contact identifier is used to determine the specific object in contact.

[0098] Specifically, during the simulation execution, in order to achieve accurate contact perception and safe closing control of the robotic arm during, for example, the grasping phase, the system introduces a real-time contact detection mechanism based on, for example, the MuJoCo physics engine. The specific process is as follows: The MuJoCo engine automatically calculates the contact information between all geometries (specifically, the gripper of the robotic arm and the target object) in each simulation step; this contact data is stored in arrays such as data.contact. Each time the gripper closes and performs a grab, the code iterates through this array; When the geometry at the fingertips of the gripper (e.g., left_pad, right_pad) is detected to be in contact with the geometry of the target object, and the force sensor data is used to determine the force on the gripper tip, when the contact force reaches a set threshold, it is determined that "the target has been contacted". Upon detection of contact, the closing action is stopped in advance to prevent excessive squeezing by the grippers.

[0099] For example, controlling the end effector of a robotic arm to perform control commands based on contact information may include: Based on the contact force or contact distance information in the contact information, determine, for example, the contact state between the gripper and the target object, and switch or adjust the control mode based on the contact state, dynamically generate or correct the position, speed or force control commands of the gripper to complete the sub-task stage.

[0100] Specifically, this includes using contact information to trigger events (such as "effective contact"), thereby terminating preset motion and initiating new control logic (such as position holding or force control), thereby achieving closed-loop, refined control of the interactive behavior of the simulated robotic arm, and synchronously recording the control decision process and its results as part of the simulation data.

[0101] Thus, in this embodiment, by introducing the underlying real-time contact detection mechanism of the physical simulation engine into the control execution loop, high-fidelity, closed-loop simulation and control of the robotic arm's interactive actions are achieved. This not only makes the simulation of fine operations such as grasping closer to physical reality, but more importantly, it enables the system to dynamically generate or adjust control commands based on real-time physical interaction events. This provides a favorable foundation for the controllable and repeatable construction of various success and failure scenarios based on physical interaction. At the same time, this intelligent control based on contact perception lays a core foundation for generating simulation data with rich physical causal relationships and fine-grained state changes, improving the data quality and training effect for subsequent training of visual language models.

[0102] In some embodiments, the method further includes: During and / or after data collection, based on the contact information provided by the physical simulation engine, it is possible to detect any abnormalities between objects. If an anomaly is detected, it is determined whether the simulation data of the corresponding sub-task stage is abnormal, and the abnormal simulation data is filtered out, or the robotic arm is re-controlled to complete the sub-task stage.

[0103] Abnormal situations can include abnormal geometric overlap and / or physical situations that violate common sense (such as penetration). Abnormal geometric overlap includes abnormal partial geometric overlap and abnormal complete geometric overlap.

[0104] Anomalous geometric overlap refers to the intersection and intrusion of two or more objects in the 3D geometric models of a simulation, violating the laws of rigid body physics in the real world. Understandably, anomalous geometric overlap is usually caused by improper simulation initialization parameters, excessively large simulation step sizes, unstable numerical integration, or high-speed collisions resulting from control commands. Anomalous geometric overlap leads to severe distortion in the generated visual images (incorrect object positions) and state data (joint angles, object poses), violating common sense physics. Using such data to train a recognition model negatively impacts the model's recognition accuracy.

[0105] Specifically, the data generated in the simulation environment may contain some abnormal data, such as objects clipping through each other (geometric overlap of two objects) or physical violations of common sense (such as penetration). These problems will seriously reduce the physical realism of the training data, causing the model to learn unrealistic behaviors and affecting real transfer performance.

[0106] First, consider geometric overlap detection based on a physics simulation engine. MuJoCo maintains contact information for each simulation step. If the contact distance in the contact information is less than 0, it indicates penetration; if it is too small (e.g., less than -1e-3), it indicates a clipping anomaly. Thus, detection can be performed directly during simulation execution. If anomalies (e.g., clipping) are detected, the task can be re-executed after scene randomization, or this data can be filtered.

[0107] Thus, the implementation method of this application, by introducing an automatic anomaly geometry detection and data cleaning or regeneration mechanism based on the underlying physical simulation engine, constructs a crucial quality firewall for large-scale simulation data production. Specifically, the implementation method of this application can ensure the physical authenticity and rationality of the data, improve the robustness and automation of the data generation process, and guarantee the effectiveness and reliability of model training.

[0108] In some implementations, step 13, constructing sub-task-level samples for identification model training based on simulation data, further includes: constructing sub-task-level samples corresponding to each simulation data according to the simulation data corresponding to each sub-task stage, so as to finally construct an embodied intelligence dataset for the entire embodied task for identification model training.

[0109] Specifically, throughout the entire path execution process, the system collects and records multimodal data in real time, forming an embodied intelligence dataset. The multimodal data specifically includes the following: Image data, for example, may be a sequence of RGB frames or a sequence of depth maps captured by cameras with multiple viewpoints; State data, for example, may include, but is not limited to, time-series data such as the position, posture, joint angles, speed, and torque of the robotic arm end effector; Motion data, for example, may include control inputs (Δx, Δθ, grasp) for the robotic arm. Task semantic data, for example, may include, but is not limited to, task instructions, task type (grab / grab-place), target object ID, grab points (pos_obj, quat_obj), path key points, etc.

[0110] Identification information for sub-task stages, such as the label of the current task stage (approach, grasp, lift, place, etc.).

[0111] Success / failure labels and failure type labels, for example: Success: Yes / No; Failure type tags include, but are not limited to: grab failure, object slippage, and wrong target object.

[0112] Thus, this application's implementation systematically defines and collects multi-modal data covering visual, state, control, semantic, and result annotations, constructing a complete embodied intelligence dataset for the task. This provides a comprehensive, structurally unified, and highly aligned super-supervisory signal source for training the recognition model. This dataset not only enables the model to learn end-to-end mappings from raw images to high-level task semantics, but also provides unprecedentedly rich context for understanding the dynamic causal chain of task execution (e.g., how specific control inputs lead to specific state changes and visual results) and performing interpretable attribution reasoning (e.g., associating visual anomalies with specific failure types and physical parameters) through strictly aligned temporal states, precise control instructions, and fine-grained stage and failure labels. This significantly improves the model's learning efficiency, depth, and generalization ability, enabling it to extract more universal and transferable embodied intelligent cognitive models from simulation data.

[0113] In some implementations, step 11, which involves acquiring image data corresponding to the sub-task stage, includes: Simultaneously acquire image data from multiple different camera perspectives during sub-task phases; The method also includes: For a complete embodied task, extract image data corresponding to each perspective in each sub-task stage; Arrange image data from different camera perspectives along the first dimension within the same subtask phase; Image data from the same perspective under different sub-task stages are arranged along the second dimension to form a multi-view temporal image matrix; the multi-view temporal image matrix is ​​used to construct or as part of the simulation data.

[0114] Understandably, by breaking down embodied tasks into multiple sub-tasks, capturing keyframes for each sub-task and encoding temporal information through a multi-view matrix, and combining explicit failure type labels and question-answer pairs, the model can learn to understand task progress, detect failures, and explain the reasons for failures, thereby improving the robot's reasoning and self-correction capabilities in the real world.

[0115] A complete task consists of multiple subtasks {S0, S1, S2, ..., St}. For each subtask Si, two keyframes are taken from the image frame sequence: - Sistart: Startup state - Siend: The result status after the subtask has just been executed. For grasping tasks, this includes moving closer, grasping, and lifting; for grasping and placing tasks, this includes moving closer, grasping, moving to the placement point, and placing.

[0116] To enhance the model's temporal understanding, all keyframes preceding the current subtask are stitched together into an image sequence in chronological order. Simultaneously, to reduce occlusion and avoid misjudgments from a single viewpoint, while also improving the model's spatial understanding, multiple camera perspectives are incorporated, and the images from each perspective are stitched together into a matrix. As shown in the figure below, the image data is constructed as matrix I, where each row corresponds to a different camera viewpoint {V0, V1, ..., Vn}, and each column represents a different time series of keyframes {S0, S1, S2, ..., St}.

[0117]

[0118] In this embodiment, by structurally reorganizing multi-view temporal image data in matrix form, spatial (multi-view) and temporal (multi-stage) information during task execution is encoded into a compact and regular two-dimensional visual representation. This provides the recognition model with an efficient input form that simultaneously understands the three-dimensional spatial structure and dynamic temporal evolution of the task. This not only enhances the model's robustness to occlusion and viewpoint changes, as well as its ability to perceive the continuity of task stages, but also simplifies the model processing flow through a unified data structure, improving training and inference efficiency. This enables the model to more accurately learn and associate visual patterns at specific moments under specific viewpoints with the corresponding sub-task semantics and execution states.

[0119] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides a computing device, such as... Figure 4 As shown, the computing device includes: a processor 410 and a memory 411 storing computer programs; wherein, Figure 4 The processor 410 shown in the diagram does not indicate that there is only one processor 410, but only indicates the positional relationship of the processor 410 relative to other devices. In practical applications, there can be one or more processors 410; similarly, Figure 4 The memory 411 shown in the diagram has the same meaning, that is, it is only used to indicate the positional relationship of memory 411 relative to other devices. In practical applications, there can be one or more memories 411. When the processor 410 runs the computer program, the model optimization method described above is implemented.

[0120] The computing device may also include at least one network interface 412. The various components of the computing device are coupled together via a bus system 413. It is understood that the bus system 413 is used to implement communication between these components. In addition to a data bus, the bus system 413 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4 The general designated all buses as Bus System 413.

[0121] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program. The computer-readable storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc. When the computer program stored in the computer-readable storage medium is run by a processor, it implements the model optimization method described above. For the specific steps implemented when the computer program is executed by the processor, please refer to [link to relevant documentation]. Figure 1 The description of the illustrated embodiments will not be repeated here.

[0122] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A model optimization method, characterized in that, include: The embodied task is divided into multiple sub-task phases of execution; In a physical simulation environment, acquire simulation data corresponding to the sub-task stage; Based on the simulation data, sub-task-level samples for identification model training are constructed; The recognition model is trained using the sub-task-level samples.

2. The method according to claim 1, characterized in that, The sub-task phases that divide the embodied task into multiple execution phases include: The embodied task is divided into multiple sub-task stages based on a predefined sequence of path key points; the movement process between adjacent path key points constitutes one of the sub-task stages. In the physical simulation environment, acquiring simulation data corresponding to the sub-task stage includes: Based on the path key points corresponding to the sub-task stage, plan the motion trajectory of the robotic arm end effector in the sub-task stage to execute the embodied task. Based on the motion trajectory, control instructions are generated to control the robotic arm to complete the sub-task stage; The control instructions are executed in the physical simulation environment to complete the sub-task phase, and image data corresponding to the sub-task phase is acquired during the execution of the control instructions. The image data and the identification information of the sub-task stage are used as at least a part of the simulation data.

3. The method according to claim 2, characterized in that, The step of planning the motion trajectory of the robotic arm's end effector during the sub-task phase based on the path key points corresponding to the sub-task phase includes: Based on the path key points corresponding to the sub-task stage, a smooth trajectory interpolation algorithm is used to generate a continuous pose trajectory of the robotic arm end effector between adjacent path key points. The step of generating control instructions for the robotic arm to complete the sub-task stage based on the motion trajectory includes: Based on inverse kinematics, the pose trajectory of the end effector of the robotic arm is converted into control commands for the joint space of the robotic arm.

4. The method according to claim 2, characterized in that, The simulation data includes task execution failure samples, and the failure samples carry failure type labels based on a predefined failure type system. The step of constructing sub-task-level samples for identification model training based on the simulation data includes: For simulation data belonging to the failed task samples, a question-and-answer pair is generated based on the sub-task stage where the failed task sample is located and the corresponding failure type label. The question part of the question-and-answer pair is used to inquire about the execution status of the sub-task stage, and the answer part includes the status judgment of the failed execution of the sub-task stage and the reason description corresponding to the failure type label. The question-and-answer pairs, the identification information of the sub-task stage, and the corresponding image data are associated to form a sub-task-level sample.

5. The method according to claim 2, characterized in that, Executing the control commands in the physical simulation environment includes: Real-time contact detection is performed using a physical simulation engine based on the physical simulation environment. Contact information is generated in response to the detection that the end effector of the robotic arm comes into contact with the target object; The robotic arm end effector is controlled based on the contact information to execute the control commands.

6. The method according to claim 5, characterized in that, The method further includes: During and / or after data collection, based on the contact information provided by the physical simulation engine, it is detected whether there are any abnormal situations between objects. If the abnormal situation is detected, the simulation data of the corresponding sub-task stage is determined to be abnormal, and the abnormal simulation data is filtered out. Alternatively, the corresponding control command is regenerated to re-control the robotic arm to complete the sub-task stage.

7. The method according to claim 2, characterized in that, The acquisition of image data corresponding to the sub-task stage includes: Image data for the sub-task phase is acquired simultaneously from multiple different camera perspectives; The method further includes: For a complete embodied task, extract the image data corresponding to each perspective in each sub-task stage; The image data from different camera perspectives within the same subtask phase are arranged along the first dimension; The image data from the same perspective under different sub-task stages are arranged along the second dimension to form a multi-view temporal image matrix; the multi-view temporal image matrix is ​​used to construct or as part of the simulation data.

8. The method according to claim 1, characterized in that, In the physical simulation environment, acquiring simulation data corresponding to the sub-task stage includes: The physical simulation environment is subjected to domain randomization, which includes at least one of the following: The types, quantities, and positions of objects in the scene of the physical simulation environment are randomly initialized; The lighting conditions and / or background of the simulation environment in the physical simulation environment are randomly adjusted. The task logic of the same embodied task issued to the physical simulation environment randomly generates multiple natural language instructions with the same semantics but different expressions.

9. A computing device, characterized in that, include: A processor and a memory storing a computer program, wherein, when the processor runs the computer program, the model optimization method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the model optimization method according to any one of claims 1 to 8.

Images