Data processing method and device, limiting piece and computer readable storage medium

Abstract

This application relates to a data processing method, apparatus, limiting member, and computer-readable storage medium. The method includes: acquiring first data generated by each data acquisition terminal when performing a target task; acquiring second data generated by a robot using each manipulator to perform the target task; and determining mixed data based on the first and second data, the mixed data being usable for training a robot control model; wherein, when each data acquisition terminal and manipulator begins performing the target task, the spacing between each data acquisition terminal matches the spacing between each manipulator; and when each data acquisition terminal and manipulator stops performing the target task, the spacing between each data acquisition terminal matches the spacing between each manipulator. This method ensures the accuracy of the first data, and the mixed data is sufficient to ensure the accuracy of model training.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] As embodied intelligence models iterate and upgrade, they rely heavily on large-scale, high-fidelity robot operation demonstration datasets. Current data acquisition methods are mainly divided into two types: one is physical robot acquisition, which uses master-slave control and other methods to collect and record data, but its acquisition scenarios are limited, resulting in high overall investment and insufficient output efficiency; the other is robot-independent acquisition, using data acquisition terminals such as the UMI portable acquisition system. This method can complete motion data acquisition without relying on a physical robot, possessing significant advantages such as low cost and mass production capability, but lacks relevant data information about the robot itself.

[0003] In the field of robot training, if data generated by a data acquisition terminal and data actually collected by the robot are directly mixed for robot training, the accuracy of training using mixed data will be poor because there are significant differences between the data acquired by the data acquisition terminal and the relevant kinematic data. Summary of the Invention

[0004] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of mixed data training in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a data processing method, the method comprising:

[0006] Acquire the first data generated by each data acquisition terminal when it executes the target task;

[0007] Acquire second data generated by the robot when it uses each robotic arm to perform the target task; and

[0008] Based on the first data and the second data, mixed data is determined, which can be used to train the control model of the robot;

[0009] Specifically, when each of the data acquisition terminals and the robotic arm begins to execute the target task, the distance between each of the data acquisition terminals matches the distance between each of the robotic arms; when each of the data acquisition terminals and the robotic arm stops executing the target task, the distance between each of the data acquisition terminals matches the distance between each of the robotic arms.

[0010] In one embodiment, the first data includes the motion trajectory information of the data acquisition terminal and the environmental data collected by the data acquisition terminal, and the second data includes the motion trajectory information of the robotic arm and the environmental data collected by the robotic arm.

[0011] In one embodiment, the spacing between the robotic arms includes the distance between the centers of the robotic arms; and / or, the spacing between the data acquisition terminals matches the spacing between the robotic arms, meaning that the spacing between the corresponding data acquisition terminals is the same as the spacing between the robotic arms.

[0012] In one embodiment, when the target task is started, each data acquisition terminal is positioned by a limiting part of a limiting member; when the target task is stopped, each data acquisition terminal is positioned by a limiting part of the limiting member; the distance between each limiting part matches the spacing between each robotic arm when the robot starts executing the target task; the spacing between each robotic arm when the robot starts executing the target task matches the spacing between each robotic arm when the robot stops executing the target task.

[0013] In one embodiment, determining the mixed data based on the first data and the second data includes,

[0014] The first data and the second data are used to determine mixed data based on a preset proportional distribution;

[0015] The preset ratio includes the ratio of the amount of data between the first data and the second data.

[0016] In one embodiment, the preset ratio is between a first ratio and a second ratio; the first ratio includes a ratio of 1:1 between the first data and the second data; the second ratio includes a ratio of 10:1 between the first data and the second data.

[0017] In one embodiment, the method further includes:

[0018] The preset ratio is obtained based on the task complexity of the target task;

[0019] The complexity of the task is negatively correlated with the preset ratio.

[0020] In one embodiment, the task complexity includes task duration, and the task duration is positively correlated with the task complexity.

[0021] In one embodiment, the task complexity includes the rigidity of the target object of the target task, and the rigidity of the target object is negatively correlated with the task complexity.

[0022] In one embodiment, the method further includes:

[0023] The preset ratio is obtained based on the robot's rated load;

[0024] The rated load is negatively correlated with the preset ratio.

[0025] In one embodiment, the method further includes:

[0026] The preset ratio is obtained based on the robot's rated load and the task complexity of the target task;

[0027] The task complexity and the rated load are both negatively correlated with the preset ratio, and the degree to which the task complexity affects the preset ratio is greater than the degree to which the rated load affects the preset ratio.

[0028] Secondly, this application also provides a limiting member, including:

[0029] Load-bearing components;

[0030] Each limiting part is disposed on the carrier member; each limiting part includes at least one of a first limiting part and a second limiting part, wherein the first limiting part and the second limiting part are the same or different limiting parts;

[0031] The distance between each of the first limiting parts is matched with the spacing between each manipulator when the robot starts to perform the target task. When each data acquisition terminal starts to perform the target task, each of the data acquisition terminals is positioned by each of the first limiting parts.

[0032] The distance between each of the second limiting parts is matched with the spacing between each manipulator when the robot stops performing the target task. When each data acquisition terminal stops performing the target task, each data acquisition terminal is positioned by each of the second limiting parts.

[0033] In one embodiment, the carrier includes a first connector and at least two second connectors;

[0034] Wherein, a first end of the first connector is detachably connected to one of the second connectors, and a second end of the first connector is detachably connected to another of the second connectors, wherein the first end and the second end include the two opposite ends of the first connector; or

[0035] Wherein, the first end of the first connector is foldably connected to one of the second connectors, and the second end of the first connector is foldably connected to another of the second connectors, wherein the first end and the second end include the two ends opposite to the first connector.

[0036] Thirdly, this application also provides a data processing apparatus, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 11.

[0037] Fourthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data processing steps in any of the above embodiments.

[0038] Fifthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the data processing steps in any of the above embodiments.

[0039] Sixthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the data processing steps in any of the above embodiments.

[0040] The aforementioned data processing methods, apparatus, computer equipment, computer-readable storage media, and computer program products involve a start-up phase and a stop-down phase in the execution process of the target task. Both phases control the spacing between the data acquisition terminals to match the spacing between the robotic arms, ensuring that the start and end points of the motion trajectories of the data acquisition terminals are highly consistent with the motion trajectories of the robotic arms, conforming to morphological constraints and improving the success rate of subsequent inverse kinematic mapping. Under these circumstances, the accuracy of the first data generated by the data acquisition terminals is relatively high, which can enhance the training effect of mixed data, help ensure training effect with less data, and ensure the efficiency and accuracy of model training. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1This is an application environment diagram of a data processing method in one embodiment;

[0043] Figure 2 This is a flowchart illustrating a data processing method in one embodiment;

[0044] Figure 3 This is a schematic diagram of the limiting member in one embodiment;

[0045] Figure 4 This is a structural block diagram of a data processing device in one embodiment;

[0046] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various types of data, but these data are not limited by these terms. These terms are only used to distinguish between the first type of data and the second type of data. The term "comprising" and any variations thereof, as used in this application, are intended to cover non-exclusive inclusion. The term "multiple" as used in this application refers to two or more. The term "and / or" as used in this application refers to one of the solutions, or any combination of multiple solutions.

[0049] In a traditional approach, data generated by data acquisition terminals and data collected by the actual robot are collected separately. These two types of data are then input into a vision-language-action (VLA) model for training in an open-loop, unsystematic (random or blindly mixed) manner. However, the morphological gap between humans and robots can lead to lower data accuracy: without body-based data collection, the initial positions of a person's hands are arbitrary. Direct data collection may result in a mismatch between the distances between the data acquisition terminals and the distances between the robot's manipulators when starting or stopping the target task. This could cause the data collected by the data acquisition terminals to fail in subsequent inverse kinematics (IK) calculations or lead to the risk of self-collision by the target robot.

[0050] The data processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the data acquisition terminal 102 communicates with the server 104 via a network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed in the cloud or on other network servers.

[0051] The data acquisition terminal 102 can be, but is not limited to, various smartphones and portable wearable devices. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Optionally, the data acquisition terminal includes, but is not limited to, smart glasses for collecting environmental data, handheld handles for collecting motion data, handheld data acquisition grippers that can be used in combination with a camera and / or have an IMU, motion capture gloves for capturing subtle finger movements and tactile sensations, and electromyography (EMG) bracelets for measuring electromyographic signals and wrist posture. It is understood that when the data acquisition terminal 102 is a handheld data acquisition gripper, in some embodiments, the data acquisition terminal 102 is equipped with a camera capable of collecting motion data from the handheld data acquisition gripper; in other embodiments, the data acquisition terminal 102 can be equipped with an electronic device with a camera, such as a smartphone, so that the electronic device with the camera can collect motion data from the handheld data acquisition gripper. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. This method can be executed independently by the processor of data acquisition terminal 102, or it can be executed during the interaction between server 104 and data acquisition terminal 102.

[0052] In one exemplary embodiment, such as Figure 2 As shown, the first aspect of this application provides a data processing method, which is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 202 to 206. Wherein:

[0053] Step 202: Obtain the first data generated by each data acquisition terminal when executing the target task.

[0054] A data acquisition terminal is a bodyless data acquisition device corresponding to the robot's manipulator; it is used by a wearable or handheld device to record the first data related to the model's motion in the target task execution environment without relying on the robot's manipulator.

[0055] The target task is a specific task performed during data collection. The training data for the target task includes first data and second data. The first data is data collected through a data acquisition terminal, and the second data is data collected through the robot itself. It can be understood that the data collected through the data acquisition terminal does not include the robot's body data. The robot's body includes at least one of the following: a robotic arm capable of moving the robot's manipulator, the robot's torso, and the robot's lower limbs. Optionally, the target task is not limited to tasks such as clothing folding, parts sorting, and product assembly.

[0056] The first data includes at least motion data from the data acquisition terminal, which is used to train the robot's manipulator to perform specific actions and clarify the manipulator's trajectory. The first data may also include images generated during the execution of the target task, used to clarify the scene during image acquisition and thus the corresponding interactive data. The first data refers to data that is not dependent on the robot's actual physical manipulator, but is directly demonstrated and recorded by a human operator in a natural environment using a handheld sensing device. It is understood that in some embodiments, the first data can be multimodal data collected by multiple sensors.

[0057] In some embodiments, the number of handheld handles corresponding to the number of robotic arms records the folding task performed by the grippers corresponding to the handheld handles. In this case, after transmitting video data through the cameras of the grippers, the motion trajectory sensor of the handheld handle can be used to control the grippers corresponding to the handheld handles to move to the side of the material and close to clamp the clothing, simultaneously recording the clamping force and the motion trajectory of the folding process. Therefore, when the complexity of the target task is high, not only environmental data and motion trajectory can be used as the first data, but force data can also be included in the first data.

[0058] Step 204: Obtain the second data generated by the robot when using each robotic arm to perform the target task.

[0059] The robot has multiple robotic arms and supports a training process for using these arms to perform target tasks. The second data refers to the data generated during the robot's execution of the target task. When the robotic arms perform the target task, they may include, but are not limited to, grippers, magnetic jigs, bionic fingers, or robotic arms used for grinding, spraying, or cleaning. The second data refers to data collected by the robot's actual robotic arm processor, including underlying physical priors such as actual motor friction and control latency; for example, the second data corresponding to the robotic arm collected through master-slave device control of the robot.

[0060] Optionally, a teach pendant can be used to control the movement of the robotic arm, and the robot itself can record the movement trajectory of its robotic arm and the actions of its grippers. Alternatively, the robot itself can directly execute the target task, and the robot's sensors can collect successful or failed data on the execution of the target task to form secondary data. For example, data collected by controlling the robot through traditional master-slave devices or VR devices includes the target robot's kinematic priors, which include conditions such as physical constraints and friction. Therefore, the kinematic priors match the robot's actual operating scenario and have high accuracy. However, due to physical space limitations, the data collection cost is high and the efficiency is extremely low.

[0061] The first data and the second data are data generated when the same target task is performed in different ways. The first data is acquired by the data acquisition terminal, while the second data is acquired by the robot itself. The data acquisition terminal and the robot are two different devices. Therefore, the first data is ontology-free data from the robot's training process, while the second data is ontology-based data from the robot's training process. The cost of acquiring the first data is lower than the cost of acquiring the second data.

[0062] In some embodiments, the target task is to pick up a cup. At this time, a cup is picked up by the gripper of the data acquisition terminal, and the data generated in this process is collected as the first data. Then, the cup is picked up by the robot's robotic arm, and the data generated in this process is collected as the second data.

[0063] Step 206: Determine mixed data based on the first data and the second data. The mixed data can be used to train the robot's control model.

[0064] Specifically, when each data acquisition terminal and the robotic arm begin to perform the target task, the spacing between each data acquisition terminal matches the spacing between each robotic arm; when each data acquisition terminal and the robotic arm stop performing the target task, the spacing between each data acquisition terminal matches the spacing between each robotic arm.

[0065] The hybrid data includes first data and second data. Data cleaning can be performed on the first and second data, and the cleaned data can be mixed to obtain the corresponding hybrid result. Thus, the process of combining the two types of data in terms of format or weight ensures the model training effect. In the hybrid data, the first data is used to construct general semantic cognition, and the second data serves as kinematic anchor points. Therefore, the accuracy of training using hybrid data is approximately the same as that of training using only the second data, and because the hybrid data also uses the first data, its data acquisition cost is relatively low. This overcomes the morphological gap, which refers to the natural spatial and dimensional differences between the limb structure of a human operator (such as arm span and joint degrees of freedom) and the physical mechanical structure of the target robot. It is understood that in some embodiments, the control model may include a neural network model applied to the robot processor or controller.

[0066] The spacing between data acquisition terminals represents the relative distance between different data acquisition terminals. It can be the distance between the centers of gravity of each data acquisition terminal, the clearance distance between any two adjacent data acquisition terminals, or the distance between the centers of each data acquisition terminal. Similarly, the spacing between robotic arms represents the relative distance between different robotic arms. It can be the distance between the centers of gravity of each robotic arm, the clearance distance between any two adjacent robotic arms, or the distance between the centers of each robotic arm. It is understood that in some embodiments, when the spacing between data acquisition terminals is the distance between their respective centers of gravity, the spacing between robotic arms is the distance between the centers of gravity of each robotic arm; in other embodiments, when the spacing between data acquisition terminals is the clearance distance between them, the spacing between robotic arms is the clearance distance between each robotic arm.

[0067] In one example, the robot includes an adjacent first manipulator and a second manipulator, and each data acquisition terminal includes a first data acquisition terminal and a second data acquisition terminal. The first data acquisition terminal is used to provide training data for the first manipulator, and the second data acquisition terminal is used to provide training data for the second manipulator. In this case, the spacing between each data acquisition terminal matches the spacing between each manipulator, including: the clearance distance between the first data acquisition terminal and the second data acquisition terminal matches the clearance distance between the first manipulator and the second manipulator; or the distance between the center of the first data acquisition terminal and the center of the second data acquisition terminal matches the distance between the center of the first manipulator and the center of the second manipulator; or the distance between the center of gravity of the first data acquisition terminal and the center of gravity of the second data acquisition terminal matches the distance between the center of gravity of the first manipulator and the center of gravity of the second manipulator.

[0068] In some embodiments, first data under different trajectories and angles, and second data in a standard format recorded by the robot's manipulator, can be mixed according to different proportions or weights to obtain mixed data.

[0069] In some embodiments, when each data acquisition terminal and the robotic arm begin to execute the target task, each data acquisition terminal is controlled to be at a preset coordinate position so that the distance between each data acquisition terminal matches the distance between each robotic arm; when each data acquisition terminal and the robotic arm stop executing the target task, each data acquisition terminal is controlled to be at a preset coordinate position so that the distance between each data acquisition terminal matches the distance between each robotic arm. Without body-based data collection, the initial position of a person's hands is arbitrary. If data is collected directly, it may cause a mismatch between the distance between each data acquisition terminal and the distance between each robotic arm of the target robot when the target task begins or stops. This could lead to the data collected by the data acquisition terminals failing in subsequent inverse kinematics (IK) calculations or causing the target robot to risk self-collision.

[0070] In the above data processing method, the execution process of the target task includes a start execution phase and a stop execution phase. Both execution phases control the spacing between each data acquisition terminal to match the spacing between each robotic arm, which can ensure that the start and end points of the motion trajectory are highly consistent with the motion trajectory of the robotic arm, conforming to morphological constraints, and improving the success rate and physical realism of subsequent inverse kinematic mapping. Under these circumstances, the accuracy of the first data generated by the data acquisition terminal is relatively high, which can enhance the training effect of mixed data, help ensure the training effect with less data, and ensure the efficiency and accuracy of model training.

[0071] In some embodiments, the first data includes motion trajectory information of the data acquisition terminal and environmental data acquired by the data acquisition terminal, and the second data includes motion trajectory information of the robot and environmental data acquired by the robot.

[0072] Motion trajectory information is the motion path when performing a target task, used to train the robot to complete the prior target task. Motion trajectory information can be a pose trajectory with multiple degrees of freedom, such as 6 degrees of freedom; and the pose of each point on the motion trajectory can include information in two dimensions: translation (x, y, z) and rotation (roll, pitch, yaw).

[0073] The motion trajectory information of the data acquisition terminal is the motion trajectory of the control device without using the robot body. For example, when using a handheld data acquisition gripper to perform a target task, the motion trajectory information of the data acquisition terminal is the motion trajectory of the handheld data acquisition gripper; when using a glove to perform a target task, the motion trajectory information of the data acquisition terminal can be the motion trajectory of the glove.

[0074] The motion trajectory information of a robotic hand refers to the motion trajectory of the robotic hand itself when it is driven by a robot. For example, if the robotic hand includes a robotic gripper and the target task is performed by the robotic gripper, the motion trajectory information of the robotic gripper includes the motion trajectory of the robotic gripper; if the robotic hand includes a robotic suction cup and the target task is performed using a bionic finger, the motion trajectory information of the robotic hand may include the motion trajectory of the bionic finger.

[0075] The environmental data includes at least the objects to be processed when performing the target task; the objects to be processed can be necessities required to perform the target task, for example, if the target task is to fold clothes, the objects to be processed can be clothes; if the target task is to pick up a water cup, the objects to be processed can be a water cup.

[0076] Optionally, when environmental data is available, data can be acquired through visual-inertial odometry (VIO) to optimize the motion trajectory using this branch of pure visual SLAM technology; for example, based on visual-inertial odometry, visual image data from a fusion camera and motion data from an IMU (inertial measurement unit) can be fused to more accurately calculate the precise three-dimensional spatial motion trajectory of the device.

[0077] Optionally, the environmental data may also include obstacle data when performing the target task; the obstacle data is products that may affect the performance of the target task. For example, when the target task is to fold clothes, the obstacle data may be a clothes hanger to avoid touching the clothes hanger during the folding process; when the target task is to pick up a water cup, the obstacle data may be a bowl next to the water cup to avoid touching the bowl during the picking up of the water cup.

[0078] Taking a user picking up two handheld data acquisition grippers and starting to fold clothes using them as an example: At the initial moment, the distance between the two data acquisition grippers is controlled by a limiting part or software used for limiting, matching the distance between the robot's two robotic arms when starting to fold the clothes. At this time, the user collects starting point data generated when starting to fold the clothes using the two data acquisition grippers. The starting point data includes the position of the starting point and environmental data, and this starting point data belongs to the first data. During the process of the user moving the clothes with the two data acquisition grippers to fold the clothes, folding process data is generated. The folding process data includes the trajectory data of the clothes moved by the two data acquisition grippers and environmental data; this folding process data belongs to the first data. At the end moment when the folding of the clothes is completed, the distance between the two data acquisition grippers is controlled by a limiting part or software used for limiting, matching the distance between the robot's two robotic arms when starting and ending the folding of the clothes. At this time, the user collects ending point data generated when starting to fold the clothes using the two data acquisition grippers. The ending point data includes the position of the ending point and environmental data, and this ending point data belongs to the first data. Therefore, the first data includes starting point data, folding process data, and ending point data, and the user data acquisition gripper completes the first data acquisition process.

[0079] In this embodiment, the data used for training includes motion trajectory information and environmental data. The motion trajectory information can represent the specific action details of the task execution, so that the action dimensions of the first data and the second data are matched, which is convenient for subsequent action analysis. The environmental data reflects the target object of the robot's target task, so it is also included as data to be collected, which can form a comparison relationship between action and environment, and help ensure the accuracy of training.

[0080] In some embodiments, the spacing between the robotic arms includes the distance between the centers of the robotic arms; and / or, the spacing between the data acquisition terminals matches the spacing between the robotic arms, including that the spacing between the corresponding data acquisition terminals is the same as the spacing between the robotic arms.

[0081] The distance between centers refers to the straight-line distance between the geometric center points of each robot arm, representing the distance between the robot arms when performing the target task. For example, when starting to perform the target task, the distance between the centers of the robot arms is a preset distance; when finishing to complete the target task, the distance between the centers is the same preset distance, to ensure that the spacing between the robot arms is appropriate.

[0082] Optionally, the spacing between each data acquisition terminal is set according to the spacing between each robotic arm. The spacing between each data acquisition terminal can be the distance between the centers of each data acquisition terminal. Optionally, each data acquisition terminal has a positioning point, which corresponds to the center of the robotic arm. The distance between each robotic arm can be reflected by the distance between these positioning points. These positioning points can be the center point of each data acquisition terminal or the end point of each data acquisition terminal, so as to ensure the convenience of operation through these end points.

[0083] A correspondence means that the data collected by the data acquisition terminal is applied to the corresponding robotic arm to train the corresponding data. Optionally, there is a one-to-one correspondence between the data acquisition terminal and the robotic arm, and the distance between any two data acquisition terminals is the same as the distance between any two robotic arms.

[0084] For example, the robot includes an adjacent first manipulator and a second manipulator, and each data acquisition terminal includes a first data acquisition terminal and a second data acquisition terminal. The first data acquisition terminal is used to provide training data for the first manipulator, and the second data acquisition terminal is used to provide training data for the second manipulator. At this time, the first data acquisition terminal and the first manipulator have a corresponding relationship, and the second data acquisition terminal and the second manipulator have a corresponding relationship. When each data acquisition terminal and manipulator starts to execute the target task, the distance between the first data acquisition terminal and the second data acquisition terminal is the same as the distance between the first manipulator and the second manipulator. When each data acquisition terminal and manipulator finishes executing the target task, the distance between the first data acquisition terminal and the second data acquisition terminal is the same as the distance between the first manipulator and the second manipulator.

[0085] In this embodiment, the relative positional relationship between the robotic arms is refined into the positional relationship of the center point or the central area by the distance between the centers of each robotic arm, ensuring that the reference objects of each robotic arm are consistent. At the beginning and end of the target task, the spacing between each data acquisition terminal and each robotic arm with corresponding relationship is ensured to be the same as the spacing between each robotic arm, so that the spacing between each data acquisition terminal and the spacing between each robotic arm are more matched, thereby further improving the success rate of subsequent inverse kinematic mapping and ensuring the efficiency and accuracy of model training.

[0086] In some embodiments, when the target task is started, each data acquisition terminal is positioned by each limiting part of the limiting member; when the target task is stopped, each data acquisition terminal is positioned by each limiting part of the limiting member; the distance between each limiting part matches the distance between each manipulator when the robot starts to execute the target task; the distance between each manipulator when the robot starts to execute the target task matches the distance between each manipulator when the robot stops to execute the target task.

[0087] A limiting element is a component used to limit distance; the limiting element has multiple limiting parts, which are set according to a preset distance to ensure that the spacing between each robot arm matches during the target task. Optionally, the limiting element can be a component or assembly used for limiting.

[0088] The limiting part is a limiting component of the limiting member. The limiting part is set on the limiting member to ensure that the spacing between each data acquisition terminal matches the spacing between the robotic arms by means of the distance between each limiting part. Optionally, the limiting part is a slot, and the distance between two slots is equal to the baseline distance of the gripper arms of this model of robot at the start or end of the target task.

[0089] In some embodiments, the limiting component may include a bracket body, and the limiting part is a sub-bracket on the bracket specifically used to hold or support the data acquisition terminal, so as to realize the positioning function of the data acquisition terminal through the sub-bracket. The sub-bracket may also be a groove or a bent surface. The limiting component may include a carrier composed of multiple connectors, and the connectors are detachably or foldably connected. Each connector is provided with a corresponding limiting part, so as to realize the positioning function of the data acquisition terminal through the limiting part.

[0090] In this embodiment, each data acquisition terminal is positioned by the limiting part of the limiting component. Since the spacing of the limiting part is fixed, even without body constraints, whether at the beginning or end of the target task, the spacing between each data acquisition terminal can be matched with, or even completely matched with, the spacing between each robot arm. This ensures that the acquired first data is more consistent with the kinematic prior and that the first data matches the second data to a high degree. Thus, the errors caused by operator intuition or pure software estimation are eliminated. Therefore, at the beginning and end of data acquisition, the spacing between the data acquisition terminals and the spacing between the robot arms can achieve millimeter-level consistency, resulting in an extremely high success rate and effectiveness in solving the inverse kinematics of the first trajectory.

[0091] In some related technologies, users randomly mix first and second sets of data for training the robot's control model. However, random mixing leads to poor training results or even systemic failure: First, there is a lack of systematic empirical guidelines on the optimal combination of these two types of data. If users randomly mix data, especially when the data ratio is extremely skewed, the model will experience a catastrophic distribution shift during training, making it unable to simultaneously grasp environmental cognition without ontology data and the physical accuracy of real machine data, ultimately reducing policy convergence performance.

[0092] In some embodiments, determining mixed data based on first data and second data includes: determining mixed data based on a preset ratio distribution of first data and second data; wherein the preset ratio includes the ratio of the amount of data between the first data and the second data.

[0093] The preset ratio represents the proportion of the first data point relative to the second data point. The preset ratio distribution can be used to control the proportion of the two types of data in the mixed data, so as to balance the needs of understanding environmental data and the accuracy requirements of the robot's actual machine data.

[0094] In one embodiment, a preset ratio is determined according to the task type of the target task; the task type can be divided according to different task complexity. Therefore, for target tasks of different task types, the preset ratio is adaptively adjusted to adaptively control the distribution of the mixed data through the proportion of the two types of data.

[0095] In this embodiment, the first data and the second data are combined into mixed data according to the ratio of data volume to avoid the problem of data ratio skew in the random mixing process and to avoid the model experiencing catastrophic distribution shift during training. Therefore, by allocating the first data and the second data in a preset ratio, the model can ensure environmental cognition through mixed data and ensure the accuracy of the robot's real machine data, thus ensuring better policy convergence effect.

[0096] In some embodiments, the preset ratio is between a first ratio and a second ratio; the first ratio includes a 1:1 ratio of the first data to the second data; the second ratio includes a 10:1 ratio of the first data to the second data.

[0097] The first ratio is the ratio of the amount of data when the proportion of the first data in the mixed data is the smallest. When the preset ratio corresponding to the target task is the first ratio, the first data and the second data in the mixed data have the same proportion. This task scenario requires more second data as kinematic anchor points to ensure that the mixed data is closer to the real device data.

[0098] The second ratio is the ratio of the amount of data when the first data accounts for the largest proportion of the mixed data. When the preset ratio corresponding to the target task is the second ratio, the first data in the mixed data is much larger than the second data. This task scenario requires more first data for environmental cognition. Less second data can be used as kinematic anchor points, which can obtain the first data at a lower cost and ensure that the mixed data is closer to the real device data.

[0099] In this embodiment, the range of the preset ratio is controlled by the first ratio and the second ratio. No matter which ratio the preset ratio is between the first ratio and the second ratio, there will be no problem of extreme data skew. Therefore, the first data ensures that the mixed data has sufficient environmental awareness, and the second data ensures that the mixed data has the physical accuracy of real machine data, thus ensuring a better strategy convergence effect.

[0100] In some embodiments, the method further includes: obtaining a preset ratio based on the task complexity of the target task; wherein the task complexity is negatively correlated with the preset ratio.

[0101] Task complexity is a quantified result of the difficulty of the target task. By quantifying the difficulty of the robot handling the target task through task complexity, a preset ratio can be dynamically adjusted. Optionally, the greater the task complexity, the smaller the preset ratio, thus the smaller the proportion of the first data in the mixed data and the larger the proportion of the second data in the mixed data; conversely, the smaller the task complexity, the larger the preset ratio, thus the larger the proportion of the first data in the mixed data and the smaller the proportion of the second data in the mixed data, so as to use more first data without affecting accuracy and to ensure data cost.

[0102] In some embodiments, task complexity includes multiple complexity factors; these factors include, but are not limited to, the complexity of actions and trajectories, and the complexity of perception and decision-making.

[0103] The complexity of motion and trajectory represents the difficulty of motion trajectory information. For example, if the first objective is for the robotic arm to move from point A to point B in a straight line, and the second objective is for the robotic arm to move from point A to point B in a straight line while maintaining its posture by following a smooth arc, then the complexity of motion and trajectory for the first objective is lower than that for the second objective.

[0104] The complexity of perception and decision-making represents the difficulty of force changes in a motion trajectory. For example, the third objective task is for a robotic arm to grasp a fixed object according to a fixed program; the fourth objective task is for the robotic arm, in conjunction with a vision camera, to identify randomly placed objects and determine whether the objects are clothing or rigid components, in order to decide the grasping angle and force in real time. Therefore, the complexity of the motion and trajectory of the third objective task is lower than that of the fourth objective task. The second and fourth objective tasks can be combined into a single objective task, such as the task of folding clothes.

[0105] In this embodiment, the preset ratio is dynamically adjusted according to the complexity of the target task; the ratio of the first data and the second data is adaptively adjusted to suit the complexity of each task, so as to ensure environmental cognition through the first data and physical accuracy of the real machine data through the second data. The ratio of the first data is dynamically adjusted according to the complexity of the task to obtain the first data as training samples more efficiently, thereby improving the data collection efficiency.

[0106] In some embodiments, task complexity includes task duration, and task duration is positively correlated with task complexity.

[0107] Task duration refers to the length of time the robot takes to execute the target task. A longer task duration means more time for the robot to complete the task, resulting in higher task complexity, a smaller preset proportion, and a lower percentage of the primary data in the mixed data. Conversely, a shorter task duration means less time for the robot to complete the task, resulting in lower task complexity, a larger preset proportion, and a higher percentage of the primary data in the mixed data. It's understandable that the time taken for the robot to execute the target task can include both the actual time taken to complete the task and the estimated time taken.

[0108] Optionally, the task complexity that is positively correlated with the task duration can be determined based on the interval in which the task duration falls; when a person obtains the second data by performing the target task through the robot's robotic arm, the task duration can be the length of time the person controls the robotic arm; when the task complexity includes the task duration and other task complexity factors besides the task duration, the task complexity can be obtained by weighted calculation based on multiple task complexity factors.

[0109] In one embodiment, a preset proportion is obtained based on the task duration corresponding to the target task. Thus, the task duration is mapped to task complexity, allowing for direct quantification of task complexity through task duration.

[0110] For example, the first objective is for the robotic arm to pick up a cup from point A, move it in a straight line to point B, and then put the cup down; the second objective is for the robotic arm to pick up clothes from point A, move it in a straight line to point B, and then put the clothes down, while walking along a smooth arc to maintain the posture of the folded clothes. In this case, the task time of the first objective is shorter than the task time of the second objective.

[0111] In this embodiment, the task duration is positively correlated with the task complexity, so as to control the task complexity more efficiently. It can also form a cross-domain task evaluation dimension, so that the target tasks in various domains can be measured by the same standard, ensuring strong generalization of task complexity.

[0112] In some embodiments, task complexity includes the rigidity of the target object of the target task, and the rigidity of the target object is negatively correlated with task complexity.

[0113] Rigidity refers to the ability of a target object to resist shape change when subjected to force. In performing a target task, the greater the rigidity of the target object, the smaller the deformation. For example, if the target object is a block of wood or metal, it will hardly deform under force, resulting in a lower task complexity. The proportion of the first data in the mixed data increases, while the proportion of the second data decreases. Conversely, objects with lower rigidity produce less deformation. For example, clothing, sponges, or rubber deform easily under force. In this case, the task complexity is greater, the proportion of the first data in the mixed data decreases, and the proportion of the second data increases.

[0114] In some embodiments, the rigidity and difficulty of the target object of the target task are determined according to the task application scenario. For example, in the scenario of folding clothes, the difficulty is high and the rigidity is low, which is a complex task. In the scenario of grasping wooden blocks, the difficulty is low and the rigidity is high, which is a simple task.

[0115] In some embodiments, when the target task represents the robot contacting a rigid object, the complexity of the rigid task is obtained; when the target task represents the robot contacting a flexible object, the complexity of the flexible task is obtained; wherein the rigidity of the rigid object is higher than that of the flexible object, and the complexity of the rigid task is lower than that of the flexible task. Thus, the task complexity is controlled by using preset ratios corresponding to the rigid and flexible objects.

[0116] In this embodiment, rigidity is negatively correlated with task complexity to more accurately control task complexity, forming a task evaluation dimension based on rigidity and ensuring controllability. Furthermore, task duration forms a positive adjustment dimension for task complexity, while rigidity forms a negative adjustment dimension. Through these positive and negative adjustment dimensions, task complexity can be accurately and efficiently quantified.

[0117] In some embodiments, the method further includes: obtaining a preset ratio based on the robot's rated load; wherein the rated load is negatively correlated with the preset ratio.

[0118] Rated load is the maximum weight a robotic arm can lift; it represents the robot's load-bearing capacity, reflecting its weight-bearing ability. For example, when the first model of the robot has high dexterity, its rated load is lower, resulting in a larger preset ratio and a higher proportion of first data in the mixed data. This allows for training with less second data, leading to lower training costs. Conversely, when the second model of the robot has high load capacity, its larger inertia and severe PID overshoot result in a smaller preset ratio and a lower proportion of first data in the mixed data. This necessitates training with more second data to ensure reliability.

[0119] In some embodiments, when the rated load is a first weight load, a first weight preset ratio is obtained; when the rated load is a second weight load, a second weight preset ratio is obtained; wherein the first rated load ratio is less than the second rated load, and the first weight preset ratio is greater than the second weight preset. Thus, robots with different rated loads facing the same object also have their own corresponding preset ratios, allowing for more precise control of the robot's load capacity.

[0120] In some embodiments, when the rated load is within a first rated load range, the preset ratio is the first load preset ratio corresponding to the first rated first load range; when the rated load is within a second rated load range, the preset ratio is the second load preset ratio corresponding to the second rated second load range; the rated loads in the first rated load range are all greater than the rated loads in the second rated load range, and the first load preset ratio is less than the second load preset ratio. For example, the first load range can be 1-5 kg, with a preset ratio of 10:1; the second load range can be greater than 5 kg, with the preset ratio increased from 10:1.

[0121] In this embodiment, the rated load is negatively correlated with the task complexity. A preset ratio is adjusted to suit the capabilities of different robots, forming the rated load evaluation dimension. This load evaluation dimension is relatively independent of the task evaluation dimension, ensuring that the preset ratio is compatible with different robot models. Furthermore, task duration forms a positive adjustment dimension for task complexity, rigidity is a negative adjustment dimension for task complexity under the target object dimension, and rated load is a negative adjustment dimension for task complexity under the robot dimension. By combining the positive and negative adjustment dimensions of task complexity with the rated load dimension, the preset ratio for each robot can be more accurately refined.

[0122] In some embodiments, the method further includes: obtaining a preset ratio based on the robot's rated load and the task complexity of the target task; wherein the task complexity and the rated load are both negatively correlated with the preset ratio, and the degree to which the task complexity affects the preset ratio is greater than the degree to which the rated load affects the preset ratio.

[0123] The preset ratio can be a specific value, or a corresponding mapping strategy or acquisition strategy. For example, the preset ratio can be obtained first based on the task complexity, and then the preset ratio can be fine-tuned using the rated load so that the degree to which the task complexity affects the preset ratio is greater than the degree to which the rated load affects the preset ratio.

[0124] In some embodiments, a load coefficient is obtained by weighted mapping based on the rated load weight and the robot's rated load; a complexity coefficient is obtained by weighted mapping based on the task complexity coefficient and the task complexity of the target task; the load coefficient and the complexity coefficient are combined to obtain the target coefficient; a preset ratio is determined by the target coefficient; wherein the load coefficient is less than the complexity coefficient, so that the degree to which the task complexity affects the preset ratio is greater than the degree to which the rated load affects the preset ratio.

[0125] In some embodiments, a complexity preset ratio is obtained based on the task complexity of the target task; the complexity preset ratio is adjusted according to the rated load to obtain an adjusted preset ratio; the adjusted preset ratio is used to combine the first data and the second data into mixed data.

[0126] In this embodiment, the degree to which the task complexity changes by the preset ratio is relatively large, thereby forming an adjustment method dominated by the target task and a control framework dominated by the task complexity. Therefore, each robot with a rated load can be fine-tuned based on its own rated load. Only a small amount of data or even real-time online fine-tuning is needed to quickly adapt to the new environment without the need for tedious repeated calibration.

[0127] The second aspect of this application provides a limiting member, such as... Figure 3As shown, the limiting component includes: a carrier 310; and various limiting portions 320 disposed on the carrier 310. Each limiting portion 320 includes at least one of a first limiting portion and a second limiting portion, wherein the first limiting portion and the second limiting portion are the same or different limiting portions 320. The distance between each first limiting portion matches the distance between each manipulator when the robot starts performing the target task, and each data acquisition terminal is positioned by each first limiting portion when it starts performing the target task. The distance between each second limiting portion matches the distance between each manipulator when the robot stops performing the target task, and each data acquisition terminal is positioned by each second limiting portion when it stops performing the target task.

[0128] The support member 310 is a structural member used to support the limiting parts 320; each limiting part 320 is disposed on the support member 310, and the distance between each limiting part 320 is controlled by the distance between the support member 310 and the limiting part 320 itself. Optionally, the support member 310 includes multiple connectors, each connector being connected to a different limiting part 320, thereby controlling the distance between the limiting parts 320. Optionally, the support member 310 can be a connecting rod, a base, or a ring-shaped support member 310.

[0129] The limiting part 320 may include a first limiting part and a second limiting part. When the first limiting part and the second limiting part are the same limiting part 320, the distance between each manipulator when the robot starts to perform the target task and the distance between each manipulator when the robot finishes performing the target task are the same. When the first limiting part and the second limiting part are different limiting parts 320 and are in different positions, the distance between each manipulator when the robot starts to perform the target task and the distance between each manipulator when the robot finishes performing the target task are different. In other words, in this case, the distance between each manipulator is different when the target task is started and when the target task is stopped.

[0130] The first limiting part is used to limit the distance when the robot starts to perform the target task; the second limiting part is used to limit the distance when the robot finishes to perform the target task. The distance between each first limiting part and the distance between each second limiting part can be the same or different, and the structure of each first limiting part and the structure of each second limiting part can be the same or different to adapt to different needs.

[0131] In some embodiments, one pair of first limiting portions is a card slot, and the other pair of first limiting portions is magnetic; one pair of second limiting portions is a card slot, and the other pair of second limiting portions 320 is magnetic.

[0132] In some embodiments, when each data acquisition terminal starts to execute the target task, each data acquisition terminal is inserted into the slots positioned by each first limiting part, so that the distance between each first limiting part matches the distance between each manipulator when the robot starts to execute the target task.

[0133] In some embodiments, when each data acquisition terminal finishes executing the target task, each data acquisition terminal is inserted into the slot positioned by each second limiting part, so that the distance between each second limiting part matches the distance between each manipulator when the robot finishes executing the target task.

[0134] In this embodiment, each data acquisition terminal is positioned by the limiting part of the limiting member. Since the spacing of the limiting parts is fixed, even without body constraints, whether at the beginning or end of the target task, the spacing between each data acquisition terminal can be matched with, or even completely matched with, the spacing between each robot arm. This ensures that the acquired first data is more consistent with the kinematic prior and that the first data matches the second data to a high degree. The limiting part can be a first limiting part or a second limiting part, and the first limiting part and the second limiting part can be the same or different to adapt to robot training processes with different kinematic constraints.

[0135] In some embodiments, the carrier 310 includes a first connector 311 and at least two second connectors 312;

[0136] Wherein, the first end of the first connector 311 is detachably connected to a second connector 312, and the second end of the first connector 311 is detachably connected to another second connector 312, wherein the first end and the second end include the two opposite ends of the first connector 311; or,

[0137] Wherein, the first end of the first connector 311 is foldably connected to a second connector 312, and the second end of the first connector 311 is foldably connected to another second connector 312. The first end and the second end include the two opposite ends of the first connector 311.

[0138] The first connector 311 is the main body of the support member 310, and the two opposite ends are the ends located on both sides of the first connector 311; the first end and the second end can have the same or different structures to form two connection methods. The second connector 312 is an extension of the first connector 311 and can be used to control the length.

[0139] Optionally, a detachable connection means that the first connector 311 and the second connector 312 are connected in a detachable manner, for example, the first connector 311 and the second connector 312 are connected by means of threads, snaps, magnetic attraction, etc.; a foldable connection means that the first connector 311 and the second connector 312 are connected in a foldable manner, for example, by means of a pivot or hinge.

[0140] In one embodiment, two second connectors 312 are provided with limiting portions 320 to determine the distance between two identical limiting portions 320 by the length of the two second connectors 312 and the length of the first connector 311; or, the first end of the first connector 311 is provided with a limiting portion 320, and one second connector 312 is provided with a limiting portion 320 to determine the distance between two identical limiting portions 320 by the length of the second connector 312 and the length of the first connector 311.

[0141] In this embodiment, the carrier 310 is composed of a first connector 311 and a second connector 312. The distance between each limiting part 320 can be controlled by these two connectors, so that the limiting part can be adapted to the spacing between different robotic arms, thus ensuring that the spacing is adjustable.

[0142] In one exemplary embodiment, this embodiment establishes a data matching principle, that is, it can distribute massive amounts of data without an ontology and a small amount of real device data according to a preset ratio, and it also pioneers a hardware and software joint spatial alignment mechanism based on physical entity structural components.

[0143] This embodiment is a "data training ratio and acquisition workstation", which includes hardware auxiliary components and software control console. It is divided into hardware auxiliary component positioning structure, software data acquisition process guidance interface, and ratio control console interface.

[0144] The hardware auxiliary positioning structure is designed for the target robot model and includes a structural component that acts as a limiting element. This limiting element has two slots as limiting parts, and the distance between the two slots is precisely equal to the baseline distance between the grippers of the robot's two arms in the initial state. This ensures that when starting or stopping the target task, the distance between each data acquisition terminal is the same as the distance between each manipulator of the target robot.

[0145] The data acquisition process guidance interface refers to the software prompting and requiring the operator to insert the two devices (handles) generated by the data acquisition terminal into the positioning structure before starting recording. At the end of recording, the operator is similarly prompted to return the handles to the structure and lock them in place to complete the closed loop.

[0146] The configuration control panel interface refers to the embedded database of machine models and configuration parameters when users prepare training data. Users import datasets without the subject or real machines, and the processor determines a preset ratio (e.g., the ratio of the first data set to the second data set is 10:1) based on the machine model corresponding to the selected rated load. It automatically performs data sampling and mixing, and outputs the optimal training set, avoiding random mixing by the user.

[0147] In this scenario, the first data collected by the robot itself and the second data collected by non-robots are combined in a certain ratio to form mixed data, which is then used for robot training. The specific forms of the data collected by the robot and the data collected by non-robots are consistent, with the preset ratio ranging from less than 10:1 to greater than 1:1. Simultaneously, when starting or ending the target task, the distance between the robot's manipulators remains at the initial position. When no data is being collected by the robot itself, the distance between the initial positions of the two user handles before data collection is equal to the distance between the robot's two grippers in their initial positions before the task begins. Different rated loads correspond to different robot models, and different robot models have different preset ratios.

[0148] In one exemplary embodiment, the data formats of the first data and the second data are consistent; the processor unifies the specific data formats of the data acquired by the actual device and the data generated by the data acquisition terminal to ensure that the two can be directly mixed proportionally. Specific formats include the following types: Vision is a synchronized, visually cleaned multi-view image stream (such as left, right, and front-view three views). Text is a natural language instruction describing the task operation. Action is a high-frequency recorded 6-DOF pose trajectory, including translation (x, y, z) and rotation (roll, pitch, yaw). Gripper state is the opening and closing control signal of the gripper.

[0149] Both the first and second sets of data contain these types of data, but the first set contains data with positioning discrepancies, which need to be eliminated through kinematic priors or inverse kinematics. The data pipeline for data without ontology data also involves human quality inspection actions, inverse kinematics calculation, and image blur detection. The inverse kinematics calculation is performed using an algorithm that simulates the robot's own body.

[0150] In one exemplary embodiment, after unifying the formats of the first and second data, to completely eliminate the morphological differences between the human operator and the target dual-arm robot, purely software-based spatial measurement errors are abandoned, and a physical hard-limiting process is introduced:

[0151] First, the physical limit initialization step is performed. Specifically, before the first data acquisition task begins, the operator uses the left and right handles as data acquisition terminals and accurately engages them in the limit components set according to the distance between the grippers of the actual robot. This ensures, from a physical hardware perspective, that the Euclidean distance between the initial positions of the two user handles is precisely equal to the distance between the two grippers in their initial positions before the controlled robot begins its task.

[0152] Next, the trajectory acquisition and unbinding steps are executed: the processor establishes a unified world coordinate system origin in this posture and begins recording data. The operator removes the handle from the structural component, performing a natural and smooth bodyless demonstration task. During this process, an inverse kinematics algorithm is used to detect whether the trajectory corresponding to the manipulator can be solved onto the manipulator; at this point, the handle is equivalent to the end effector of the manipulator, performing inverse kinematics operations within the manipulator. Inverse kinematics is used to detect in real-time whether any point on the current trajectory can be inversely solved onto the manipulator. If not, it is deleted, and a voice prompt indicates that there is no solution. This check can be performed every time a data point is acquired. Then, during subsequent mixed training, multiple qualified data points are used for model training.

[0153] Next, the task completion and loop alignment steps are executed. Specifically, after the task is completed, the operator repositions the two handles back into the structure to end the recording. This ensures the starting and ending points are aligned, and the entire process is performed using inverse kinematics. This ensures that the trajectory's start and end points are closed under absolutely consistent morphological constraints, greatly improving the success rate and physical realism of subsequent inverse kinematics calculations.

[0154] In an exemplary embodiment, after obtaining the first data and the second data, it is necessary to dynamically mix the first data and the second data using a specific ratio range to obtain mixed data. Wherein:

[0155] First, the core logic of the ratio is: a massive amount of first data is used to build general semantic cognition, and a very small amount of second data serves as kinematic anchors to provide low-level physical priors for specific hardware.

[0156] Next comes the step of defining a specific ratio range. At this point, the processor limits the ratio between the first data and the second data from the real device to a preset ratio, which is between the first and second ratio values, specifically between 1:1 and 10:1. This preset ratio is negatively correlated with the complexity of the target task. Task complexity includes the task duration and / or the rigidity of the target object. In one example, the task complexity can be determined by the length of time a human operates the robotic arm to perform the target task; the longer the time, the higher the task complexity. In another example, rigid contact is a simple task, while flexible contact is a complex task. The proportion of the first data used in a rigid contact task is greater than that in a flexible contact task. Optionally, the ratio can be adjusted based on the difficulty of the task's application scenario; for example, folding clothes is a complex task, while grasping data is relatively simple.

[0157] Taking a preset ratio of 10:1 as an example, the expression is: Data volume of the first data : Data volume of the first data = 10:1. In the steps defining a specific ratio range, two steps are required: data preparation and hybridization. During data preparation, 500 first data points calibrated using the aforementioned cardboard and 50 master-slave operation real machine data points can be imported. In the hybridization mechanism, the 500 first data points act as a semantic and spatial generalization engine, helping the VLA model understand the environment, identify object availability, and plan macroscopic topological trajectories. The 50 real machine data points act as "kinematic anchors," forcibly aligning the motion features output by the large model to the friction coefficient, joint limits, and control latency of the specific robot. Optionally, data collection can be tailored to different backgrounds; for kitchens, dining rooms, and bedrooms, the corresponding collection strategies can be adaptively adjusted. This involves data obtained by controlling the robot's movements through human intervention, allowing it to approach or adapt to the robot's target task, based on the gripper's body movements and the robot's working environment.

[0158] Tests show that the strategy trained using a 10:1 processor ratio achieves the same physical execution success rate in tasks such as folding towels as a model trained with only 500 expensive real-device data sets. This forms the core theoretical basis for the exponential cost reduction in this embodiment.

[0159] In an exemplary embodiment, after selecting data based on factors such as task complexity, the robot can select a preset ratio according to its rated load for different models. For example, when a user selects a highly dexterous, lightweight, multi-jointed robot on the interface, the processor determines that it has good compliance and a moderate dependence on real physical priors, and automatically adjusts the ratio to 8:1 or 10:1. However, when the user selects a heavy-load, rigid robot, the processor determines that it has large inertia and severe PID overshoot, and automatically adjusts a conservative ratio of 1:1 or 3:1 to increase the proportion of real machine data to suppress unstable motion output.

[0160] Therefore, by using physical hard-limit calibration, the problem of data mismapping is completely solved. Specifically, by using customized structural components to hold the first device in place, errors caused by operator intuition or pure software estimation are eliminated. This ensures that the distance between the handles and the distance between the grippers on the real machine are physically consistent at the millimeter level at the beginning and end of data acquisition, resulting in an extremely high success rate and effectiveness in solving the inverse kinematics of the first trajectory. Simultaneously, it overcomes strategy degradation caused by random mixing; specifically, it clarifies and solidifies the optimal complementary ratio (e.g., 10:1) between the first and real machine data, avoiding data feature conflicts caused by blind mixing. Moreover, it exponentially reduces data costs. While ensuring an extremely high training success rate, it uses a large-scale, low-cost first dataset mixed with a very small amount of second data from the real machine according to a preset ratio, reducing the expensive workload of real machine data acquisition to a fraction of the original.

[0161] Therefore, the non-random proportional training mechanism provides a method for training robot strategies. Data collected by the robot itself and data collected by non-robots must be mixed in a certain proportion before participating in the strategy training of the embodied intelligent robot, to prevent poor results caused by random mixing. The specific form of multimodal data is defined as follows: the first data collected by non-robots and the second data collected by the robot itself have a unified specific form; this specific form includes synchronized multi-view image streams, language commands, high-frequency 6-DOF pose trajectories, and gripper state data. Physical calibration of the data acquisition start and end points based on a customized structural component: to ensure accurate adaptation to the real machine during the first data training, the processor uses a limiting component customized according to the gripper spacing of the real machine. Through this physical structural component, during the first data acquisition, the first acquisition device (handle) is accurately locked, so that the distance between the two user handle positions before and after data acquisition is physically constrained to be equal to the distance between the two grippers of the robot in the initial position before the controlled robot begins its task.

[0162] Specifically, during data mixing, the processor uses a specific ratio range (between 1:1 and 10:1) for non-robot body data and robot body data. This range is made available to users upon device delivery, guiding them to build datasets according to this ratio. For multi-model, differentiated preset ratios are also available. Due to different physical characteristics, the processor sets different preset data mixing ratios for different robot models; the processor can automatically match and retrieve the optimal ratio for data mixing training based on the robot model selected by the user.

[0163] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0164] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0165] In one exemplary embodiment, such as Figure 4 As shown, a data processing apparatus is provided, comprising:

[0166] The first acquisition module 402 is used to acquire the first data generated when each data acquisition terminal performs the target task;

[0167] The second acquisition module 404 is used to acquire second data generated by the robot when it uses each robotic arm to perform the target task;

[0168] The mixing module 406 is used to determine mixed data based on the first data and the second data, and the mixed data can be used to train the control model of the robot;

[0169] Specifically, when each of the data acquisition terminals and the robotic arm begins to execute the target task, the distance between each of the data acquisition terminals matches the distance between each of the robotic arms; when each of the data acquisition terminals and the robotic arm stops executing the target task, the distance between each of the data acquisition terminals matches the distance between each of the robotic arms.

[0170] In one embodiment, the first data includes the motion trajectory information of the data acquisition terminal and the environmental data collected by the data acquisition terminal, and the second data includes the motion trajectory information of the robotic arm and the environmental data collected by the robotic arm.

[0171] In one embodiment, the spacing between the robotic arms includes the distance between the centers of the robotic arms; and / or, the spacing between the data acquisition terminals matches the spacing between the robotic arms, meaning that the spacing between the corresponding data acquisition terminals is the same as the spacing between the robotic arms.

[0172] In one embodiment, when the target task is started, each data acquisition terminal is positioned by a limiting part of a limiting member; when the target task is stopped, each data acquisition terminal is positioned by a limiting part of the limiting member; the distance between each limiting part matches the spacing between each robotic arm when the robot starts executing the target task; the spacing between each robotic arm when the robot starts executing the target task matches the spacing between each robotic arm when the robot stops executing the target task.

[0173] In one embodiment, the mixing module 406 is used for,

[0174] The first data and the second data are used to determine mixed data based on a preset proportional distribution;

[0175] The preset ratio includes the ratio of the amount of data between the first data and the second data.

[0176] In one embodiment, the preset ratio is between a first ratio and a second ratio; the first ratio includes a ratio of 1:1 between the first data and the second data; the second ratio includes a ratio of 10:1 between the first data and the second data.

[0177] In one embodiment, the mixing module 406 is configured to:

[0178] The preset ratio is obtained based on the task complexity of the target task;

[0179] The complexity of the task is negatively correlated with the preset ratio.

[0180] In one embodiment, the task complexity includes task duration, and the task duration is positively correlated with the task complexity.

[0181] In one embodiment, the task complexity includes the rigidity of the target object of the target task, and the rigidity of the target object is negatively correlated with the task complexity.

[0182] In one embodiment, the mixing module 406 is configured to:

[0183] The preset ratio is obtained based on the robot's rated load;

[0184] The rated load is negatively correlated with the preset ratio.

[0185] In one embodiment, the mixing module 406 is configured to:

[0186] The preset ratio is obtained based on the robot's rated load and the task complexity of the target task;

[0187] The task complexity and the rated load are both negatively correlated with the preset ratio, and the degree to which the task complexity affects the preset ratio is greater than the degree to which the rated load affects the preset ratio.

[0188] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0189] In one exemplary embodiment, a computer device is provided, which may be a data acquisition terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external data acquisition terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a data processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a handheld handle for collecting motion data; a handheld sensing gripper with a camera and / or IMU; a motion capture glove for capturing subtle finger movements and tactile sensations; a myoelectric wristband for measuring muscle electrical signals and wrist posture; or an external keyboard, touchpad, or mouse.

[0190] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0191] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0192] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0193] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0194] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0195] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0196] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0197] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Acquire the first data generated by each data acquisition terminal when it executes the target task; Acquire second data generated by the robot when it uses each robotic arm to perform the target task; and Based on the first data and the second data, mixed data is determined, which can be used to train the control model of the robot; Specifically, when each of the data acquisition terminals and the robotic arm begins to execute the target task, the distance between each of the data acquisition terminals matches the distance between each of the robotic arms; when each of the data acquisition terminals and the robotic arm stops executing the target task, the distance between each of the data acquisition terminals matches the distance between each of the robotic arms.

2. The method according to claim 1, characterized in that, The first data includes the motion trajectory information of the data acquisition terminal and the environmental data collected by the data acquisition terminal, and the second data includes the motion trajectory information of the robotic arm and the environmental data collected by the robotic arm.

3. The method according to claim 1, characterized in that, The spacing between each of the robotic arms includes the distance between the centers of each of the robotic arms; and / or, the spacing between each of the data acquisition terminals matches the spacing between each of the robotic arms, meaning that the spacing between each data acquisition terminal with a corresponding relationship is the same as the spacing between each of the robotic arms.

4. The method according to claim 1, characterized in that, When the target task is started, each data acquisition terminal is positioned by a limiting part of the limiting member; when the target task is stopped, each data acquisition terminal is positioned by a limiting part of the limiting member; the distance between each limiting part matches the spacing between each robotic arm when the robot starts executing the target task; the spacing between each robotic arm when the robot starts executing the target task matches the spacing between each robotic arm when the robot stops executing the target task.

5. The method according to claim 1, characterized in that, The process of determining mixed data based on the first data and the second data includes, The first data and the second data are used to determine mixed data based on a preset proportional distribution; The preset ratio includes the ratio of the amount of data between the first data and the second data.

6. The method according to claim 5, characterized in that, The preset ratio is between a first ratio and a second ratio; the first ratio includes a 1:1 ratio of the first data to the second data; the second ratio includes a 10:1 ratio of the first data to the second data.

7. The method according to claim 5, characterized in that, The method further includes: The preset ratio is obtained based on the task complexity of the target task; The complexity of the task is negatively correlated with the preset ratio.

8. The method according to claim 7, characterized in that, The task complexity includes the task duration, and the task duration is positively correlated with the task complexity.

9. The method according to claim 7, characterized in that, The task complexity includes the rigidity of the target object of the target task, and the rigidity of the target object is negatively correlated with the task complexity.

10. The method according to claim 5, characterized in that, The method further includes: The preset ratio is obtained based on the robot's rated load; The rated load is negatively correlated with the preset ratio.

11. The method according to claim 5, characterized in that, The method further includes: The preset ratio is obtained based on the robot's rated load and the task complexity of the target task; The task complexity and the rated load are both negatively correlated with the preset ratio, and the degree to which the task complexity affects the preset ratio is greater than the degree to which the rated load affects the preset ratio.

12. A limiting member, characterized in that, include: Load-bearing components; Each limiting part is disposed on the carrier member; each limiting part includes at least one of a first limiting part and a second limiting part, wherein the first limiting part and the second limiting part are the same or different limiting parts; The distance between each of the first limiting parts is matched with the spacing between each manipulator when the robot starts to perform the target task. When each data acquisition terminal starts to perform the target task, each of the data acquisition terminals is positioned by each of the first limiting parts. The distance between each of the second limiting parts is matched with the spacing between each manipulator when the robot stops performing the target task. When each data acquisition terminal stops performing the target task, each data acquisition terminal is positioned by each of the second limiting parts.

13. The limiting member according to claim 12, characterized in that, The support member includes a first connector and at least two second connectors; Wherein, a first end of the first connector is detachably connected to one of the second connectors, and a second end of the first connector is detachably connected to another of the second connectors, wherein the first end and the second end include the two opposite ends of the first connector; or, Wherein, the first end of the first connector is foldably connected to one of the second connectors, and the second end of the first connector is foldably connected to another of the second connectors, wherein the first end and the second end include the two ends opposite to the first connector.

14. A data processing apparatus, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images