Control method, training method, control assembly and robot

Abstract

The embodiment of the invention discloses a control method, a training method, a control assembly and a robot, the control method is applied to the robot, and the control method comprises the steps of determining a first interaction pose for a to-be-interacted object based on an image of an environment where the robot is located; generating a splicing feature of the current time step based on an output instruction of the previous time step and the first interaction pose; inputting the splicing feature of the current time step and the splicing feature of at least one historical time step into a trained attribute prediction module to obtain an attribute feature; the attribute features are used for representing attribute data, related to interaction influence, of the to-be-interacted object; and inputting the attribute features and the splicing features of the current time step into a trained reinforcement learning module to obtain an output instruction of the current time step. In this way, the reinforcement learning module can better understand the intrinsic characteristics of the object, and has high generalization ability when processing unseen objects.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a control method, a training method, a control component and a robot. Background Art

[0002] Manipulating articulated objects with a robot is a challenging task that requires a deep understanding of the object and compliant movement to avoid damaging the object and the robot. In related technologies, point cloud input is used to predict the affordance graph, and then the operation action is predicted through reinforcement learning strategy; or, multi-view RGB images (Red Green Blue, RGB) or partial point clouds are used to estimate the pose of the component, and then the planned action is performed through a heuristic operation module. These methods do not have high generalization ability when dealing with unseen objects, nor have they shown high success rates in the real world. Summary of the invention

[0003] In view of this, the embodiments of the present application provide at least a control method, a training method, a control component and a robot.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] On the one hand, an embodiment of the present application provides a control method applied to a robot, the method comprising: determining a first interaction posture for an object to be interacted with based on an image of the environment in which the robot is located; generating a splicing feature of a current time step based on an output instruction of a previous time step and the first interaction posture; inputting the splicing feature of the current time step and the splicing feature of at least one historical time step into a trained attribute prediction module to obtain an attribute feature; the attribute feature is used to characterize the attribute data of the object to be interacted with that is related to the influence on the interaction; inputting the attribute feature and the splicing feature of the current time step into a trained reinforcement learning module to obtain an output instruction of the current time step.

[0006] On the other hand, an embodiment of the present application provides a training method for a control model, wherein the control model includes an attribute prediction module and a reinforcement learning module; the training method includes: generating a simulation splicing feature of the second time step based on a simulation output instruction of a first time step and observation data of a second time step; the first time step is the previous time step of the second time step; inputting the simulation splicing feature of the second time step and the simulation splicing feature of at least one historical time step into the attribute prediction module to obtain a predicted simulation attribute feature; the simulation attribute feature is used to characterize attribute data related to the interaction of a sample object to be interacted in a simulation environment; inputting the simulation output instruction of the second time step and a target attribute feature into the reinforcement learning module to obtain a simulation output instruction of the second time step; the target attribute feature is the predicted simulation attribute feature or a simulation attribute feature generated based on the attribute data of the sample object; after controlling the movement of the robot based on the simulation output instruction of the second time step, obtaining a motion reward; training the attribute prediction module based on the predicted simulation attribute feature and the generated simulation attribute feature, and training the reinforcement learning module through reinforcement learning based on the motion reward.

[0007] On the other hand, an embodiment of the present application provides a control component, which is arranged in a robot, and the control component includes: a determination unit, which is used to determine a first interaction posture for an object to be interacted with based on an environment image in which the robot is located; a generation unit, which is used to generate a splicing feature of a current time step based on an output instruction of a previous time step and the first interaction posture; a first input unit, which is used to input the splicing feature of the current time step and the splicing feature of at least one historical time step into a trained attribute prediction module to obtain an attribute feature; the attribute feature is used to characterize the attribute data of the object to be interacted with that is related to the influence on the interaction; a second input unit, which is used to input the attribute feature and the splicing feature of the current time step into a trained reinforcement learning module to obtain an output instruction of the current time step.

[0008] On the other hand, an embodiment of the present application provides a robot, which includes the above-mentioned control component.

[0009] In an embodiment of the present application, the reinforcement learning module can consider the attribute data, interaction posture and historical output instructions of the object to be interacted with that affect the interaction when outputting instructions, reduce dependence on vision, better understand the intrinsic characteristics of objects, have a higher generalization ability when processing unseen objects, show a high success rate in the real world, and reduce the gap between simulation and reality.

[0010] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 A schematic diagram of an implementation process of a control method provided in an embodiment of the present application Figure 1 ;

[0013] Figure 2 A schematic diagram of an implementation process of a control method provided in an embodiment of the present application Figure 2 ;

[0014] Figure 3 A schematic diagram of an implementation process of a control method provided in an embodiment of the present application Figure 3 ;

[0015] Figure 4 A schematic diagram of an implementation process of a control method provided in an embodiment of the present application Figure 4 ;

[0016] Figure 5 A schematic diagram of an implementation process of a control method provided in an embodiment of the present application Figure 5 ;

[0017] Figure 6 A schematic diagram of the implementation process of a control model training method provided in an embodiment of the present application Figure 1 ;

[0018] Figure 7 A schematic diagram of the implementation process of a control model training method provided in an embodiment of the present application Figure 2 ;

[0019] Figure 8 Schematic diagram of the application of the reinforcement learning strategy provided in the embodiment of the present application Figure 1 ;

[0020] Fig. 9 Schematic diagram of the application of the reinforcement learning strategy provided in the embodiment of the present application Figure 2 ;

[0021] Fig.10 A schematic diagram of the experimental setup of an embodiment of the present application;

[0022] Fig.11 This is a schematic diagram of the experimental results of the embodiment of the present application;

[0023] Fig.12 A schematic diagram of the structure of a control component provided in an embodiment of the present application;

[0024] Fig.13 A schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0025] Fig.14 A hardware entity diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0027] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0029] The present application provides a control method and a control model training method, which can be executed by a processor of a computer device. The computer device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other devices with data processing capabilities.

[0030] In the field of robot control, for the manipulation of articulated objects, one method is to use point cloud input to predict the affordance graph, and then predict the manipulation action through reinforcement learning strategy. This method uses point cloud features as policy input, resulting in a large gap between simulation and reality. In addition, during execution, the affordance graph is blocked by the robot arm. Another method is to use multi-view RGB images to estimate the pose of the component, and then perform the planning action through the heuristic operation module. Among them, multi-view RGB requires more time for component pose estimation, and the predicted waypoints need to be adjusted at each local point, resulting in an uneven trajectory. The third method is to use partial point clouds to estimate the pose of the component, and then perform the planning action through the heuristic operation module. For heuristic operation, due to the lack of feedback mechanism to adjust the planned action, the intrinsic characteristics of the object are ignored, which may lead to unsafe behavior. These methods do not have high generalization ability when dealing with unseen objects, nor do they show high success rates in the real world.

[0031] The embodiment of the present application provides a control method, which is applied to a robot, and the method includes: determining a first interactive posture for an object to be interacted with based on an environmental image in which the robot is located; generating a splicing feature of the current time step based on the output instruction of the previous time step and the first interactive posture; inputting the splicing feature of the current time step and the splicing feature of at least one historical time step into a trained attribute prediction module to obtain an attribute feature; the attribute feature is used to characterize the attribute data of the object to be interacted with related to the interaction; inputting the attribute feature and the splicing feature of the current time step into a trained reinforcement learning module to obtain an output instruction of the current time step. In this way, the reinforcement learning module can consider the attribute data, interactive posture and historical output instructions of the object to be interacted with related to the interaction when outputting instructions, reduce dependence on vision, better understand the intrinsic characteristics of objects, have a higher generalization ability when processing unseen objects, show a high success rate in the real world, and reduce the gap between simulation and reality.

[0032] In some embodiments, a hinged object refers to an object composed of multiple movable parts, which are connected by hinges, slides or other joints. Typical hinged objects include doors, drawers, robotic arms, etc. The connection between the joints enables the object to perform certain movements, such as rotation, sliding, etc. Reinforcement learning strategy is a machine learning method that optimizes action strategies based on a trial-and-error process through interaction with the environment to maximize cumulative rewards. Reinforcement learning strategy refers to the action selection rules that the reinforcement learning algorithm continuously adjusts during the training process to make optimal decisions in different situations. Point cloud is a three-dimensional data representation consisting of a large number of points, each of which has its three-dimensional coordinates (X, Y, Z). Point clouds are usually collected by devices such as laser radar (LiDAR) or RGB-D cameras, and are widely used in tasks such as 3D modeling, object detection, and scene understanding.

[0033] Figure 1 A schematic diagram of an implementation process of a control method provided in an embodiment of the present application Figure 1 , the control method is applied to robots, such as Figure 1 As shown, the method includes the following steps S101 to S104:

[0034] Step S101: determining a first interaction posture for an object to be interacted with based on an image of the environment in which the robot is located.

[0035] The environment image refers to the image of the environment where the robot is located, and the environment image at least contains the object to be interacted with. The environment image can be obtained by shooting the environment where the robot is located by an image acquisition module (such as a camera) carried by the robot.

[0036] The object to be interacted with is an object that the robot operates. For example, the robot needs to perform a task of opening a door by grabbing a door handle and opening the door, and the object to be interacted with is the door handle.

[0037] The first interactive posture is the position and posture that the robot's end effector (such as a gripper, a suction cup, etc.) is expected to reach when the robot performs an interactive task. For example, when the robot performs a door opening task, the first interactive posture is the position and posture that the robot's gripper is expected to reach.

[0038] The robot's interactive operation on the object to be interacted with may be a grabbing operation, a lifting operation, a translation operation, or a push-pull operation. The embodiment of the present application does not limit the specific interaction type.

[0039] In some embodiments, the environment image includes a depth image and an RGB image. Object recognition is performed on the RGB image through machine learning to obtain a mask image of the object to be interacted with; the depth image is converted to obtain environment point cloud data; multiple second interaction postures are determined using the environment point cloud data; and the mask image is used to determine a first interaction posture for the object to be interacted with from the multiple second interaction postures.

[0040] In some embodiments, a model for generating the first interaction posture is pre-trained, and the model can identify the objects to be interacted in the environment image and generate the first interaction posture for the objects to be interacted. The training samples can be multiple environment images containing the objects to be interacted, and the labels can be the objects to be interacted in the environment image and the corresponding first interaction posture.

[0041] Step S102: Generate a splicing feature of the current time step based on the output instruction of the previous time step and the first interaction posture.

[0042] When the robot performs an operation task, the entire action execution process of the robot is divided into multiple time steps, and the corresponding action is executed at each time step based on the output instruction of the reinforcement learning module. The output instruction contains information related to the robot's execution of the action.

[0043] In some embodiments, the output instructions include target displacement, target posture, gripper action, and impedance control parameters. The target displacement is the target position to which the robot needs to move. The target posture refers to the complete direction or posture information of the robot in three-dimensional space relative to a reference coordinate system (such as a world coordinate system or a camera coordinate system). The gripper action refers to whether the robot grabs an object. The impedance control parameters are used to perform impedance control on the output action.

[0044] Among them, the splicing features of the current time step include the output instructions of the previous time step and the observation data of the current time step. The observation data refers to the environmental state information of the robot's environment. The splicing features are used to characterize the robot's historical execution actions and the current environmental state.

[0045] In some embodiments, the observation data includes a first interaction posture, a robot joint configuration, a relative distance between the robot and the object, an end effector posture, and a graspability signal. The robot joint configuration refers to the selection and configuration of the type, quantity, arrangement, and drive mode of the robot's joints during the design and manufacturing process. The relative distance between the robot and the object refers to the distance between the robot's end effector and the object to be interacted with. The end effector posture refers to the position and posture of the end effector in a specified coordinate system. The graspability signal refers to whether the robot's end effector grasps the object. The graspability signal is a condition based on distance and contact perception, rather than a command to directly control the opening and closing of the gripper.

[0046] In some embodiments, at each time step, the robot obtains observation data of the current time step, the observation data includes a first interaction posture, and combines the observation data of the current time step with the output instruction of the previous time step to obtain a splicing feature of the current time step.

[0047] Step S103: input the concatenated features of the current time step and the concatenated features of at least one historical time step into the trained attribute prediction module to obtain attribute features; the attribute features are used to characterize the attribute data of the object to be interacted with that is related to the influencing interaction.

[0048] The attribute prediction module is used to infer the attribute characteristics of the object to be interacted from the historical observation data and the output instructions. The attribute characteristics are used to characterize the attribute data of the object to be interacted that are related to the interaction.

[0049] In some embodiments, in order to better understand the environment, it is necessary to obtain attribute data of the object to be interacted that is related to the interaction, but this data is difficult to obtain directly in the real world. Therefore, by training an attribute prediction module, the splicing features of the current time step and the splicing features of at least one historical time step are input into the trained attribute prediction module, and the attribute features can be obtained, and the attribute features can approximately reflect the attribute data of the object to be interacted that is related to the interaction.

[0050] In some embodiments, at least one historical time step is a historical time step adjacent to the current time step. Exemplarily, when the current time step is t, the three historical time steps include time steps t-1, t-2, and t-3.

[0051] In some embodiments, the attribute data includes at least one of the following: pivot center, pivot radius, object stiffness, object mass, object joint position, and handle grab signal. The pivot center refers to the center point or axis of the rotational motion of an object or system. The pivot radius refers to the distance from the pivot center to a point on the rotating object. Object stiffness refers to the ability of an object to resist deformation when subjected to external forces. Object mass is a physical quantity that describes the magnitude of an object's inertia, and it is a measure of the amount of matter contained in the object. Object joint position is the current posture of the object's joints. The handle grab signal is the current state of whether the handle is grabbed.

[0052] In some embodiments, during the simulation process, at any time step, the attribute data of the object to be interacted can be directly obtained through the simulation software, and the simulation attribute features can be generated based on the attribute data; the attribute prediction module can obtain the predicted simulation attribute features from the splicing features of the current time step and the splicing features of at least one historical time step. The parameters of the attribute prediction module are adjusted based on the difference between the generated simulation attribute features and the predicted simulation attribute features, and finally the trained attribute prediction module is obtained. During the application process, the splicing features of the current time step and the splicing features of at least one historical time step are input into the trained attribute prediction module to obtain the attribute features.

[0053] Step S104: input the attribute feature and the concatenated feature of the current time step into the trained reinforcement learning module to obtain an output instruction of the current time step.

[0054] In some embodiments, for the task of the robot operating an object, a reinforcement learning module is trained, which can output the action that the robot should perform at each time step based on the current environment state and historical actions. The robot performs actions based on the action instructions output by the reinforcement learning module to complete the task.

[0055] Among them, at each time step, the reinforcement learning module can determine the output instruction of the current time step based on the attribute features output by the attribute prediction module and the splicing features of the current time step. The robot can perform actions based on the output instructions.

[0056] In some embodiments, during the simulation training process of the reinforcement learning module, a generated simulation attribute feature and a predicted simulation attribute feature are obtained at each time step. The generated simulation attribute feature or the predicted simulation attribute feature and the splicing feature of the current time step are input into the reinforcement learning module to obtain an output instruction. The robot will receive a motion reward after executing the output instruction. The parameters of the reinforcement learning module are adjusted based on the motion reward, and finally the trained reinforcement learning module is obtained. During the application process, the attribute feature and the splicing feature of the current time step are input into the trained reinforcement learning module to obtain the output instruction of the current time step.

[0057] For example, the robot performs a door opening task, and the object to be interacted with is a door handle. Based on the environment image, the robot determines a first interaction posture g for the door handle. t , and then get the observation index o of the current time step t t , observation index o t Including the first interactive pose g t . The output instruction a of time step t-1 t-1 With observation indicators o t Perform splicing to obtain the splicing feature p t =(o t ⊕a t-1 ), the splicing feature p t and the concatenated feature p at time step t-1 t-1 Input the trained attribute prediction module to obtain the predicted attribute features The predicted attribute features and the concatenated feature p of the current time step t Input the trained reinforcement learning module to get the output instruction a of the current time step t .

[0058] In an embodiment of the present application, a first interactive posture for an object to be interacted is determined based on an image of the environment in which the robot is located; a splicing feature of the current time step is generated based on the output instruction of the previous time step and the first interactive posture; the splicing feature of the current time step and the splicing feature of at least one historical time step are input into a trained attribute prediction module to obtain an attribute feature; the attribute feature is used to characterize the attribute data of the object to be interacted related to the interaction; the attribute feature and the splicing feature of the current time step are input into a trained reinforcement learning module to obtain an output instruction of the current time step. In this way, the reinforcement learning module can consider the attribute data of the object to be interacted related to the interaction, the interactive posture and the historical output instructions when outputting instructions, reduce dependence on vision, better understand the intrinsic characteristics of objects, have a higher generalization ability when processing unseen objects, show a high success rate in the real world, and reduce the gap between simulation and reality.

[0059] In some embodiments, the trained attribute prediction module and the trained reinforcement learning module are obtained after training using attribute data related to influencing interaction of sample objects to be interacted in a simulation environment;

[0060] Wherein, in any time step of the training process, the attribute prediction module performs parameter adjustment based on the difference between the simulated attribute features generated in any time step and the simulated attribute features predicted by the attribute prediction module; the generated simulated attribute features are obtained based on the attribute data of the sample object;

[0061] The reinforcement learning module adjusts parameters based on motion rewards, and the motion rewards are obtained after the robot moves in response to simulation output instructions. The simulation output instructions are output by the reinforcement learning module at any time step based on the generated simulation attribute features or the predicted simulation attribute features.

[0062] In some embodiments, the attribute prediction module needs to be trained in a simulation environment. The attribute data of the sample object to be interacted with that is related to the interaction can be directly obtained at any time step using the simulation environment, and simulation attribute features can be generated based on the acquired attribute data of the sample object. The splicing features of the current time step and the splicing features of at least one historical time step are input into the attribute prediction module to obtain predicted simulation attribute features. The parameters of the attribute prediction module can be adjusted based on the difference between the generated simulation attribute features and the simulation attribute features predicted by the attribute prediction module.

[0063] For example, the attribute data of the sample object Input the encoding module to get the generated simulation attribute feature z. The encoding module can be a multilayer perceptron (MLP). Input the concatenated features of the current time step and the concatenated features of at least one historical time step into the attribute prediction module to get the predicted simulation attribute feature based on Adjust the parameters of the attribute prediction module, sg[.] represents the stop gradient operator, and λ is the preset parameter.

[0064] In some embodiments, the reinforcement learning module needs to be trained in a simulation environment. The simulation environment can be used to directly obtain the attribute data of the sample object to be interacted with that affects the interaction at any time step, and the simulation attribute features can be generated based on the acquired attribute data of the sample object. The simulation attribute features and splicing features generated at the current time step are input into the reinforcement learning module, or the simulation attribute features and splicing features predicted at the current time step are input into the reinforcement learning module, and the reinforcement learning module outputs a simulation output instruction. After the robot moves in response to the simulation output instruction, it will obtain a motion reward, and the reinforcement learning module will adjust parameters based on the motion reward.

[0065] In the embodiment of the present application, the attribute prediction module and the reinforcement learning module are trained using the attribute data related to the interaction of the sample objects to be interacted in the simulation environment. In this way, the trained attribute prediction module can accurately predict the attribute characteristics of the object to be interacted in the current time step, and the reinforcement learning module can determine the output instructions of the current time step based on the intrinsic attributes of the object to be interacted, so as to better understand the intrinsic characteristics of the object and show a high success rate in the real world.

[0066] Figure 2 This is a schematic diagram of the implementation process of a control method provided in an embodiment of the present application. Figure 2 , the method can be executed by a processor of a computer device. Figure 1 , the method also includes steps S201 and S202.

[0067] Step S201: Store the splicing features of the current time step in a history cache.

[0068] In some embodiments, after obtaining the splicing feature at any time step, the splicing feature of the time step may be stored in a history buffer. The history buffer stores the splicing features of all historical time steps.

[0069] In some embodiments, the history cache stores a preset number of recent splicing features, for example, the history cache stores splicing features of the most recent 10 historical time steps. After obtaining the splicing features of the current time step, the splicing features in the history cache are updated based on the splicing features of the current time step.

[0070] Step S202: Acquire the splicing features of the at least one historical time step from the historical cache.

[0071] In some embodiments, based on the requirements of the attribute prediction module, the splicing features of at least one historical time step are obtained from the historical cache. The splicing features of at least one historical time step can be obtained by reading the historical cache.

[0072] In the embodiment of the present application, the splicing features of the current time step are stored in the history cache; and the splicing features of at least one history time step are obtained from the history cache. In this way, the splicing features of the history time step can be effectively stored and accessed through the history cache, thereby improving the access efficiency of the splicing features.

[0073] In some embodiments, the method further includes: scaling the output instruction to obtain a target control instruction.

[0074] Among them, the output instructions obtained by the reinforcement learning module are not standard action instructions. The robot cannot directly perform actions according to the output instructions. Therefore, the output instructions need to be scaled to obtain the target control instructions. The robot performs actions based on the target control instructions. The output instructions are adjusted or scaled to ensure that they can be safely and effectively executed by the robot, which may involve adjusting parameters such as the speed, strength, and range of the action. The actions after scaling become "robot instructions" that can be understood and executed by the robot. These instructions are then sent to the robot's control system to drive the robot to perform the corresponding actions.

[0075] In some embodiments, the output instruction is scaled based on a preset formula to obtain a target control instruction. For example, the target control instruction c t The calculation formula is formula (1):

[0076] c t =clip(a t ,-1,1)*40+100 (1);

[0077] Among them, a t The clip function is a range limiting function that limits the output instruction to [-1,1].

[0078] In some embodiments, the output instruction is scaled by a pre-trained scaling module, the output instruction is input into the scaling module, and the target control instruction is output.

[0079] In the embodiment of the present application, the output instruction is scaled to obtain the target control instruction. In this way, by scaling the output instruction, the target control instruction that the robot can accurately identify can be obtained.

[0080] In some embodiments, the method further includes: controlling an interactive device based on the target control instruction to execute an interactive action at a current time step.

[0081] The robot interacts with the object to be interacted with through the interaction device, and the interaction device is a part of the robot. Exemplarily, the interaction device is the gripper of the robot.

[0082] The target control instruction includes data related to the interactive device performing the interactive action, and the interactive device can be controlled to perform the interactive action at the current time step based on the target control instruction. The target control instruction is an action instruction that the robot can understand and effectively execute. Exemplarily, the target control instruction includes target displacement, target posture, gripper action, and impedance control parameters.

[0083] In the embodiment of the present application, the interactive device is controlled based on the target control instruction to execute the interactive action of the current time step. In this way, the interactive device can accurately execute the interactive action according to the target control instruction.

[0084] In some embodiments, the method also includes: generating a stiffness matrix of the robot based on the output instruction; constructing an impedance-controlled dynamic model based on the robot's mass inertia matrix, damping matrix and stiffness matrix; the dynamic model is used to control the balance between the robot's motion and contact force.

[0085] Among them, the impedance control principle of the robot is a force-based control method. Its core idea is to regard the robot as a programmable spring-mass-damper system, and to achieve interactive control with the environment by adjusting the impedance of the robot (that is, the resistance of the robot when subjected to external forces). Impedance control is a control strategy that controls the interactive behavior of the robot and the environment by adjusting the dynamic relationship between the robot and the environment (that is, impedance parameters, including stiffness, damping and inertia). It enables the robot to have better adaptability and flexibility when interacting with the environment.

[0086] Among them, stiffness represents the robot's ability to resist external deformation. By adjusting the stiffness, the deformation degree of the robot when subjected to external forces can be controlled. Damping represents the friction or energy dissipation capacity between the robot and the environment. By adjusting the damping, the speed change of the robot when subjected to external forces can be controlled. Inertia represents the robot's response speed to external forces. By adjusting the inertia, the acceleration change of the robot when subjected to external forces can be controlled.

[0087] Among them, the dynamic model of impedance control is a mathematical model based on the stiffness matrix, mass inertia matrix and damping matrix. This model describes the dynamic response characteristics of the robot when subjected to external forces, that is, the balance between the robot's motion state (position, velocity, acceleration) and the contact force. By adjusting the parameters in these matrices, the robot's behavior in the contact environment can be controlled, such as maintaining a stable contact force and avoiding excessive deformation.

[0088] In some embodiments, the output instruction of the reinforcement learning module includes stiffness, the stiffness is scaled to obtain the target stiffness, and the target stiffness is expanded to obtain the stiffness matrix. The dynamic model is constructed based on the adjusted stiffness matrix, the unadjusted mass inertia matrix and the damping matrix. For example, the stiffness output by the reinforcement learning module is The stiffness is converted to the target stiffness by formula (2):

[0089]

[0090] Among them, the clip function is a range limiting function. Limit the target stiffness to [-1,1]. Expand to get the stiffness matrix K.

[0091] For example, a dynamic model is constructed based on the mass inertia matrix M, the damping matrix D and the stiffness matrix K, as shown in formula (3):

[0092]

[0093] Among them, F ext is the external force generated by the interaction between the robot and the environment, x d is the expected trajectory, is the prefetch speed, is the expected acceleration, x c For impedance controlled output trajectory, represents the impedance-controlled acceleration, Represents the speed of impedance control. Based on the dynamic model, the balance between the robot's motion (position, velocity, acceleration) and contact force can be controlled.

[0094] The expected trajectory serves as a target or reference point to guide the robot's movement, while the output trajectory is the result of the robot's adjustment based on the expected trajectory and real-time environmental information during actual operation.

[0095] Figure 3 This is a schematic diagram of the implementation process of a control method provided in an embodiment of the present application. Figure 3 , the method can be executed by a processor of a computer device. Figure 1 , the environment image includes a depth image and a color image, Figure 1 S101 in can be updated to S301 to S304, which will be combined with Figure 3 The steps shown are explained.

[0096] Step S301: Perform object recognition on the color image to obtain a mask image of the object to be interacted with.

[0097] Among them, the depth image mainly expresses the distance information from each point in the scene to the image collector, and the grayscale value or specific value of each pixel represents the distance from the point to the image collector. The color image mainly expresses the color information of the objects in the scene, and the RGB value of each pixel represents the color component of the point, thus forming a colorful image. Exemplarily, the color image is an RGB image.

[0098] Among them, the mask image is a special image used to specify the area to be operated on the original image. The mask image is usually a binary image (each pixel has only two possible values, such as 0 and 255), or it can be a grayscale image or a multi-channel image. In the mask image, the area of ​​interest is set to white (or a higher grayscale value), while the area of ​​no interest remains black (or a lower grayscale value).

[0099] In some embodiments, an image segmentation model is pre-trained, and the color image to be processed is input into the model, and the model processes the input image and outputs a segmentation result, and a binary mask image is generated according to the segmentation result. In this mask image, pixels belonging to the object to be interacted are set to white (or a specific value), and pixels not belonging to the object to be interacted are set to black (or another specific value).

[0100] Step S302: convert the depth image to obtain environment point cloud data.

[0101] Among them, environmental point cloud data is a form of three-dimensional spatial data expression, which is mainly composed of a large number of three-dimensional spatial points, which represent sampling points on the surface of an object or specific positions in space. Each point usually contains three-dimensional coordinate information (X, Y, Z), and sometimes also includes other attribute information such as color, intensity, reflectivity, etc.

[0102] Among them, the depth image and camera parameters are obtained. The camera parameters include the intrinsic parameters and extrinsic parameters of the camera. The intrinsic parameters describe the internal properties of the camera, such as focal length, principal point coordinates, image resolution, and distortion parameters. The extrinsic parameters describe the position and posture of the camera in the world coordinate system.

[0103] In some embodiments, for each pixel in the depth image, the camera intrinsic parameters are used to convert it into normalized plane coordinates; the depth value and the corrected normalized plane coordinates, as well as the depth scale factor, are used to calculate the coordinates of the point cloud in three-dimensional space.

[0104] Step S303: Determine a plurality of second interaction postures using the environmental point cloud data.

[0105] The environmental point cloud data reflects the morphology of the surface of the object and can represent multiple objects in the environment the robot is in. The pre-trained interactive pose determination module can be used to determine the second interactive pose of each object in the environment. Specifically, the environmental point cloud data is input into the interactive pose determination module to output multiple second interactive poses.

[0106] Step S304: Determine the first interaction posture from the plurality of second interaction postures using the mask image.

[0107] The mask image may represent the spatial position of the object to be interacted with in the environment, and the mask image may be used to determine the first interaction posture corresponding to the object to be interacted with from a plurality of second interaction postures.

[0108] In the embodiment of the present application, object recognition is performed on the color image to obtain a mask image of the object to be interacted with; the depth image is converted to obtain environmental point cloud data; multiple second interactive postures are determined using the environmental point cloud data; and the mask image is used to determine the first interactive posture among the multiple second interactive postures. In this way, the first interactive posture of the object to be interacted with can be accurately determined through the depth image and the color image.

[0109] Figure 4 This is a schematic diagram of the implementation process of a control method provided in an embodiment of the present application. Figure 4 , the method can be executed by a processor of a computer device. Figure 1 , the training method of the attribute prediction module includes steps S401 to S404.

[0110] Step S401, based on the simulation output instruction of the first time step and the observation data of the second time step, generate the simulation splicing features of the second time step; the first time step is the previous time step of the second time step.

[0111] Among them, the historical execution actions and current environment state of the robot are represented by simulation splicing features. The historical execution actions are represented by the simulation output instructions of the first time step, and the current environment state is represented by the observation data of the second time step. Therefore, the simulation output instructions of the first time step and the observation data of the second time step are combined into a data pair as the simulation splicing feature of the second time step. The first time step is the previous time step of the second time step. In this way, the latest environment state and the most recent historical execution actions can be represented by the simulation splicing feature of the second time step.

[0112] In some embodiments, the simulation output instructions include target displacement, target posture, gripper action, and impedance control parameters.

[0113] In some embodiments, the observation data includes the first interaction pose, the robot joint configuration, the relative distance between the robot and the object, the end effector pose, and the graspability signal.

[0114] Step S402: input the simulation splicing features of the second time step and the simulation splicing features of at least one historical time step into an attribute prediction module to obtain predicted simulation attribute features; the simulation attribute features are used to characterize attribute data related to the influencing interaction of the sample objects to be interacted in the simulation environment.

[0115] The attribute prediction module is used to predict the simulation attribute features from the simulation splicing features of at least two time steps. The simulation attribute features are used to characterize the attribute data related to the interaction of the sample objects to be interacted in the simulation environment. Therefore, the simulation splicing features of the second time step and the simulation splicing features of at least one historical time step are input into the attribute prediction module to obtain the predicted simulation attribute features. The at least one historical time step includes the first time step.

[0116] In some embodiments, the attribute data includes at least one of: pivot center, pivot radius, object stiffness, object mass, object joint position, handle grasping signal.

[0117] Step S403, input the simulation splicing features and target attribute features of the second time step into the reinforcement learning module to obtain the simulation output instructions of the second time step; the target attribute features are the predicted simulation attribute features or the simulation attribute features generated based on the attribute data of the sample object.

[0118] Among them, the reinforcement learning module is used to comprehensively consider the current environment state, historical execution actions and attribute data of the object to be interacted with, and determine the simulation output instruction. The current environment state and historical execution actions correspond to the simulation splicing features, and the attribute data of the object to be interacted with corresponds to the target attribute features. Therefore, the simulation splicing features and target attribute features of the second time step are input into the reinforcement learning module to obtain the simulation output instruction of the second time step.

[0119] Among them, the target attribute feature is a predicted simulation attribute feature or a simulation attribute feature generated based on the attribute data of the sample object. The predicted simulation attribute feature is the attribute feature output by the attribute prediction module, and the generated simulation attribute feature is the attribute feature obtained based on the attribute data of the sample object. By considering two different ways to determine the target attribute feature, the simulation splicing feature and the target attribute feature of the second time step are input into the reinforcement learning module, which can improve the adaptability of the reinforcement learning module to different attribute features and improve the generalization ability of the reinforcement learning module.

[0120] In some embodiments, the attribute data of the sample object is input into the encoding module to obtain the generated simulated attribute features. The encoding module can be a multi-layer perceptron, which is trained based on the difference between the predicted simulated attribute features and the generated simulated attribute features.

[0121] Step S404: training the attribute prediction module based on the predicted simulation attribute features and the generated simulation attribute features.

[0122] Among them, the attribute prediction module is based on the predicted simulation attribute features The difference between the generated simulation attribute feature z and the training is performed. For example, a supervised regularization loss is formulated based on Adjust the parameters of the attribute prediction module, sg[.] represents the stop gradient operator, and λ is the preset parameter. The training goal is to minimize the loss.

[0123] In the embodiment of the present application, based on the simulation output instruction of the first time step and the observation data of the second time step, the simulation splicing feature of the second time step is generated; the simulation splicing feature of the second time step and the simulation splicing feature of at least one historical time step are input into the attribute prediction module to obtain the predicted simulation attribute feature; the simulation splicing feature of the second time step and the target attribute feature are input into the reinforcement learning module to obtain the simulation output instruction of the second time step; the attribute prediction module is trained based on the predicted simulation attribute feature and the generated simulation attribute feature. In this way, an accurate attribute prediction module can be obtained based on the simulation output instruction and the observation data training.

[0124] Figure 5 This is a schematic diagram of the implementation process of a control method provided in an embodiment of the present application. Figure 5 , the method can be executed by a processor of a computer device. Figure 4 , the training method of the reinforcement learning module includes steps S501 to S502.

[0125] Step S501: After controlling the robot to move based on the simulation output instruction of the second time step, obtaining a movement reward.

[0126] After the reinforcement learning module outputs the simulation output instruction of the second time step, the robot can perform the corresponding action according to the simulation output instruction, and after the robot moves, it will get a motion reward. The motion reward is used to evaluate the quality of the robot's execution of the action.

[0127] In some embodiments, the motion reward includes a task-aware reward and a motion-aware reward. The task-aware reward is used to encourage the robot to perform tasks in the correct order, and the motion-aware reward is used to encourage the robot to generate smooth motion when performing tasks.

[0128] Step S502: training the reinforcement learning module by reinforcement learning based on the sports reward.

[0129] In some embodiments, each time step corresponds to a simulation output instruction, and the robot obtains a motion reward after executing the corresponding action, and the environmental state changes. The environmental state, simulation output instruction, corresponding motion reward, and new environmental state after the action execution at each time step are saved, and the strategy of the reinforcement learning module is updated at each time step based on these data by the reinforcement learning algorithm. Exemplarily, the strategy of the reinforcement learning module is updated by the proximal policy optimization (PPO) algorithm. In this way, at each time step, the strategy is updated by the reinforcement learning algorithm based on the motion reward, and the training process of the reinforcement learning module is stopped when the iteration stop condition is met or the strategy reaches the optimization target.

[0130] In some embodiments, an episode includes multiple time steps, and the environment state, simulation output instructions, corresponding motion rewards, and new environment state after the action is executed at each time step are saved, and the reinforcement learning strategy is updated once in each episode. When the accumulated number of time steps reaches a threshold corresponding to the episode, the strategy is updated based on the motion reward through the reinforcement learning algorithm.

[0131] In the embodiment of the present application, after controlling the robot to move based on the simulation output instruction of the second time step, a motion reward is obtained; and the reinforcement learning module is trained by reinforcement learning based on the motion reward. In this way, the reinforcement learning module can be trained based on the motion reward corresponding to the simulation output instruction to obtain an accurate reinforcement learning module.

[0132] Figure 6 This is a schematic diagram of the implementation process of a control model training method provided in an embodiment of the present application. Figure 1 ,like Figure 6 As shown, the control model includes a property prediction module and a reinforcement learning module; the training method includes steps S601 to S605.

[0133] Step S601, based on the simulation output instruction of the first time step and the observation data of the second time step, generate the simulation splicing features of the second time step; the first time step is the previous time step of the second time step.

[0134] The attribute prediction module is used to infer the attribute characteristics of the object to be interacted from the historical observation data and the output instructions. The attribute characteristics are used to characterize the attribute data of the object to be interacted that are related to the interaction.

[0135] Among them, the reinforcement learning module can output the action that the robot should perform at each time step based on the current environment state and historical actions. The robot performs actions based on the action instructions output by the reinforcement learning module to complete the task.

[0136] Among them, the historical execution actions and current environment state of the robot are represented by simulation splicing features. The historical execution actions are represented by the simulation output instructions of the first time step, and the current environment state is represented by the observation data of the second time step. Therefore, the simulation output instructions of the first time step and the observation data of the second time step are combined into a data pair as the simulation splicing feature of the second time step. The first time step is the previous time step of the second time step. In this way, the latest environment state and the most recent historical execution actions can be represented by the simulation splicing feature of the second time step.

[0137] In some embodiments, the simulation output instructions include target displacement, target posture, gripper action, and impedance control parameters.

[0138] In some embodiments, the observation data includes the first interaction pose, the robot joint configuration, the relative distance between the robot and the object, the end effector pose, and the graspability signal.

[0139] Step S602: input the simulation splicing features of the second time step and the simulation splicing features of at least one historical time step into an attribute prediction module to obtain predicted simulation attribute features; the simulation attribute features are used to characterize attribute data related to the influencing interaction of the sample objects to be interacted in the simulation environment.

[0140] The attribute prediction module is used to predict the simulation attribute features from the simulation splicing features of at least two time steps. The simulation attribute features are used to characterize the attribute data related to the interaction of the sample objects to be interacted in the simulation environment. Therefore, the simulation splicing features of the second time step and the simulation splicing features of at least one historical time step are input into the attribute prediction module to obtain the predicted simulation attribute features. The at least one historical time step includes the first time step.

[0141] In some embodiments, the attribute data includes at least one of: pivot center, pivot radius, object stiffness, object mass, object joint position, handle grasping signal.

[0142] Step S603, input the simulation output instruction and target attribute feature of the second time step into the reinforcement learning module to obtain the simulation output instruction of the second time step; the target attribute feature is the predicted simulation attribute feature or the simulation attribute feature generated based on the attribute data of the sample object.

[0143] Among them, the reinforcement learning module is used to comprehensively consider the current environment state, historical execution actions and attribute data of the object to be interacted with, and determine the simulation output instruction. The current environment state and historical execution actions correspond to the simulation splicing features, and the attribute data of the object to be interacted with corresponds to the target attribute features. Therefore, the simulation splicing features and target attribute features of the second time step are input into the reinforcement learning module to obtain the simulation output instruction of the second time step.

[0144] Among them, the target attribute feature is a predicted simulation attribute feature or a simulation attribute feature generated based on the attribute data of the sample object. The predicted simulation attribute feature is the attribute feature output by the attribute prediction module, and the generated simulation attribute feature is the attribute feature obtained based on the attribute data of the sample object. By considering two different ways to determine the target attribute feature, the simulation splicing feature and the target attribute feature of the second time step are input into the reinforcement learning module, which can improve the adaptability of the reinforcement learning module to different attribute features and improve the generalization ability of the reinforcement learning module.

[0145] Step S604: After controlling the robot to move based on the simulation output instruction of the second time step, obtaining a movement reward.

[0146] After the reinforcement learning module outputs the simulation output instruction of the second time step, the robot can perform the corresponding action according to the simulation output instruction, and after the robot moves, it will get a motion reward. The motion reward is used to evaluate the quality of the robot's execution of the action.

[0147] In some embodiments, the motion reward includes a task-aware reward and a motion-aware reward. The task-aware reward is used to encourage the robot to perform tasks in the correct order, and the motion-aware reward is used to encourage the robot to generate smooth motion when performing tasks.

[0148] Step S605: training the attribute prediction module based on the predicted simulation attribute features and the generated simulation attribute features, and training the reinforcement learning module through reinforcement learning based on the motion reward.

[0149] Among them, the attribute prediction module is based on the predicted simulation attribute features The difference between the generated simulation attribute feature z and the training is performed. For example, a supervised regularization loss is formulated based on Adjust the parameters of the attribute prediction module, sg[.] represents the stop gradient operator, and λ is the preset parameter. The training goal is to minimize the loss.

[0150] In some embodiments, each time step corresponds to a simulation output instruction. After the robot performs the corresponding action, it obtains a motion reward and the environmental state changes. The environmental state, simulation output instruction, corresponding motion reward and new environmental state after the action is executed at each time step are saved, and the strategy of the reinforcement learning module is updated at each time step based on these data through the reinforcement learning algorithm. Exemplarily, the strategy of the reinforcement learning module is updated through the proximal strategy optimization algorithm. In this way, at each time step, the strategy is updated through the reinforcement learning algorithm based on the motion reward, and the training process of the reinforcement learning module is stopped when the iteration stop condition is met or the strategy reaches the optimization target.

[0151] In the embodiment of the present application, based on the simulation output instruction of the first time step and the observation data of the second time step, the simulation splicing feature of the second time step is generated; the simulation splicing feature of the second time step and the simulation splicing feature of at least one historical time step are input into the attribute prediction module to obtain the predicted simulation attribute feature; the simulation output instruction of the second time step and the target attribute feature are input into the reinforcement learning module to obtain the simulation output instruction of the second time step; after controlling the movement of the robot based on the simulation output instruction of the second time step, a motion reward is obtained; the attribute prediction module is trained based on the predicted simulation attribute feature and the generated simulation attribute feature, and the reinforcement learning module is trained by reinforcement learning based on the motion reward. In this way, the attribute prediction module and the reinforcement learning module can be effectively trained to obtain accurate attribute prediction modules and reinforcement learning modules. When outputting instructions, the reinforcement learning module can consider the attribute data, interaction posture and historical output instructions of the object to be interacted that are related to the interaction, reduce the dependence on vision, better understand the intrinsic characteristics of the object, have a higher generalization ability when processing unseen objects, show a high success rate in the real world, and reduce the gap between simulation and reality.

[0152] Figure 7 This is a schematic diagram of the implementation process of a control model training method provided in an embodiment of the present application. Figure 2 , the method can be executed by a processor of a computer device. Figure 6 The control model also includes a coding module, and the training method also includes steps S701 and S702.

[0153] Step S701: Acquire attribute data of the sample object.

[0154] In some embodiments, the attribute data includes at least one of the following: pivot center, pivot radius, object stiffness, object mass, object joint position, and handle grab signal. The pivot center refers to the center point or axis of the rotational motion of an object or system. The pivot radius refers to the distance from the pivot center to a point on the rotating object. Object stiffness refers to the ability of an object to resist deformation when subjected to external forces. Object mass is a physical quantity that describes the magnitude of an object's inertia, and it is a measure of the amount of matter contained in the object. Object joint position is the current posture of the object's joints. The handle grab signal is the current state of whether the handle is grabbed.

[0155] In some embodiments, the attribute data of the object to be interacted with is difficult to obtain in the real world, but in the simulation process, the attribute data can be directly obtained based on the simulation software. The attribute data of the sample object is the attribute data of the second time step.

[0156] Step S702: Input the attribute data of the sample object into the encoding module to generate the simulation attribute feature.

[0157] The encoding module is used to generate simulated attribute features from the attribute data of the sample object.

[0158] In some embodiments, the attribute data of the sample object is input into an encoding module, and the encoding module generates a simulated attribute feature. The encoding module can be a multi-layer perceptron.

[0159] In the embodiment of the present application, the attribute data of the sample object is obtained, and the attribute data of the sample object is input into the encoding module to generate the simulation attribute feature. In this way, accurate simulation attribute features can be extracted from the attribute data of the sample object.

[0160] In some embodiments, the training method further comprises: training the encoding module based on the predicted simulation attribute features and the generated simulation attribute features.

[0161] Among them, the encoding module is based on the predicted simulation attribute features The training is based on the difference between the generated simulation attribute feature z and the predicted simulation attribute feature z. At each time step, the predicted simulation attribute feature z is output by the attribute prediction module. The generated simulation attribute feature z is obtained through the encoding module, based on the predicted simulation attribute feature and the generated simulation attribute feature z, and determine the difference between the two. For example, a supervised regularization loss is formulated based on Adjust the parameters of the encoding module, sg[.] represents the stop gradient operator, and λ is the preset parameter. The goal of training is to minimize the loss.

[0162] In the embodiment of the present application, the encoding module is trained based on the predicted simulation attribute features and the generated simulation attribute features. In this way, the encoding module can be accurately trained.

[0163] In some embodiments, the motion reward includes at least one of the following: a task-aware reward and a motion-aware reward; the task-aware reward is used to encourage the robot to perform a desired motion sequence; the motion-aware reward is used to encourage the robot to perform smooth motion.

[0164] After the robot performs an action at each time step, the system rewards the action according to the reward function and the current environment state. Task-aware rewards are used to encourage the robot to perform the desired motion sequence. For example, the focus of task-aware rewards is to execute the correct motion sequence and follow the order of A first and then B, rather than cheating to get an immediate success reward. For example, at time step t, the door is opened and the door handle is tightly grasped by the gripper. The reward is significantly greater than the state of not grabbing the door handle Motion-aware rewards are used to encourage the robot to perform smooth movements. Motion-aware rewards encourage the robot to generate smooth movements while maintaining a high success rate.

[0165] In the embodiment of the present application, by setting task perception rewards and motion perception rewards, it is possible to ensure that the task execution order is correct and generate smooth movements.

[0166] In some embodiments, the task perception reward includes at least one of the following: a success reward, used to encourage the robot to perform and complete the interaction task; a distance reward, used to encourage reducing the distance between the robot's interaction device and the object to be interacted with; an object state reward, used to encourage the robot to increase the change in the joint position of the object to be interacted with; an interaction reward, used to encourage the robot to perform and complete the target action.

[0167] Among them, for the success reward, after the robot performs the interaction action of the current time step, the distance δ between the robot's interaction device and the object to be interacted is obtained, and the first distance index is determined based on the distance δ and the first interaction indicator And time indicator 1 s , the formula for success reward is When δ≤0.05, is 0.05, otherwise 0; when δ≤0.015∧1 contact In the case of is 0.5, otherwise 0; 1 s The unit time is 1 second. contact is the contact condition, and its value is 1 or 0. For example, for the door opening task, δ is the distance between the robot's gripper and the door handle.

[0168] For the distance reward, the distance δ between the robot's interactive device and the object to be interacted is obtained, and the second distance indicator is determined based on the distance δ. and the second interaction indicator The formula for distance reward is: When δ≤0.015∧1 contact In the case of is 0.8, otherwise is 0.

[0169] Among them, for the object state reward, the distance δ between the robot's interactive device and the object to be interacted with, and the change in the joint position q of the object to be interacted with are obtained. obj , determine the first interaction index based on the distance δ and the third distance indicator and the round length w len , the formula for object state reward is When δ≤0.05, is 0.5, otherwise is 0. For example, for the door opening task, the joint position change q of the object to be interacted with is obj The angle at which the door opens.

[0170] Among them, for interaction, the formula for interaction reward is 0.2*1 g . When δ≤0.015∧1 contact In the case of 1 g is 1, otherwise 1 g For example, for the door opening task, 1 g 1 means the gripper grabs the door handle, 1 g A value of 0 means the gripper is not gripping the door handle.

[0171] Each reward corresponds to a weight, and each reward is multiplied by the corresponding weight, and then added together to obtain the task-aware reward. For example, the weight of the success reward is 40, the weight of the distance reward is 0.6, the weight of the object state reward is 1, and the weight of the interaction reward is 0.05.

[0172] In an embodiment of the present application, by dividing the task-aware rewards into success rewards, distance rewards, object status rewards and interaction rewards, and taking into account the task completion status and the order of action execution, the task-aware rewards can be accurately divided.

[0173] In some embodiments, the motion perception reward includes at least one of the following: an energy reward for encouraging the robot to reduce energy consumption; a tracking position reward for encouraging the robot to reduce the difference between the actual position and the expected position; a tracking rotation reward for encouraging the robot to reduce the difference between the actual angle and the expected angle; a smoothness reward for encouraging the robot to perform smooth movements; and a direction reward for encouraging the robot to reduce the movement deviation in the said direction.

[0174] The direction reward includes: a first direction reward, which is used to encourage the robot to reduce the movement deviation in the first direction; and a second direction reward, which is used to encourage the robot to reduce the movement deviation in the second direction.

[0175] Among them, for the energy reward, obtain the joint torque τ of the robot's interactive device and the joint speed The formula for energy bonus is The weight is -0.05. For example, for the door opening task, when the gripper grabs the door handle, the smaller the joint torque and joint velocity of the robot are, the better.

[0176] Among them, for the tracking position reward, get the robot's expected position c pos and the real location ee pos , the formula for tracking position reward is exp(-4(c pos -ee pos ))*1 d , with a weight of 0.025. When δ≤0.05, 1 d is 1, otherwise 1 d is 0. The robot's expected position c pos and the real location ee pos The smaller the difference, the higher the tracking position reward.

[0177] Among them, for tracking rotation reward, get the expected rotation angle c of the robot ori and the actual rotation angle ee ori , the formula for tracking the spin reward is exp(-4Δ(c ori -ee ori ))*1 d , with a weight of 0.004. When δ≤0.05, 1 d is 1, otherwise 1 d is 0. The robot's expected rotation angle c ori and the actual rotation angle ee ori The smaller the difference, the higher the track spin reward.

[0178] Among them, for the smoothness reward, get the action a of the current time step tand the action a at the previous time step t-1 , the formula for smoothness reward is The weight is -0.001. t ) is a t The symbol, In sgn(a t )≠sgn(a t-1 ) takes the value 1, otherwise takes the value 0. The action a at the current time step t and the action a at the previous time step t-1 The smaller the difference, the smoother the robot's movements.

[0179] Among them, for the first direction reward, obtain the action component a in the first direction t [y], the formula for the first direction reward is 1 dy *(a t [y]*15) 2 , with a weight of -0.005. For example, the first direction is along the y-axis, and the action component a in the first direction is t The smaller [y], the better. In the case of 0.02≤δ≤0.08, 1 dy The value is 1, otherwise 1 dy The value is 0.

[0180] Among them, for the second direction reward, obtain the action component a in the second direction t [z], the formula for the second direction reward is 1 g *(a t [z]*15) 2 , with a weight of -0.07. For example, the second direction is along the z-axis, and the action component a in the second direction is t [z]The smaller the better.

[0181] Among them, each reward corresponds to a weight. Each reward is multiplied by the corresponding weight, and then added up to obtain the task-aware reward.

[0182] In the embodiment of the present application, the motion perception reward can be accurately divided by dividing the motion perception reward into energy reward, tracking position reward, tracking rotation reward, smoothness reward, first direction reward and second direction reward.

[0183] In some embodiments, the simulation output instruction based on the second time step controls the movement of the robot, including: scaling the simulation output instruction to obtain a simulation control instruction; and controlling the movement of the robot based on the simulation control instruction.

[0184] Among them, the simulation output instructions obtained by the reinforcement learning module are not standard action instructions. The robot cannot directly perform actions according to the simulation output instructions. Therefore, the simulation output instructions need to be scaled to obtain simulation control instructions. The robot performs actions based on the simulation control instructions. The simulation output instructions are adjusted or scaled to ensure that they can be safely and effectively executed by the robot, which may involve adjusting parameters such as the speed, strength, and range of the action. After the scaling process, the action becomes a "robot instruction" that can be understood and executed by the robot. These instructions are then sent to the robot's control system to drive the robot to perform the corresponding actions.

[0185] In some embodiments, the simulation output instruction is scaled based on a preset formula to obtain a simulation control instruction. t The calculation formula is formula (4):

[0186] c t =clip(a t ,-1,1)*40+100 (4);

[0187] Among them, a t To simulate output instructions, the clip function is a range limiting function that limits the simulation output instructions to [-1,1].

[0188] In some embodiments, the simulation output instruction is scaled by a pre-trained scaling module, the simulation output instruction is input into the scaling module, and the simulation control instruction is output.

[0189] The simulation control instruction includes data related to the robot's interactive action, and the robot can be controlled to perform the interactive action at the current time step based on the simulation control instruction. The simulation control instruction is an action instruction that the robot can understand and effectively execute. Exemplarily, the simulation control instruction includes target displacement, target posture, gripper action, and impedance control parameters.

[0190] In the embodiment of the present application, the simulation output instruction is scaled to obtain the simulation control instruction, and the robot movement is controlled based on the simulation control instruction. In this way, by scaling the simulation output instruction, the simulation control instruction that the robot can accurately recognize can be obtained. The robot can accurately perform interactive actions according to the simulation control instruction.

[0191] In some embodiments, the method also includes: generating a stiffness matrix of the robot based on the simulation output instructions; constructing an impedance-controlled dynamic model based on the robot's mass inertia matrix, damping matrix and stiffness matrix; the dynamic model is used to control the balance between the robot's motion and contact force.

[0192] Among them, stiffness represents the robot's ability to resist external deformation. By adjusting the stiffness, the deformation degree of the robot when subjected to external forces can be controlled. Damping represents the friction or energy dissipation capacity between the robot and the environment. By adjusting the damping, the speed change of the robot when subjected to external forces can be controlled. Inertia represents the robot's response speed to external forces. By adjusting the inertia, the acceleration change of the robot when subjected to external forces can be controlled.

[0193] Among them, the dynamic model of impedance control is a mathematical model based on the stiffness matrix, mass inertia matrix and damping matrix. This model describes the dynamic response characteristics of the robot when subjected to external forces, that is, the balance between the robot's motion state (position, velocity, acceleration) and the contact force. By adjusting the parameters in these matrices, the robot's behavior in the contact environment can be controlled, such as maintaining a stable contact force and avoiding excessive deformation.

[0194] In some embodiments, the simulation output instruction of the reinforcement learning module includes stiffness, the stiffness is scaled to obtain a target stiffness, and the target stiffness is expanded to obtain a stiffness matrix. A dynamic model is constructed based on the adjusted stiffness matrix, the unadjusted mass inertia matrix, and the damping matrix.

[0195] In the embodiment of the present application, the stiffness matrix of the robot is generated based on the simulation output instruction; and the dynamic model of impedance control is constructed based on the mass inertia matrix, damping matrix and stiffness matrix of the robot. In this way, the stiffness parameters are output by the reinforcement learning module, and impedance control is applied to the movement of the robot to ensure that the robot can perform smooth movements.

[0196] The following describes the application of the control method and control model training method provided in the embodiments of the present application in actual scenarios.

[0197] The embodiment of the present application uses observation history as the input of reinforcement learning, replacing point cloud features, to reduce the gap between simulation and reality and better understand the intrinsic characteristics of objects. Impedance control is used to achieve seamless switching from simulation to reality, thereby obtaining smooth and compliant movement.

[0198] In some embodiments, articulated object manipulation is a unique challenge compared to rigid object manipulation because the objects themselves represent a dynamic environment. A novel reinforcement learning (RL)-based pipeline equipped with variable impedance control and motion adaptation is proposed for general articulated object manipulation using observation history, focusing on smooth and dexterous motion in zero-shot simulation-to-real transfer. To reduce the simulation-to-real gap, the pipeline does not directly utilize visual data features (image RGBD / point cloud) as policy input, but first extracts useful low-dimensional data through off-the-shelf modules, thereby reducing the reliance on vision. In addition, the object motion and its intrinsic properties are inferred from observation history, and impedance control is applied in both simulation and real world, thus reducing the gap from simulation to reality. In addition, the object motion and its intrinsic properties are inferred from observation history, and impedance control is applied in both simulation and real world, thus reducing the gap from simulation to reality. In addition, a carefully designed training environment with large randomness and specialized reward systems (task-aware and motion-aware) is developed to achieve multi-stage, end-to-end manipulation without heuristic motion planning. Through extensive experiments on a variety of objects, the trained policy achieves an 84% success rate in the real world.

[0199] Figure 8 Schematic diagram of the application of the reinforcement learning strategy provided in the embodiment of the present application Figure 1 .like Figure 8 As shown, an RL policy for opening doors and drawers is trained in simulation, which adjusts its output actions based on the motion of the object by leveraging historical observations. This policy is directly applied to closed-loop variable impedance control in the real world to achieve 80% joint limits and an 84% success rate, using only one first frame RGBD image.

[0200] Among them, in a diverse training setting, the observation history 801 is input into the adaptation module 802, the adaptation module 802 outputs the action understanding 803, and then obtains the strategy action 804; the strategy action 804 is input into the impedance control module 805, and the impedance control module 805 outputs the robot command 806; the trained strategy is applied to the real world, and the trained strategy is used to operate a variety of objects 807 using only one RGBD image.

[0201] General purpose robotics is a major milestone in the field of robotic learning and has the potential to revolutionize everyday life. As articulated objects become ubiquitous in domestic and industrial settings, learning how to manipulate them effectively is one of the main challenges in achieving this goal. Despite tremendous progress in the field of embodied AI, generalizable articulated object manipulation remains an unsolved problem for a variety of reasons. One of the main challenges is that true articulation properties (e.g., pivot center, friction, stiffness) can only be identified after physical contact. For example, two objects may appear identical but have very different physical properties. Therefore, in order to achieve a generalizable articulated object manipulation pipeline that can seamlessly interact with unseen objects, it is necessary to build a closed-loop pipeline that adaptively infers these features during the manipulation phase. Another difficulty lies in the object's joint constraints, which require that the applied actions must conform to the actual object's joint motion. If the robot's actions are not tolerant of joint motion and prioritize completing the given instructions, large forces may be generated, causing damage to both the object and the robot.

[0202] In the related art, articulated object manipulation usually relies on visual information as the main input of its pipeline. The visual input of the first frame (in the form of point cloud or RGB image) is used to predict the operable part and the operation sequence or waypoint trajectory. Then, the sequence or waypoint trajectory is directly executed in an open-loop manner without considering any physical interaction that may occur with the object. Although this mode is naturally intuitive and ignores the intrinsic properties of the object, it may lead to unsafe behavior. Other related technologies use RL backbones to output actions in a closed-loop manner based on visual feedback. However, since this pipeline relies heavily on visual feedback at each iteration, there will be a huge gap between the visual simulation left by the visual module and the reality. In addition, during the operation phase, this method may output suboptimal operations due to the occlusion of the operable part. Related technologies try to use impedance control as a ready-made underlying controller to adaptively adjust the predicted waypoints according to some sample-based heuristic rules. However, this method only affects the local trajectory between two predefined set points, resulting in non-smooth motion.

[0203] In an embodiment of the present application, closed-loop RL is combined with learnable impedance control. First, the observation history is used to close the loop control of the object instead of visual input. Take the example of how humans open a door in the dark to demonstrate the intuition: given information about the position of the door handle and whether the door is on the left or right hinge, the circular motion of the door is estimated based on the actions taken and the actual movement of the door. Then, even without direct visual input, the next action is gradually adjusted based on this feedback information to complete the task. Based on this intuition, using observation history and reducing dependence on vision has two benefits: 1) Using vision only as auxiliary information can reduce the gap between visual simulation and reality; 2) Using observation and action history, the movement of the object can be implicitly learned based on the position error after each execution, thereby realizing a universal closed-loop pipeline.

[0204] Second, variable impedance control is introduced into the pipeline, addressing the importance of compliant motion in articulated object manipulation. Impedance control is applicable to tasks with high tolerance requirements for equilibrium setpoint tracking and object joint motion, which fundamentally distinguishes articulated object manipulation from rigid object manipulation. While implementing a high-frequency variable impedance controller in simulation, its parameters are also learned in conjunction with an RL policy. The inclusion of impedance control in a carefully designed training setup enables the policy to learn smooth, continuous motion that complies with the object's joint motion. Learning motions rather than single actions or discrete waypoints can achieve higher success rates in real-world scenarios.

[0205] The contributions of the embodiments of the present application are summarized as follows: A novel RL-based articulated object manipulation pipeline is proposed, with observation and action history as the main input, and vision only as auxiliary information. A training environment is designed in which each component simulates reality, and a reward function system is designed to achieve smooth multi-stage end-to-end manipulation without any heuristic motion planning. A variable impedance controller is introduced into RL to improve the tolerance to object motion, thereby facilitating direct transmission from simulation to reality. Through extensive experiments on 4 tasks and 500 promotions in the real world, the success rate of zero-point reasoning in simulation and the real world reached 96% and 84%, respectively, and it has high versatility for a wide range of objects.

[0206] In some embodiments, manipulating articulated objects is extremely challenging due to the wide variety of geometric shapes and physical properties of objects. The benchmarks that have been achieved in the related art are still limited in coverage. Research on the manipulation of articulated objects can be roughly divided into capability-based methods and RL-based methods. Capability-based methods rely on visual capability heat maps, where each point corresponds to the success rate of manipulation, thereby selecting contact points and predicting actions. However, these heat maps are often ambiguous and difficult to accurately label. Due to the gap between visual simulation and reality, they cannot be generalized well. On the other hand, RL-based methods show better generalization capabilities. However, using point cloud features as input to the policy makes the exploration space vast and the task complex. In addition, training with unrealistic entities, such as flying grippers or movable Franka robots, to relax restrictions on inverse dynamics solvers (IK solvers) or collision avoidance conditions simplifies tasks in the real world. The embodiment of the present application only uses low-dimensional visual information captured by the first frame and combines historical observations in the manipulation stage to better understand the motion of the object using RL.

[0207] In some embodiments, impedance control belongs to the family of position-force control, in which position and force are not decoupled but processed simultaneously, thereby improving tolerance to feedback force while maintaining good tracking. Many contact-rich robotic tasks, such as object placement or tool assembly, have successfully demonstrated the compatibility of such controllers with tasks that simultaneously consider position set point tracking and object-robot force constraints. For learning-based methods, the relevant technology uses impedance control as a ready-made low-level controller for executing downstream instructions under policy guidance. Impedance control parameters are directly incorporated into RL, inverse RL or analytical optimization methods as learnable variables. Compared with manually adjusted impedance control, variable impedance control is more suitable for different task settings and has lower labor costs. In the embodiments of the present application, the application of impedance control is extended to the manipulation of articulated objects by learning impedance control parameters in training and applying them directly to the real world.

[0208] In some embodiments, given a hinged object O (corresponding to the object to be interacted with in the above embodiment) and an operation task θ, a strategy π is trained to output a dexterous action each time to complete the task in a closed-loop manner. The task is a more challenging and realistic adjustment based on the relevant technology. For pulling tasks (opening doors, drawers), the strategy is required to reach, grasp the operable part, and then open it until the joint position of the object reaches at least 80% of the joint limit, rather than about half of the position. Since the robot needs to track the rigid body motion (including translation and rotation) of the object in the actual three-dimensional space, this criterion, especially when applied to rotational joints, requires the robot to perform a lot of dexterous and long-term movements. In addition, in the setting, only robots (such as Franka robots with fixed bases) are allowed to adopt realistic inverse kinematics (IK) configurations, and unlike other path point prediction pipelines using flying grippers or suction cups, the absolute feasibility of predicted motion is assumed.

[0209] In some embodiments, the designed framework aims to predict one dexterous action at a time, rather than a series of short primitive actions. At time step t, the output action a of the reinforcement learning policy (corresponding to the reinforcement learning module in the above embodiment) t ∈R 11 (corresponding to the output instructions in the above embodiment) including the target displacement Target posture R t ∈R 6 , Gripper action G t ∈R 1 and impedance control parameters Subsequently, these actions a t will be converted into robot commands c using the motion scaler t ∈R 9 (corresponding to the target control instruction in the above embodiment).

[0210] In some embodiments, at time step t, the observation index o is obtained. t (corresponding to the observation data in the above embodiment), including the target grasping posture g t ∈R 7 (corresponding to the first interactive posture in the above embodiment), robot joint configuration q t ∈R 7 , the relative distance between the robot and the object δ t ∈R 1 , end effector position ee t ∈R 9 Includes 3D position and 6D rotation, graspability signals

[0211] Among them, in simulation, the target grasping pose is directly inferred from the handle bounding box, while in reality it is obtained using a ready-made grasp prediction module. In simulation, the robot joint configuration can be obtained based on the simulation software, and in reality it can be obtained based on the robot's control system. The relative distance between the robot and the object can be obtained based on the simulation software in simulation and through sensors in reality. The graspability signal is based on the conditions of distance and contact perception, rather than a direct command to control the opening and closing of the gripper. For the graspability signal, it can be obtained based on the simulation software in simulation and through sensors in reality.

[0212] Among them, the observation index o t It also includes task-related observations, such as the pivot center with noise added in the door opening task. Pivot radius and right hinge boolean Therefore, the observation index o t Formula (5):

[0213]

[0214] In some embodiments, privileged observations are only used in simulations to better understand the environment. (corresponding to the attribute data in the above embodiment), which is a value that is difficult to track in the real world. Include Pivot Center Pivot radius Object stiffness Object mass Object joint positions Handle grab signal Therefore, the privileged observation Formula (6):

[0215]

[0216] In some embodiments, articulated object manipulation is a unique challenge compared to rigid object manipulation because the object itself is a dynamic environment. The motion of the object can only be observed through physical interactions, or the joint ground truth position is hidden inside the object, which is similar to motion tasks where environmental parameters (such as terrain friction, slope) are difficult to predict. To this end, an online policy refinement pipeline is adopted, which is widely used in motion tasks and learns two independent modules: an adaptation module σ (corresponding to the attribute prediction module in the above embodiments) and a privileged observation encoder module φ (corresponding to the encoding module in the above embodiments). Privileged observations (such as pivot center, object stiffness, object mass, etc.) are used in simulation, these features are extracted by the privileged observation encoder module, and the adaptation module is trained to infer this information from historical observation and action pairs.

[0217] The privileged observation encoder module φ is a shallow MLP that is used to learn the latent representation z of privileged observations during training. t (corresponding to the generated simulation attribute features in the above embodiment). This 20-dimensional vector is consistent with the (observation, action) pair of the current time step. The adaptation module σ is designed as a temporal architecture, starting from H = 10 p t Extract the potential information about the environment from the pair (corresponding to the predicted simulation attribute features in the above embodiment). Only retain part of the action history as the input of σ: target displacement Gripper action G t and impedance control parameters

[0218] Among them, since the traditional teacher-student two-stage pipeline may lead to the feasibility gap and simulation-reality gap, the adaptation module and the privileged observation encoder module are trained simultaneously in one training. Specifically, when jointly training the adaptation module with the reinforcement learning RL backbone, it also learns to extract similar privileged information from the history buffer The method is to formulate a supervised regularization loss on top of the PPO objective. (sg[.] stands for stop gradient operator.) Linear scheduling of λ is performed to prevent the policy from taking conservative actions at the beginning.

[0219] In some embodiments, although the proposed framework has been widely used for motion tasks, how to use this pipeline for fine manipulation tasks (such as articulated object manipulation) remains a challenge. In order to facilitate the use of a single end-to-end strategy to efficiently execute multi-stage motion, stage-conditional rewards are introduced, including task-aware rewards and motion-aware rewards. After the robot performs an action at each time step, the system rewards the action based on the reward function and the current environment state. A task- and action-related reward system is developed to ensure the correct order of task execution and encourage the strategy to generate smooth movements.

[0220] Among them, the focus of task-aware rewards is to execute the correct sequence of movements and follow the order of A first and then B, rather than cheating to get immediate success rewards. For example, at time step t, the state where the door is opened and the door handle is tightly grasped by the gripper The reward is significantly greater than the state of not grabbing the door handle

[0221] Among them, motion-aware rewards encourage the policy to generate smooth motion while maintaining a high success rate. These terms are usually activated after the policy is trained to complete the main task, thus playing a fine-tuning role and incentivizing the policy to perform more smoothly. Including these regularization terms is crucial and helps to bridge the gap between simulation and reality by preventing unnecessary motion or unattainable target poses.

[0222] In some embodiments, Table 1 is a reward function. As shown in Table 1, it includes: 1 d The corresponding conditions are δ≤0.05, 1 dy The corresponding conditions are 0.02≤δ≤0.08, 1 g The corresponding condition is δ≤0.015∧1 contact , τ is the joint torque, is the joint velocity, w len is the round length weight, a t [y] is the action on the y-axis, a t [z] is the action on the z axis. For task-aware rewards, the corresponding formula for "success" is The weight is 40.0; the corresponding formula for "distance" is The weight is 0.6; the corresponding formula for "object state" The weight is 1.0; the corresponding formula for "grab" is 0.2*1 g , with a weight of 0.05. For the action-aware reward, the corresponding formula for “energy” is The weight is -0.05; the corresponding formula for "tracking position" is exp(-4(c pos -ee pos ))*1 d , with a weight of 0.025; the corresponding formula for "tracking rotation" is exp(-4Δ(c ori -ee ori ))*1 d , with a weight of 0.004; the corresponding formula for "smoothness" is The weight is -0.001; "y regularization" corresponds to formula 1 dy *(a t [y]*15) 2 , weight is -0.005; "z regularization" corresponds to formula 1 g *(a t [z]*15) 2 , with a weight of -0.07.

[0223] Among them, for When δ≤0.05 is 0.05, when δ>0.05, is 0; for When δ≤0.015∧1contact hour is 0.5, under other conditions, 0; 1 s Indicates 1 second. d , 1 dy , 1 g , takes the value 1 when the corresponding condition is met, and takes the value 0 when the corresponding condition is not met. c pos is the expected position of the strategy output, ee pos is the actual position; c ori is the desired rotation angle output by the strategy, ee ori is the actual rotation angle.

[0224] Among them, for task-aware rewards, the corresponding formula of each reward indicator is multiplied by the corresponding weight and then added to obtain the task-aware reward; for action-aware rewards, the corresponding formula of each reward indicator is multiplied by the corresponding weight and then added to obtain the action-aware reward.

[0225] Table 1

[0226]

[0227]

[0228] In some embodiments, training policies with domain randomization may facilitate simulation-to-reality transfer. The main focus is on addressing the physics gap problem, requiring the policy to understand the motion of an object through its interaction with the robot, while the intrinsic properties of the object are noisy. In simulation, training is performed by randomizing parameters such as friction, stiffness, mass, and a high-frequency variable impedance controller is used to perform predicted actions. During training, object positions and object yaw rotations are randomized to cover a reasonable workspace for real-world settings. In terms of physical intrinsics, joint friction, stiffness, and mass are varied to achieve a more robust simulation-to-reality transition. For the desired grasping pose, after inferring the pose from the part bounding box, random noise is introduced along the y-axis and z-axis, and a random rotation target is introduced from a predefined spherical cone.

[0229] In some embodiments, the goal of impedance control is to take into account the external force F generated by the interaction between the robot and the environment. ext In the case of d The impedance control is designed using a mass-spring-damper system that dynamically adjusts the target set point based on the feedback force and the environmental stiffness. The dynamic model of the impedance control is formula (7):

[0230]

[0231] Among them, M is the mass inertia matrix of the robot, D is the damping matrix, K is the stiffness matrix, is the impedance trace output. represents acceleration, represents speed, x c Indicates location.

[0232] In some embodiments, in the pipeline, learning predicts the stiffness coefficient k of the Cartesian impedance controller p , and expand it into a six-dimensional diagonal matrix K. Assume that M, K, and D are positive definite diagonal matrices to ensure system stability. The k predicted by the strategy is given by formula (8) p To scale:

[0233]

[0234] Among them, this range of values ​​can produce reasonable motion in both simulation and actual experiments. According to the stiffness matrix K, it can be inferred that there is a critical damping condition The damping matrix.

[0235] In some embodiments, in the policy output Then, the formula (8) is used to calculate Will Expand to get K, and M and D are known, so we can get F ext Based on F ext Impedance control can be applied to the robot's movements.

[0236] Fig. 9 Schematic diagram of the application of the reinforcement learning strategy provided in the embodiment of the present application Figure 2 .like Fig. 9 As shown, during the simulation, the privileged observation encoder module φ is trained to extract the latent representation z of the privileged observation t , and train the adaptation module σ from H = 10 previous (o t , a t-1 ) to infer similar privileged information Then, the latent representation z t The pair p with the current time step t Concatenated to form the policy input. The trained policy is executed in an end-to-end manner in the real world based on the adaptation module σ to perform reaching, grasping, and manipulation operations. Using an RGBD image captured in the first frame, the desired grasping pose is extracted through an off-the-shelf vision module. The reinforcement learning policy is trained in simulation and directly transferred to the real world.

[0237] Among them, during the simulation training process, at time step t, the observation index o of the current time step is t (901) and the action a at the previous time step t-1t-1 (902) constitutes an observation-action pair p t (903), p t (903) is stored in the history buffer 904. The adaptation module 905 adapts the H observation-action pairs (p t-1 to p t-H ) Extract similar privileged information (906). The privileged observation encoder module 907 (908) Learning the latent representation z of privileged observations t (909). Randomly set p t and Combination, or p t With z t The combination is used as the input of the reinforcement learning module 910. The downstream strategy can be adapted to z t and At each time step t, the reinforcement learning module outputs action a t (911), for a t Zoom in and get the robot command c t (912), c t The input impedance control system 913 applies impedance control during the execution of the action. The adaptation module and the privileged observation encoder module are based on z t and The reinforcement learning module is trained based on the difference between the actions, and the reinforcement learning module is trained through the PPO algorithm based on the rewards corresponding to the actions.

[0238] In the real world application, a RGB image 914 and a corresponding depth image 915 are obtained through the robot's camera, and a point cloud 916 corresponding to the environment is obtained based on the depth image. The RGB image is input into the first module (SAM) 917, and a mask image of the image to be grasped is output; the point cloud is input into the second module (GSNet) 918, and multiple grasping postures are output; the mask image of the image to be grasped is used to determine the target grasping posture from multiple grasping postures 919. The observation index o of the current time step containing the target grasping posture is t (920) and the output action a of the previous time step t-1 (921) constitutes the observation-action pair p t (922), p t Stored in the history buffer 923. The adaptation module 924 adapts the H observation-action pairs (p t-1 to p t-H ) Extract similar privileged information (925). and p tThe combined input is sent to the reinforcement learning module 926, which outputs the action a at the current time step t (927), for a t Zoom in and get the robot command c t (928), c t An impedance control system 929 is input to apply impedance control during the execution of the action.

[0239] In some embodiments, extensive evaluations were performed in simulation and real-world environments to verify the effectiveness of the proposed method. For data and task settings, in simulation, experiments were conducted in the IsaacGym simulator and the large-scale PartNet-Mobility dataset following the settings of the related art. A Franka robot with a fixed base and a total of 346 articulated 3D objects, including doors and drawers (modified furniture), were used for simulation experiments. In a real environment, experiments were conducted on various household objects using a Franka Emika robotic arm equipped with a handheld RealSense D415 camera to capture RGBD images. The off-the-shelf module Segment Anything (SAM) and the off-the-shelf module GSNet were used to extract the operable part point cloud of the first frame of the RGBD image and perform grasping prediction.

[0240] Among them, the proposed pipeline is evaluated with the following two tasks: OpenDoor / OpenDoor+ and OpenDrawer / OpenDrawer+. OpenDoor / OpenDoor+: A door is initially closed, and the robot needs to open the door by more than 15% or 80% of the maximum door swing. The key requirement of the task setting is that the gripper should firmly grasp the door handle when opening the door, and cannot cheat by opening the door from the side or using the robot body to open the door. OpenDrawer / OpenDrawer+: The drawer is initially closed, and the robot needs to open the drawer by more than 20% or 80% of the maximum opening length. Similar to the door opening task, the gripper is required to firmly grasp the handle when opening the drawer. The success rate (SR) is used as the main evaluation metric in both simulation and real-world settings.

[0241] In some embodiments, the proposed method is compared with an articulated object manipulation pipeline that adopts a simulation-to-real RL paradigm. Comparative Method 1: A proximal policy optimization algorithm is directly used to learn a state-based policy to handle each task. The detailed PPO parameters and training strategies are similar to the methods of the embodiments of the present application. Comparative Method 2: This is an affordability learning framework that uses partial point clouds to predict the affordability of visual operability, incorporates partial masks as additional dimensions into the task, and other aspects remain unchanged. Comparative Method 3: This is a vision-based policy learning method that first trains a state-based expert using part-based normalization and part-aware rewards, and then refines the knowledge into a vision-based student policy. Comparative Method 4: This is a pure image learning method that uses a hand-eye monocular camera to actively perceive articulated objects from multiple angles to improve the accuracy of 6D poses. Comparative Method 5: A vision-based method that first performs cross-category part segmentation and pose estimation, and then uses the predicted part pose for heuristic manipulation.

[0242] Among them, four comprehensive ablation studies are conducted to highlight the contribution and effectiveness of each module: No-strategy distillation: only using the current time step o t The policy is trained with observations from the original dataset, omitting the adaptation module and the privileged observation encoder module. No variable impedance control: Cartesian position control is used as a low-level controller for the policy. No regularization: Motion-aware rewards are excluded from the reward function. No randomization: All forms of randomization are excluded, including object pose, desired grasp pose, friction, stiffness, mass, and noise intrinsics.

[0243] In some embodiments, the results of the simulation experiments are shown in Table 2. It can be seen that although most baseline methods perform well on the training set, their performance on the test set has a clear downward trend. In contrast, the performance of the method of the embodiment of the present application on the evaluation set remains strong without a sharp drop, which highlights the excellent generalization ability of the method of the embodiment of the present application. Even without any direct gain reward, the controller can learn to adapt to different operation stages.

[0244] Table 2

[0245]

[0246]

[0247] Fig.10 Schematic diagram of the experimental setup of the present invention. The strategy was extensively evaluated in the real world, with a large number of unseen objects that differed in appearance 1001, size 1002, hinge orientation 1003, hinge stiffness 1004, and 6D pose 1005. Fig.10 As shown, Fig.10 The different images in show that different objects vary in appearance 1001, size 1002, hinge orientation 1003, hinge stiffness 1004, and 6D pose 1005. Our performance is demonstrated in a reasonable workspace, with the test objects facing straight on or slightly tilted about the z-axis.

[0248] Fig.11 Schematic diagram of experimental results of an embodiment of the present application. Fig.11 The horizontal axis is the time frame number. The larger the frame number, the later the time. The vertical axis is the stiffness parameter k p It can be seen that in the original k p When it is greater than 1, the original k p Scaling, scaling k p is 140; in the original k p When it is less than -1, the original k p Scaling, scaling k p is 60. At the initial time, the robot's stiffness parameter k p Large, as time goes by, the robot's stiffness parameter k increases as it approaches the object. p It will become smaller and the robot arm will become softer.

[0249] In some embodiments, from Fig.11 It can be seen that the learned impedance control parameters can actively adapt to the operation phase even without direct gain reward: the impedance control becomes harder during approaching the target and becomes softer during opening the target. Fig.11 It can be seen that when the manipulator is far away from the object, by setting the impedance control parameter to a higher k p , the manipulator becomes harder; on the other hand, when the distance is shortened to minimize the collision penalty, the manipulator becomes softer, k p The smaller.

[0250] In some embodiments, the performance of the policy output in the real world is listed in Table 3. 50 experiments were conducted on different objects for the pipeline and each ablation model (500 runs in total).

[0251] Table 3

[0252]

[0253] In some embodiments, based on Fig.10, we further investigated the success rate by distinguishing cases due to grasp pose estimation failures during the grasping phase and cases due to pipeline failures during the opening phase. For OpenDoor+, we found that 6 / 50 of the inferences failed during the grasping phase, while only 4 / 50 failed during the opening phase, suggesting that the strategy could have yielded a success rate of 40 / 44 = 0.90 if a stable grasp pose was initiated. For OpenDrawer+, 7 / 8 of the failure cases were due to unsuccessful grasps.

[0254] In some embodiments, the ablation study results shown in Table 3 show that in addition to the SR drop in simulation and real world, some other issues were found, aiming to highlight the non-smooth motion execution in the real world. For the control without variable impedance, the main reason for the failure cases (40% drop) was found to be the low flexibility of position control, which requires precise execution of each predicted action. This will generate large joint torques to overcome the feedback force of the object, causing the robot arm to be triggered to stop. In simulation, this behavior does not seem to seriously affect the performance, as evidenced by the success rate greater than 0.8.

[0255] However, in the real world, large torques can be very dangerous and trigger emergency stops, which emphasizes the necessity of impedance control. Even with manual adjustments to the baseline values ​​for impedance control, “No Policy Distillation” and “No Randomization” tend to fall by the wayside. We believe this behavior is due to the gap between the physical simulation and reality caused by the non-diverse training setting and short-term observations. In the “No Regularization” case, the reaching and opening movements are stiff, which is highly undesirable and leads to grasp failures and loss of contact during execution.

[0256] In some embodiments, the modified StorageFurniture subset of PartNet-Mobility was evaluated in simulation, containing 346 objects, with a success rate of 93% for door opening tasks and 96% for drawer opening tasks. In the real world, 10 different articulated objects were used for verification, which varied in appearance, size, hinge orientation, hinge stiffness, and 6D pose. The strategy achieved an 80% success rate in door opening tasks and an 84% success rate in drawer opening tasks. The strategy showed good generalization ability on unseen objects, with a success rate higher than all previous work. In addition, smooth and continuous motions that have not been seen in previous work were generated.

[0257] The embodiment of the present application introduces a reliable RL strategy and deploys it seamlessly into various real-world environments. Experiments in simulation and real-world scenarios show that in simulation, the operation phase should be learned as a smooth, continuous motion rather than a discrete waypoint. Coupled with the tolerance of impedance control, closed-loop real-world transmission will be more efficient even if the action prediction is slightly imperfect.

[0258] The embodiment of the present application introduces a new RL framework equipped with variable impedance control for end-to-end articulated object manipulation, which can adaptively learn object motion through observation and action history records, rather than naively executing the trajectory predicted before the robot contacts the object. Powerful simulation-to-reality conversion capabilities are demonstrated on various test objects in the real world, and success rates of 80% and 84% are achieved in door opening tasks and drawer opening tasks, respectively, which are better than related technologies. While obtaining quantitative results, the strategy is able to produce smooth and dexterous movements due to the careful design of training settings and reward functions. The embodiment of the present application can provide an alternative method for utilizing visual information and other potential modes (such as tactile grasping signals), thereby better bridging the gap between simulation and reality for future RL-based manipulation work.

[0259] In some embodiments, the application scenario in which this solution has been used is the opening and closing of doors and drawers in home scenarios. Implementation: Through simulation and real-world reinforcement learning strategies, the robot is trained to recognize the 6D posture of doors and drawers, and high-frequency variable impedance control is used to ensure smooth movement. The grasping posture input provided by the vision module (such as SAM and GSNet) ensures that the robot can successfully grasp and open and close household objects. Effect: It is possible to perform opening and closing operations in simulation with a high success rate (93% opening doors, 96% opening drawers), and a success rate of 80% or more can be achieved in reality. This method improves the automation capability of household service robots in the operation of daily objects and ensures smooth and continuous movements.

[0260] In some embodiments, the application scenario in which this solution may be used in the future is assembly line robots in automobile manufacturing. Implementation method: During the automobile production process, robots can use this technology to realize the automatic assembly of various articulated parts (such as doors, hoods, etc.). The robot can automatically adjust its operation strategy and achieve precise motion control according to the different assembly component postures and 6D posture information. Effect: During the automobile manufacturing process, this technology can improve assembly accuracy and efficiency, especially when dealing with complex mechanical parts, it can achieve smooth assembly movements, reduce equipment wear, and improve the overall automation level of the production line.

[0261] Based on the foregoing embodiments, the embodiments of the present application provide a control component, which includes the units included and the modules included in the units, which can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0262] Fig.12 A schematic diagram of the structure of a control component provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the control component 1200 includes: a determination unit 1210, a generation unit 1220, a first input unit 1230 and a second input unit 1240, wherein:

[0263] A determination unit 1210, configured to determine a first interaction posture for an object to be interacted with based on an environment image in which the robot is located;

[0264] A generating unit 1220, configured to generate a splicing feature of a current time step based on an output instruction of a previous time step and the first interaction posture;

[0265] The first input unit 1230 is used to input the splicing features of the current time step and the splicing features of at least one historical time step into the trained attribute prediction module to obtain attribute features; the attribute features are used to characterize the attribute data of the object to be interacted with that affects the interaction;

[0266] The second input unit 1240 is used to input the attribute feature and the concatenated feature of the current time step into the trained reinforcement learning module to obtain the output instruction of the current time step.

[0267] In some embodiments, the trained attribute prediction module and the trained reinforcement learning module are obtained after training using attribute data related to the influencing interaction of sample objects to be interacted in a simulation environment; wherein, at any time step of the training process, the attribute prediction module adjusts parameters based on the difference between the simulation attribute features generated at any time step and the simulation attribute features predicted by the attribute prediction module; the generated simulation attribute features are obtained based on the attribute data of the sample objects; the reinforcement learning module adjusts parameters based on motion rewards, the motion rewards are obtained after the robot moves in response to simulation output instructions, and the simulation output instructions are output by the reinforcement learning module at any time step based on the generated simulation attribute features or the predicted simulation attribute features.

[0268] In some embodiments, the first input unit 1230 is further used to store the splicing features of the current time step in a history cache; and obtain the splicing features of the at least one history time step from the history cache.

[0269] In some embodiments, the second input unit 1240 is further used to scale the output instruction to obtain a target control instruction.

[0270] In some embodiments, the second input unit 1240 is further used to control the interactive device based on the target control instruction to perform the interactive action at the current time step.

[0271] In some embodiments, the environment image includes a depth image and a color image; the determination unit 1210 is also used to perform object recognition on the color image to obtain a mask image of the object to be interacted with; convert the depth image to obtain environment point cloud data; use the environment point cloud data to determine multiple second interaction postures; use the mask image to determine the first interaction posture among the multiple second interaction postures.

[0272] In some embodiments, the training method of the attribute prediction module includes: generating a simulation splicing feature of the second time step based on the simulation output instruction of the first time step and the observation data of the second time step; the first time step is the previous time step of the second time step; the simulation splicing feature of the second time step and the simulation splicing feature of at least one historical time step are input into the attribute prediction module to obtain a predicted simulation attribute feature; the simulation attribute feature is used to characterize the attribute data related to the interaction of the sample objects to be interacted in the simulation environment; the simulation splicing feature of the second time step and the target attribute feature are input into the reinforcement learning module to obtain the simulation output instruction of the second time step; the target attribute feature is the predicted simulation attribute feature or the simulation attribute feature generated based on the attribute data of the sample object; the attribute prediction module is trained based on the predicted simulation attribute feature and the generated simulation attribute feature.

[0273] In some embodiments, the training method of the reinforcement learning module includes: obtaining a motion reward after controlling the robot's motion based on the simulation output instruction of the second time step; and training the reinforcement learning module by reinforcement learning based on the motion reward.

[0274] An embodiment of the present application provides a robot, which includes the above-mentioned control component.

[0275] Fig.13 A schematic diagram of the structure of a training device provided in an embodiment of the present application is shown in FIG. Fig.13 As shown, the training device 1300 includes: a simulation generation module 1310, a first input module 1320, a second input module 1330, an acquisition module 1340 and a training module 1350, wherein:

[0276] A simulation generation module 1310 is used to generate a simulation splicing feature of a second time step based on a simulation output instruction of a first time step and observation data of a second time step; the first time step is a previous time step of the second time step;

[0277] A first input module 1320 is used to input the simulation splicing features of the second time step and the simulation splicing features of at least one historical time step into an attribute prediction module to obtain predicted simulation attribute features; the simulation attribute features are used to characterize attribute data related to the influencing interaction of the sample objects to be interacted in the simulation environment;

[0278] A second input module 1330 is used to input the simulation output instruction and the target attribute feature of the second time step into the reinforcement learning module to obtain the simulation output instruction of the second time step; the target attribute feature is the predicted simulation attribute feature or the simulation attribute feature generated based on the attribute data of the sample object;

[0279] An acquisition module 1340, configured to acquire a motion reward after controlling the robot to move based on the simulation output instruction of the second time step;

[0280] The training module 1350 is used to train the attribute prediction module based on the predicted simulation attribute features and the generated simulation attribute features, and to train the reinforcement learning module through reinforcement learning based on the motion reward.

[0281] In some embodiments, the control model further includes a coding module; the simulation generation module 1310 is further used to obtain the attribute data of the sample object; the attribute data of the sample object is input into the coding module to generate the simulation attribute characteristics.

[0282] In some embodiments, the training module 1350 is further used to train the encoding module based on the predicted simulation attribute features and the generated simulation attribute features.

[0283] In some embodiments, the motion reward includes at least one of the following: a task-aware reward and a motion-aware reward; the task-aware reward is used to encourage the robot to perform a desired motion sequence; the motion-aware reward is used to encourage the robot to perform smooth motion.

[0284] In some embodiments, the task perception reward includes at least one of the following: a success reward, used to encourage the robot to perform and complete the interaction task; a distance reward, used to encourage reducing the distance between the robot's interaction device and the object to be interacted with; an object state reward, used to encourage the robot to increase the change in the joint position of the object to be interacted with; an interaction reward, used to encourage the robot to perform and complete the target action.

[0285] In some embodiments, the motion perception reward includes at least one of the following: an energy reward, used to encourage the robot to reduce energy consumption; a tracking position reward, used to encourage the robot to reduce the difference between the actual position and the expected position; a tracking rotation reward, used to encourage the robot to reduce the difference between the actual angle and the expected angle; a smoothness reward, used to encourage the robot to perform smooth movements; a direction reward, used to encourage the robot to reduce the movement deviation in the direction.

[0286] In some embodiments, the acquisition module 1340 is further used to scale the simulation output instruction to obtain a simulation control instruction; and control the movement of the robot based on the simulation control instruction.

[0287] In some embodiments, the acquisition module 1340 is also used to generate a stiffness matrix of the robot based on the simulation output instructions; construct an impedance-controlled dynamic model based on the mass inertia matrix, damping matrix and stiffness matrix of the robot; and the dynamic model is used to control the balance between the movement of the robot and the contact force.

[0288] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0289] It should be noted that in the embodiments of the present application, if the above-mentioned control method and the training method of the control model are implemented in the form of a software function module and sold or used as an independent product, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0290] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0291] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0292] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0293] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0294] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0295] Fig.14 A hardware entity diagram of a computer device provided in an embodiment of the present application is shown in FIG. Fig.14 As shown, the hardware entity of the computer device 1400 includes: a processor 1401 and a memory 1402, wherein the memory 1402 stores a computer program that can be run on the processor 1401, and the processor 1401 implements the steps in the method of any of the above embodiments when executing the program.

[0296] The memory 1402 stores computer programs that can be run on the processor. The memory 1402 is configured to store instructions and applications executable by the processor 1401. It can also cache data to be processed or processed by the processor 1401 and various modules in the computer device 1400 (for example, image data, audio data, voice communication data, and video communication data). This can be achieved through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0297] When the processor 1401 executes the program, the steps of any of the above control methods or control model training methods are implemented. The processor 1401 generally controls the overall operation of the computer device 1400.

[0298] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the control method or control model training method of any of the above embodiments.

[0299] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0300] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0301] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0302] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A control method, characterized in that: Applied to a robot, the method comprises: Determine a first interaction posture for the object to be interacted with based on the environment image in which the robot is located; Generate a splicing feature of a current time step based on the output instruction of a previous time step and the first interaction posture; Inputting the concatenated features of the current time step and the concatenated features of at least one historical time step into the trained attribute prediction module to obtain attribute features; the attribute features are used to characterize the attribute data of the object to be interacted with that is related to the influencing interaction; The attribute feature and the concatenated feature of the current time step are input into the trained reinforcement learning module to obtain an output instruction of the current time step.

2. The method according to claim 1, characterized in that The trained attribute prediction module and the trained reinforcement learning module are obtained after training using attribute data related to the influencing interaction of sample objects to be interacted in the simulation environment; Wherein, in any time step of the training process, the attribute prediction module performs parameter adjustment based on the difference between the simulated attribute features generated in any time step and the simulated attribute features predicted by the attribute prediction module; the generated simulated attribute features are obtained based on the attribute data of the sample object; The reinforcement learning module adjusts parameters based on motion rewards, and the motion rewards are obtained after the robot moves in response to simulation output instructions. The simulation output instructions are output by the reinforcement learning module at any time step based on the generated simulation attribute features or the predicted simulation attribute features.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Storing the splicing features of the current time step in a history cache; acquiring the splicing features of at least one history time step from the history cache; and / or, scaling the output instruction to obtain a target control instruction; controlling the interactive device based on the target control instruction to perform an interactive action at the current time step; And / or, the environment image includes a depth image and a color image; and determining a first interaction posture for the object to be interacted with based on the environment image in which the device is located includes: Performing object recognition on the color image to obtain a mask image of the object to be interacted with; Converting the depth image to obtain environmental point cloud data; Determining a plurality of second interaction postures using the environmental point cloud data; The first interaction pose is determined among the plurality of second interaction poses using the mask image.

4. The method according to claim 1 or 2, characterized in that: The training method of the attribute prediction module includes: Generate a simulation splicing feature of the second time step based on the simulation output instruction of the first time step and the observation data of the second time step; the first time step is the previous time step of the second time step; Inputting the simulation splicing features of the second time step and the simulation splicing features of at least one historical time step into the attribute prediction module to obtain predicted simulation attribute features; the simulation attribute features are used to characterize attribute data related to the influencing interaction of the sample objects to be interacted in the simulation environment; Inputting the simulation splicing feature and the target attribute feature of the second time step into the reinforcement learning module to obtain the simulation output instruction of the second time step; the target attribute feature is the predicted simulation attribute feature or the simulation attribute feature generated based on the attribute data of the sample object; training the attribute prediction module based on the predicted simulation attribute features and the generated simulation attribute features; The training method of the reinforcement learning module includes: After controlling the robot to move based on the simulation output instruction of the second time step, obtaining a movement reward; The reinforcement learning module is trained by reinforcement learning based on the motion reward.

5. A control model training method, characterized in that: The control model includes an attribute prediction module and a reinforcement learning module; the training method includes: Generate a simulation splicing feature of the second time step based on the simulation output instruction of the first time step and the observation data of the second time step; the first time step is the previous time step of the second time step; Inputting the simulation splicing features of the second time step and the simulation splicing features of at least one historical time step into the attribute prediction module to obtain predicted simulation attribute features; the simulation attribute features are used to characterize attribute data related to the influencing interaction of the sample objects to be interacted in the simulation environment; Inputting the simulation output instruction and the target attribute feature of the second time step into the reinforcement learning module to obtain the simulation output instruction of the second time step; the target attribute feature is the predicted simulation attribute feature or the simulation attribute feature generated based on the attribute data of the sample object; After controlling the robot to move based on the simulation output instruction of the second time step, obtaining a movement reward; The attribute prediction module is trained based on the predicted simulation attribute features and the generated simulation attribute features, and the reinforcement learning module is trained through reinforcement learning based on the motion reward.

6. The method according to claim 5, characterized in that The control model also includes an encoding module; the training method also includes: Acquire attribute data of the sample object; Inputting the attribute data of the sample object into the encoding module to generate the simulation attribute feature; The encoding module is trained based on the predicted simulation attribute features and the generated simulation attribute features.

7. The method according to claim 5 or 6, characterized in that: The motion reward includes at least one of the following: a task perception reward and a motion perception reward; the task perception reward is used to encourage the robot to perform a desired motion sequence; the motion perception reward is used to encourage the robot to perform smooth motion; The task-aware reward includes at least one of the following: Success reward, used to encourage the robot to complete the interactive task; Distance reward, used to encourage reducing the distance between the robot's interactive device and the object to be interacted with; Object state reward, used to encourage the robot to increase the change in joint position of the object to be interacted with; Interaction rewards are used to encourage the robot to perform target actions; And / or, the motion perception reward includes at least one of the following: Energy rewards, used to encourage robots to reduce energy consumption; Tracking position rewards, used to encourage the robot to reduce the difference between its actual position and its desired position; Tracking rotation rewards to encourage the robot to reduce the difference between the actual angle and the desired angle; Smoothness reward, used to encourage the robot to perform smooth movements; Direction reward, used to encourage the robot to reduce the deviation of the action in the said direction.

8. The method according to claim 5 or 6, characterized in that: The control of the robot motion by the simulation output instruction based on the second time step includes: scaling the simulation output instruction to obtain a simulation control instruction; and controlling the robot motion based on the simulation control instruction; And / or, the method further comprises: generating a stiffness matrix of the robot based on the simulation output instructions; An impedance-controlled dynamic model is constructed based on the mass inertia matrix, the damping matrix and the stiffness matrix of the robot; the dynamic model is used to control the balance between the movement of the robot and the contact force.

9. A control component, characterized in that: Set in the robot, the control component includes: A determination unit, configured to determine a first interaction posture for an object to be interacted with based on an image of an environment in which the robot is located; A generating unit, configured to generate a splicing feature of a current time step based on an output instruction of a previous time step and the first interaction posture; A first input unit is used to input the splicing features of the current time step and the splicing features of at least one historical time step into the trained attribute prediction module to obtain attribute features; the attribute features are used to characterize the attribute data of the object to be interacted with that is related to the influencing interaction; The second input unit is used to input the attribute feature and the concatenated feature of the current time step into the trained reinforcement learning module to obtain the output instruction of the current time step.

10. A robot, characterized in that: The robot comprises the control assembly of claim 9.

Images