Robot assembly training method, robot assembly control method and computer readable storage medium

Abstract

The embodiment of the invention discloses a robot assembly training method, a robot assembly control method and a computer readable storage medium. The training method comprises the following steps: constructing an initial agent, and generating a preset number of joint state combinations based on the initial agent; in the simulation environment, the simulation robot is controlled to execute the assembly action according to the joint state combination; determining a reward mode, and scoring the executed assembly action based on the reward mode to obtain a reward score; and training the initial agent by using the reward score. The control method comprises the following steps: acquiring a trained agent; inputting the obtained environment data at the current moment into the intelligent agent, and adjusting parameters of the intelligent agent to obtain an implementation intelligent agent; and inputting the task instruction into the implementation agent, and executing an assembly action according to a strategy output by the implementation agent. Therefore, the adaptability of the assembly robot can be effectively improved, and the assembly action can be accurately completed in more application scenes.

Description

Technical Field

[0001] This application belongs to the field of robot control technology, and in particular relates to a robot assembly training method, a robot assembly control method, and a computer-readable storage medium. Background Technology

[0002] Assembly is one of the most common production activities worldwide, and the automation of robotic assembly processes is of great significance for saving production costs and shortening production cycles. Thanks to advancements in technologies such as precision machining, precision pose calibration, machine vision, and visual servo control, the assembly of many products is now automated. However, the assembly of many products still often requires the control of assembly forces, and the degree of control over these forces directly determines the assembly quality.

[0003] The difficulty in controlling contact forces during assembly lies primarily in the numerous contacts encountered during the assembly process. This necessitates repeated adjustments to the relative positions of components, generating contact forces and torques of varying directions and magnitudes. These forces are multidirectional, transient, and random. Due to the complex nonlinear relationships involved, the changes in contact forces and torques during assembly are uncertain. This often renders traditional rules and pre-programmed methods inadequate, increasing the difficulty of assembly. Using machine vision to control the assembly process may lead to abrupt changes in contact forces due to visual feature occlusion and positioning errors, increasing the risk of assembly failure. Existing assembly methods can only effectively address assembly needs in specific scenarios and are ill-suited to complex and variable contact states and force control precision requirements. Therefore, improving the adaptability of robots to complex contact states during assembly is a pressing technical problem that needs to be solved by those skilled in the art.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] The purpose of this application is to provide a robot assembly training method, a robot assembly control method, and a computer-readable storage medium, which can effectively improve the adaptability of assembly robots and enable them to accurately complete assembly actions in more application scenarios.

[0006] To achieve the above objectives: In a first aspect, embodiments of this application provide a robot assembly training method, comprising the following steps: constructing an initial agent, generating a preset number of joint state combinations based on the initial agent; controlling a simulated robot to perform assembly actions in a simulation environment according to the joint state combinations; determining a reward mode, scoring the performed assembly actions based on the reward mode to obtain a reward score; training the initial agent using the reward score, iteratively optimizing the initial agent until a preset convergence condition is met; and marking the trained initial agent as an agent and outputting it.

[0007] In an optional embodiment of the present application, the preset number of joint state combinations are generated based on the initial agent, including: obtaining design parameters, including the number of joints of the robot, the motion range, and the degree of freedom configuration; randomly generating a preset number of joint state combinations to be screened according to the design parameters; performing feasibility verification on the joint state combinations to be screened; and marking the joint state combinations that pass the feasibility verification as joint state combinations.

[0008] In an optional embodiment of the present application, the simulated robot is controlled to perform the assembly action according to the joint state combination, including: initializing the simulated robot and randomizing the pose of the target workpiece, the target workpiece being a target for the robot to perform a task; generating a corresponding control instruction according to the joint state combination to drive the end effector of the simulated robot to move towards the target workpiece; determining whether the simulated robot successfully performs the assembly action; if so, saving the assembly action; if not, re-performing the assembly action until a preset threshold of the number of attempts is reached; and if the threshold is reached and the assembly action is still not successfully performed, marking the joint state combination as invalid.

[0009] In an optional embodiment of the present application, the simulated robot is initialized and the pose of the target workpiece is randomized, including: setting an elastic body at the target workpiece, the elastic body being used to connect the target workpiece and the simulated robot; and applying a random disturbance force to the elastic body, the random disturbance force including at least one of stiffness, damping, friction coefficient, and direction variable.

[0010] In an optional embodiment of the present application, the reward mode is determined, including: obtaining task information performed by the initial agent; when the task information meets a complex task type and the number of steps of the assembly action is greater than a preset threshold of the number of steps, determining that the reward mode is a dense reward mode; and when the task information meets a simple task type or the number of steps of the assembly action is less than or equal to the preset threshold of the number of steps, determining that the reward mode is a sparse reward mode.

[0011] In an optional embodiment of the present application, when the reward mode is the dense reward mode, the performed assembly action is scored based on the reward mode, including: obtaining a real-time pose of the assembly action; determining a real-time coordinate corresponding to the real-time pose, obtaining a target coordinate of the target workpiece; calculating an error value between the real-time coordinate and the target coordinate, and determining a distance reward according to the error value; calculating a pose deviation according to a preset ideal pose and the real-time pose, and determining a success reward; obtaining the number of execution steps of the assembly action, and determining a step reward according to the number of execution steps; determining the number of times of over-limit state in the process of performing the assembly action, and determining a penalty reward according to the number of times of over-limit state; and weighting and summing the distance reward, the success reward, the step reward, and the penalty reward to generate a reward score in the dense reward mode.

[0012] In an optional embodiment of the present application, when the reward mode is a sparse reward mode, the executed assembly action is scored based on the reward mode, including: obtaining an end pose of the assembly action; determining whether the assembly task is completed according to the end pose; if yes, a preset reward value is given; if no, no reward is given; determining whether the end pose meets a preset ideal pose; if no, a preset penalty value is given; if yes, no penalty value is given; and adding the reward value and the penalty value to generate a reward score in the sparse reward mode.

[0013] In an optional embodiment of the present application, the initial agent is trained by using the reward score, and the initial agent is iteratively optimized until a preset convergence condition is met, including: combining the assembly action and the corresponding joint state, the reward score, and a completion flag to generate a sample, the completion flag being used to indicate whether the assembly action completes the assembly; aggregating all the samples to obtain a training sample set; obtaining a preset training network, the training network being constructed by using a deep reinforcement learning algorithm or an actor-critic algorithm; inputting the training sample set into the training network, and adjusting policy parameters of the initial agent by policy gradient update; in each iteration, performing the assembly action according to the current policy and collecting new samples, and continuously updating the training sample set; and iteratively training until the policy parameters meet the preset convergence condition, the preset convergence condition including at least one of a change amount of the policy parameters being less than a preset threshold, a change rate of a mean value of the reward score within a preset training period being less than a preset threshold, and an iteration number reaching a preset upper limit.

[0014] In a second aspect, the embodiments of the present application provide a robot assembly control method, including the following steps: obtaining a pre-trained agent, the agent being obtained by training through the robot assembly training method provided in the foregoing; inputting environment data of a current time into the agent to adjust the agent, and obtaining an implemented agent; inputting a task instruction into the implemented agent, and performing an assembly action according to a policy output by the implemented agent.

[0015] In a third aspect, the embodiments of the present application provide a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the robot assembly training method and / or the robot assembly control method.

[0016] By using the embodiments of the present application, the following beneficial effects are achieved: The robot assembly training method provided in the application comprises the following steps: constructing an initial agent, generating a preset number of joint state combinations based on the initial agent; in a simulation environment, controlling a simulated robot to perform an assembly action according to the joint state combinations; determining a reward mode, scoring the performed assembly action based on the reward mode to obtain a reward score; training the initial agent by using the reward score, and iteratively optimizing the initial agent to meet a preset convergence condition; and marking the trained initial agent as an agent and outputting it. The robot assembly control method provided comprises the following steps: obtaining a pre-trained agent, which is obtained by training through the robot assembly training method provided in the foregoing; inputting environment data of a current time into the agent to adjust the agent, and obtaining an implemented agent; inputting a task instruction into the implemented agent, and performing an assembly action according to a strategy output by the implemented agent. Therefore, the application can train an adaptive assembly control model by using dynamic physical parameter randomization, and can adapt to more complex and variable assembly environments. In the training process, different reward modes are distinguished to differentially train the agent, so that the agent has stronger generalization ability and decision accuracy when facing different assembly tasks. In addition, the method provided in the application can automatically learn an assembly strategy based on a simulation environment, can be trained through a large amount of virtual data, and can accelerate the training and shorten the development cycle.

[0017] The above description is only a summary of the technical solutions of the application. In order to enable the technical means of the application to be more clearly understood, and to be implemented according to the content of the description, and in order to enable the above and other purposes, characteristics and advantages of the application to be more apparent and easy to understand, the following preferred embodiments are described in detail, and the accompanying drawings are used for description. It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or the prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0019] Figure 1 A flowchart of a robot assembly training method provided for an embodiment.

[0020] Figure 2 An assembly action diagram provided for an embodiment.

[0021] Figure 3 A flowchart of a robot assembly control method provided for an embodiment.

[0022] Figure 4 A schematic diagram of a communication mechanism in a robot pick-and-place control method according to an embodiment.

[0023] Figure 5 A schematic block diagram of a computer device according to an embodiment. DETAILED DESCRIPTION

[0024] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, unless otherwise indicated, like numbers in the attached drawings refer to the same or similar elements. Additionally, the term "coupled" as used herein means the elements so connected are electrically connected, magnetically connected, communicatively connected, or any combination thereof.

[0025] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus. In the absence of more constraints, the same element (1001) in different embodiments can or can not be the same. The specific meaning of the terms "comprises", "comprising", or any other variations thereof, should be determined in light of the description and / or context of the particular embodiment and / or by the specific description applying to a certain terminology.

[0026] It should be understood that, although the terms first, second, third, etc. can be employed in this disclosure to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one category of information from another category of information. For example, a first information can also be termed a second information, similarly, a second information can also be termed a first information without departing from the scope of this disclosure. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining". Also, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It will be further understood that the terms "comprises", "comprising", "includes" and / or "including", as used herein, specify the presence of stated features, steps, operations, elements, components, items, kinds and / or groups but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, items, kinds and / or groups. As used herein, the terms "or" and "and / or" are to be interpreted as inclusive, i.e., as meaning one or any combination of items. Thus, "A, B or C" or "A, B and / or C" means any of the following: A; B; C; A and B; A and C; B and C; A, B and C. This definition applies regardless of the lack of any occurring adjectives that can precede, follow, or be interspersed, these adjectives including: complementary, alternative, additional, equivalent, supplemental, superfluous, excess, or redundant.

[0027] It should be understood that, although the steps in the flowcharts of the embodiments of the present application are displayed in sequence according to the direction of the arrows, these steps are not necessarily executed in sequence according to the direction of the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the figure can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0028] It should be noted that in this document, step codes such as S110, S120, etc. are used, the purpose of which is to express the corresponding content more clearly and concisely, and does not constitute a substantial limitation in sequence. Those skilled in the art may, for example, perform S120 first and then perform S110 in specific implementation, but these should be within the scope of protection of the present application.

[0029] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0030] In the following description, the suffixes such as "module", "part", or "unit" used for an element are used only for convenience of explanation of the present application, and have no specific meaning by themselves. Thus, "module", "part", or "unit" can be mixedly used.

[0031] In order to facilitate the understanding of the present application, the following will first explain the terms and technical terms that may be involved in the present application, as follows: Robot refers to a subject that performs an assembly task, usually an industrial robot or a collaborative robot. It has multiple degrees of freedom and can flexibly grasp, move and place parts.

[0032] Adaptation refers to the ability of a robot system to perceive changes in the environment (uncertainty) and adjust its actions and strategies in real time to cope with these changes, thereby successfully completing the assembly task.

[0033] Assembly control refers to the control technology of robots performing the specific process of "assembly", and the core goal is to successfully combine two or more parts together in the correct position (x, y, z), attitude / direction (rx, ry, rz) and force (force / torque: Fx, Fy, Fz, Tx, Ty, Tz).

[0034] In actual robot assembly processes, due to various reasons, contact-rich assembly phenomena such as friction and collision occur, which involve complex nonlinear contact dynamics. The key to solving such problems lies in developing assembly action strategies to enable the robot to effectively cope with contact force disturbances during the assembly process, which requires measuring contact forces and torques through force / torque sensors, dynamically adjusting the end pose of the robot arm, and ensuring smooth assembly. There are mainly two categories of common assembly contact force control methods. One is assembly control based on accurate models, the first category belongs to traditional assembly control methods, including impedance control, force / position hybrid control, etc. The other is learning-based assembly control. The second category belongs to intelligent assembly control methods, including imitation learning and deep reinforcement learning, etc. In addition, a part of learning-based methods need to be combined with Sim2Real to complete.

[0035] However, the prior art has the following defects. 1. Accurate model: the assembly control method based on accurate model regards the assembly process as a motion process constrained by geometry and environment, describes the contact state using a series of transformations, models the kinematics, dynamics and other parameters involved in the assembly process, and adjusts the action strategy of the assembly contact process. However, there are many parameters involved in the assembly process, making the modeling complex and requiring a large number of experimental verifications. At the same time, traditional robot assembly tasks often rely on a large amount of manual programming and hard coding, and the model lacks universality and will fail when the scene changes slightly.

[0036] 2. Imitation learning: Learning assembly skills based on imitation learning is an effective method, but it is affected by the level of demonstration, personal habits, etc. The demonstration data has a lot of drift, error, noise and other uncertainties. Imitation learning requires a large amount of data and time to train the robot, and needs repeated training and adjustment. This limits the popularization and application of this method to some extent, and it is also difficult to use for learning general assembly skills.

[0037] 3. Deep reinforcement learning: Deep reinforcement learning has achieved a lot of success in this field, but its assembly effect is affected by factors such as sampling efficiency, safety, and consistency of virtual and real data, which limits the further use of reinforcement learning methods in actual assembly. Therefore, many researchers have turned their attention to Sim2Real, developing more robust strategies or researching more realistic simulation environments. That is, first simulate, then migrate.

[0038] 4. Sim2Real: Most current research is based on simulators such as PyBullet, Mujoco and IsaacSim for development, however, in the face of assembly tasks with rich contact, it is usually necessary to collect visual and force sensation samples simultaneously, or to collect simulation and reality samples simultaneously, or to only achieve a single assembly task.

[0039] In order to overcome the technical defects existing in the prior art, the present application provides a robot assembly training method and a robot assembly control method. In order to clearly describe the method provided by the embodiment, please refer to Figures 1-4 .

[0040] The robot assembly training method provided by the present application constructs an assembly contact skill learning framework based on Sim2Real (deep reinforcement learning + dynamic physical parameter randomization + communication architecture of robot assembly system). The framework uses deep reinforcement learning to train robot assembly skills, first learns the mapping relationship between assembly contact state and assembly action adjustment in a simulation environment, and obtains a force control assembly skill model. Then, the Sim2Real method is used to improve the generalization ability of the skill model to realize the Sim2Real migration of the assembly skill. Finally, the trained skill model is applied to an actual industrial robot, so that it can effectively perform assembly tasks in a real environment.

[0041] The overall scheme mainly includes the following three parts: assembly skill training based on a simulation environment (Sim), skill model migration based on physical parameter randomization (Sim2Real), and actual assembly control based on the skill model (To Real). By establishing a state-action-reward (SAR) closed-loop learning mechanism for the assembly process, end-to-end skill migration from simulation training to real deployment is realized.

[0042] In the Sim simulation training stage, deep reinforcement learning, data collection and model training are adopted. Under this framework, first, a simulation assembly environment is constructed, then an assembly state, action and reward are designed, a reinforcement learning framework is constructed, a deep reinforcement learning model is trained in the simulation environment, and the mapping relationship and rules between the contact state and action adjustment are found.

[0043] In the Sim2Real simulation stage, dynamic random based on physics is adopted, and specifically, a domain randomization method can be used to construct a dynamic random agent training process based on the stiffness, damping and friction parameters of the assembly environment to achieve effective model migration.

[0044] In the Real actual control stage, that is, after the model training is completed, a communication system between the actual robot, the six-dimensional force sensor and the model is built through the ROS framework, the reinforcement learning framework is loaded, and the real-time acquired environment information is input as the actual state. Then the real action is output by the model, which is transmitted to the robot through the communication system and controls the robot to execute the action to complete the Sim2Real transfer of the assembly skill.

[0045] Step S110: constructing an initial agent, generating a preset number of joint state combinations based on the initial agent.

[0046] In an embodiment, generating a preset number of joint state combinations based on the initial agent includes: obtaining design parameters, including the number of joints of the robot, the motion range and the degree of freedom configuration; randomly generating a preset number of joint state combinations to be screened according to the design parameters; performing feasibility verification on the joint state combinations to be screened; and marking the joint state combinations to be screened that pass the feasibility verification as joint state combinations.

[0047] In an embodiment, a virtual simulation environment is constructed, for example, a simulation assembly environment is constructed based on the Mujoco simulation engine, and a robot model, a force / torque sensor, an assembly component and the like are loaded in the simulation environment. Physical parameters such as environmental gravity, friction coefficient, damping and stiffness can be loaded. In the Markov Decision Process (Markov Decision Process) commonly used in reinforcement learning, the state (State) of the agent is changed by taking an action (Action), and the reward (Reward) is maximized. According to the Markov Decision Process, the robot, the force / torque sensor and the assembly component in the assembly environment are regarded as an assembly system, and a corresponding robot assembly control method is designed.

[0048] An initial agent is constructed, which can be a decision unit responsible for outputting the next assembly control instruction according to the current assembly state. The initial agent outputs an action A by inputting a state S. The state S can be determined by a joint state combination. The joint state combination can be obtained by acquiring the design parameters of the robot, including the number of joints, the motion range, and the degree of freedom configuration. A large number of joint state combinations are generated by Monte Carlo simulation according to the degree of freedom and joint limit of the robot arm, and each joint state combination covers the pose state of each joint of the robot arm. For example: for a 7-DOF robot arm, 10 7 A set of joint state combinations to be screened, referred to as joint state combinations to be screened.

[0049] The joint state combinations to be screened are verified for feasibility, which can specifically determine the reachability and collision-free of the joint state combinations to be screened in the physical space, exclude invalid poses caused by singular configurations or self-collision, and retain valid states that meet the dynamics constraints and task requirements.

[0050] The self-collision condition constraints of the robot arm for the joint state combinations are obtained, and the end pose data corresponding to the joint state combinations that do not meet the self-collision constraints among the preset number of joint state combinations are determined as the unreachable end pose data. The self-collision condition constraints are obtained, such as joint torque limit, obstacle avoidance requirement or path continuity, or whether the generated joint state will collide with itself based on the URDF file of the robot / robot arm, and the poses that will cause self-collision are marked as unreachable. The joint state combinations to be screened that do not meet the self-collision constraints are marked as not passing the verification.

[0051] The task workspace corresponding to the current task of the robot arm is obtained, and the task workspace is the neighborhood space around the target object corresponding to the current task; the end pose data to be screened that is not in the neighborhood space is determined as the unreachable end pose data. The task workspace is the neighborhood space around the target object corresponding to the current task of the robot arm, that is, the destination that the robot arm needs to go to. The joint state combinations to be screened that are not in the neighborhood space are marked as not passing the verification.

[0052] Obtain the obstacle point cloud in the external environment of the robot arm; determine the end pose data corresponding to the joint state combination to be screened as the unreachable end pose data, which is obtained based on the design parameters and the robot arm point cloud collides with the obstacle point cloud among the preset number of joint state combinations. The obstacle point cloud is used to describe the static obstacle distribution, dynamic interference area and space limiting information in the working environment of the robot arm. Through the obstacle point cloud, the pose data that may collide with the environment of the robot arm relative to the working environment can be determined. The joint state combinations to be screened are marked as not passing the verification.

[0053] The joint state combination to be screened that passes the feasibility verification is marked as a joint state combination. The joint state combination specifically refers to assembly state information inputted by a plurality of modal parameters as an agent, and can specifically include robot end position and posture , force / torque , and robot joint angle information . A mathematical expression of the joint state combination can refer to the following formula.

[0054] (1) Step S120: In the simulation environment, control the simulation robot to perform the assembly action according to the joint state combination.

[0055] In an embodiment, controlling the simulation robot to perform the assembly action according to the joint state combination includes: initializing the simulation robot, and randomizing a target workpiece pose, the target workpiece being a target for the robot to perform a task; generating a corresponding control instruction according to the joint state combination to drive the end effector of the simulation robot to move towards the target workpiece; judging whether the simulation robot successfully performs the assembly action; if yes, saving the assembly action; if no, re-performing the assembly action until a preset threshold of attempt times is reached; and if the threshold is reached but the assembly action is still not successfully performed, marking the joint state combination as invalid.

[0056] In an embodiment, in the simulation case, the simulation robot is initialized, and a target workpiece position is randomized, the target workpiece being a target for the robot to perform a task. The randomization operation is to randomly disturb the initial position and posture of the target workpiece each time, so as to simulate positioning errors and tooling deviations in a real assembly scene. The randomization operation will be described later, and the simulation assembly action is described here.

[0057] According to the joint state combination, a corresponding control instruction is generated to drive the end effector of the simulation robot to move towards the target workpiece. As described above, the joint state combination is the state input of the initial agent, and the initial agent will obtain the assembly action according to the processing output. To adapt to the application of robot position control, the present application uses a robot kinematics model, and uses the increments of the robot end position and posture as the discrete / continuous action space of the agent. Therefore, the agent with the joint state combination as the input can obtain the corresponding assembly action, which can refer to the following formula.

[0058] (2) According to the assembly action, the control instruction for the robot can be determined. The robot is controlled to simulate the completion of the assembly task by using the control instruction. For ease of understanding, the assembly action can refer to Figure 2The robot automatic assembly task can be divided into four stages according to the contact state of the assembly parts: 1, shaft hole approach state; 2, shaft hole preliminary contact state, the relative position of the shaft hole needs to be adjusted; 3, insertion adjustment process, for single shaft hole assembly with large gap, the relative pose of the shaft hole is adjusted and then directly inserted; for single shaft hole assembly with small gap, multi-shaft hole parallel assembly and complex assembly trajectory, dynamic pose adjustment is also needed in this process; 4, complete shaft hole assembly.

[0059] Since stage 1 can be quickly positioned by vision, the current difficulty of robot automatic assembly is mainly in the contact process of assembly, that is, the subsequent three stages, which need to repeatedly adjust the relative position between parts.

[0060] In actual robot assembly process, due to various reasons, it causes the contact-rich assembly phenomenon such as friction, collision, which involves complex nonlinear contact dynamics. The key to solving this problem lies in formulating assembly action strategy, so that the robot can effectively cope with the contact force disturbance in the assembly process, which needs to measure the contact force and torque through force / torque sensor, dynamically adjust the end pose of the robot arm, and ensure smooth assembly. That is to say, the simulated assembly action may not necessarily complete the assembly task, and there is a possibility of failure.

[0061] After each assembly action simulation, it is judged whether the simulation robot successfully executes the assembly action. Whether the assembly action is successful can be determined by the relative pose deviation of the end effector and the target workpiece and the change of the contact force. When the pose deviation is less than the set threshold and the contact force tends to be stable, it is considered that the assembly at this stage is successful.

[0062] If successful, save the assembly action. The saving will also save the number of steps consumed to complete the assembly task and the success mark. Subsequently, the assembly action and the corresponding joint state combination will be analyzed and learned to complete the training of the initial agent. After completing the saving, continue to simulate the next joint state combination.

[0063] If failed, re-execute the assembly action until the preset threshold of the number of attempts is reached. If it is still unsuccessful after reaching the threshold, mark the joint state combination as invalid. It is worth noting that whether successful or not, the joint state combination and the corresponding assembly action will be saved. For the successful one, it will be learned as a positive example, and the failed one will be learned as a negative example, thereby improving the training effect.

[0064] In an embodiment, the simulation robot is initialized, and the pose of the target workpiece is randomized, including: setting an elastic body at the target workpiece, the elastic body being used to connect the target workpiece and the simulation robot; applying a random disturbance force to the elastic body, the random disturbance force including at least one of stiffness, damping, friction coefficient and direction variable.

[0065] In an embodiment, the robot assembly environment constructed and simulated above is fully rigid, but in the actual environment, some parts are flexible, such as the flexible material at the bottom of the force sensor. Therefore, there is always a certain deviation between the force / torque obtained by the agent in the simulation environment and the actual environment, and the assembly state of the simulation environment is difficult to match the actual environment. When the assembly skill is migrated from the simulation environment to the real environment, it is easy to cause assembly failure.

[0066] Therefore, in the simulation environment, the sensor with flexible material is constructed. Through the simulation platform, an elastic body is added between the sensor and the workpiece to connect the sensor and the workpiece. The elastic body is randomized every time a new joint state combination is simulated. The randomization process can be achieved by setting physical parameters such as random disturbance force (e.g. stiffness k, damping c, friction f), and the workpiece can rotate in different degrees and directions, so that the sensor obtains different degrees of force / torque (F / T) parameters. An assembly skill training method based on dynamic randomization of physical parameters is designed to adapt to the physical parameter mismatch problem between the simulation assembly environment and the actual assembly environment.

[0067] In the actual assembly environment, the physical parameters of the overall assembly system are determined, but due to various factors, it is difficult to find the real physical parameters. The application does not need to manually find the physical parameter values, but only needs to set reasonable parameter thresholds k max , k min , c max , c min , f max , f min} In the simulation environment, the assembly control skill is trained by big data, and the physical parameters of the assembly environment are updated every round to realize the generalization ability of the assembly skill and realize the migration from simulation to reality. By adding an elastic body to randomize every round, the Sim2Real process is realized, which can improve the robustness of the model to adapt to changes in sensor or workpiece materials.

[0068] Step S130: Determine the reward mode, score the executed assembly action based on the reward mode, and obtain the reward score.

[0069] In one embodiment, determining the reward mode includes: acquiring task information executed by the initial agent; when the task information matches a complex task type and the number of steps in the assembly action is greater than a preset step threshold, determining the reward mode as a dense reward mode; when the task information matches a simple task type, or the number of steps in the assembly action is less than or equal to a preset step threshold, determining the reward mode as a sparse reward mode.

[0070] In one embodiment, the reward design can be based on information such as the robot end-effector position and the force / torque generated during assembly, as shown in the following formula.

[0071] (3) In the above formula, Total reward, including distance reward Success Rewards Step rewards and punishments and rewards Among them, distance reward Step rewards and punishments and rewards This is an instant reward, calculated in real-time; success reward. The success of the assembly task, as simulated earlier, determines whether it was completed. Success grants a positive reward, while failure incurs a negative penalty. (Distance Reward) Specifically, the distance between the robot's end effector and the target workpiece is determined. When they are close, a positive reward is given in real time; when they are far apart, a negative reward is given in real time. (Step reward) Rewards are given based on the number of steps the robot takes to complete a task, encouraging it to complete the task quickly with fewer steps. When the number of steps exceeds a certain threshold, a negative value is applied as a penalty. (Penalty / Reward) Generally, penalties are imposed for whether the simulated assembly actions during the assembly process of the intelligent agent exceed the preset force / torque / position thresholds. If they exceed the thresholds, a preset negative value is given; otherwise, the value is 0. and As a penalty weight, both are set to 1 in dense reward modes and 0 in sparse reward modes. Furthermore, distance reward varies depending on the mode. and success rewards The calculation methods are also different, which will be described in detail later.

[0072] In addition, there are different reward modes, including a dense reward mode and a sparse reward mode. The dense reward mode is used for complex tasks using a continuous action space, while the sparse reward mode is used for simple tasks using a discrete action space. Therefore, the corresponding reward mode needs to be determined before the reward is calculated. To this end, the task information executed by the initial agent can be obtained to make a judgment. When the task information conforms to the complex task type and the number of assembled actions is greater than a preset step threshold, it is determined that the reward mode is the dense reward mode. When the task information conforms to the simple task type or the number of assembled actions is less than or equal to the preset step threshold, it is determined that the reward mode is the sparse reward mode. The number of assembled actions is directly obtained according to the simulated assembled action.

[0073] If it is determined in the task information that the corresponding assembled action is based on real-time perception information to select an action strategy, adjust parameters, or make a retry decision, the task is determined to be a complex task type, and the dense reward mode is used. Correspondingly, if the task information only involves preset trajectory execution and does not require strategy adjustment based on perception feedback, it is determined to be a simple task type, and the sparse reward mode is used.

[0074] In addition, the specific reward calculation methods of the dense reward mode and the sparse reward mode are different.

[0075] In an embodiment, when the reward mode is the dense reward mode, a score of the executed assembled action is calculated based on the reward mode, including: obtaining a real-time pose of the assembled action; determining a real-time coordinate corresponding to the real-time pose, obtaining a target coordinate of a target workpiece; calculating an error value between the real-time coordinate and the target coordinate, and determining a distance reward according to the error value; calculating a pose deviation according to a preset ideal pose and the real-time pose, and determining a success reward; obtaining an execution step number of the executed assembled action, and determining a step reward according to the execution step number; determining a number of out-of-limit states in the process of executing the assembled action, and determining a penalty reward according to the number of out-of-limit states; and weighting and summing the distance reward, the success reward, the step reward, and the penalty reward to generate a reward score in the dense reward mode.

[0076] In an embodiment, the dense reward mode is to calculate and assign the reward in real time during the completion of the assembled action. Therefore, the end pose of the assembled action obtained at this time is real-time, which is referred to as a real-time pose. The real-time coordinate corresponding to the real-time pose is determined, and the target coordinate of the target workpiece is obtained. The real-time coordinate and the target coordinate each include three sub-quantities, corresponding to the hole center upper position, the hole center position, and the hole bottom center position, respectively. The hole refers to a hole structure that needs to be inserted during the completion of the assembled action, and similar positioning holes or matching holes. The deviations of the real-time coordinate and the target coordinate in the three sub-quantities are calculated, and the distance reward is finally obtained by weighting and summing. The calculation method can refer to the following formula.

[0077] (4) In the above formula, , , is a preset weighting coefficient. , and are weight values corresponding to three sub-quantities. Among them, corresponding to the real-time deviation of the robot end and the three directions above the hole surface, the calculation method can refer to the following formula.

[0078] (5) In the above formula, , and are preset top deviation weights. is a real-time coordinate, which is obtained in real time; is the hole top center coordinate in the target coordinate, which is a preset value and can be directly obtained.

[0079] corresponding to the real-time deviation of the robot end and the three directions of the hole center position, the calculation method can refer to the following formula.

[0080] (6) In the above formula, , and are preset center deviation weights. is the hole center position coordinate in the target coordinate, which can be directly obtained.

[0081] corresponding to the real-time deviation of the robot end and the three directions of the hole bottom position, the calculation method can refer to the following formula.

[0082] (7) In the above formula, , and are preset bottom deviation weights. is the hole center position coordinate in the target coordinate, which can be directly obtained.

[0083] Success reward Through the calculation of the preset ideal pose and the real-time pose, the calculation process refers to the following formula.

[0084] (8) In the above formula, is a preset initial success reward, is a preset reward coefficient. The reward coefficient An update will be performed after each successful assembly, following the rules below.

[0085] (9) In the above formula, This represents the end-effector depth in the real-time pose. The target depth is defined as the depth within the ideal pose. When the end depth exceeds the target depth, the target workpiece is considered to have been inserted into the corresponding assembly hole, thus completing the assembly. Success or failure can also be determined directly using success or invalid labels obtained during simulated assembly.

[0086] Obtain the number of steps taken to perform the assembly action, and determine the step reward based on the number of steps taken. Step Reward The calculation method is shown in the following formula.

[0087] (10) In the above formula, It is the real-time execution step count. The maximum number of steps to attempt per round is set to the preset number of steps. According to the above formula, the step reward... A negative number means that a penalty is imposed during the real-time reward process, thereby stimulating the agent to complete the assembly as quickly as possible.

[0088] Punishment and Reward The calculation method is shown in the following formula.

[0089] (11) In the above formula, The value is a preset penalty weight, which is negative. The above formula represents the number of times the robot exceeds the limit during the assembly process, and a corresponding penalty weight is given for each instance. The thresholds set for exceeding the limit for the robot include, but are not limited to, force, torque, or position.

[0090] Ultimately, the distance to the reward will be... Success Rewards Step rewards and punishments and rewards The total reward is obtained by weighted summation according to the calculation method shown in formula (3). The total reward This is considered a reward score.

[0091] In an embodiment, when the reward mode is a sparse reward mode, the performed assembly action is scored based on the reward mode, including: obtaining an end pose of the assembly action; determining whether the assembly task is completed according to the end pose; if yes, a preset reward value is given; if no, no reward is given; determining whether the end pose meets a preset ideal pose; if no, a preset penalty value is given; if yes, no penalty value is given; and adding the reward value and the penalty value to generate a reward score in the sparse reward mode.

[0092] In an embodiment, for the sparse reward mode, referring to formula (3), the distance reward and the success reward are also included, but the calculation methods are completely different. The distance reward is a negative number, which is referred to as a penalty value, and the specific calculation formula is as follows.

[0093] (12) In the above formula, the is a preset penalty value; The success of the end pose meeting the preset ideal pose is determined, specifically, during the execution process, the end pose is always advancing to complete the assembly, that is, the steps performed are all valid steps, and the end pose is gradually approaching the ideal pose. Otherwise, a corresponding penalty value is given.

[0094] For the success reward , in the sparse reward mode, it is referred to as a reward value. The calculation method is as follows.

[0095] (13) In the above formula, the is a preset reward value; indicates that when the end depth is greater than the target depth, it is determined that the target workpiece is inserted into the corresponding assembly hole, that is, it is determined that the assembly is completed. Whether it is successful or not can also be directly determined by the success label or invalid label obtained in the simulation assembly.

[0096] Finally, the reward value and the penalty value in the sparse reward mode are added to generate a reward score in the sparse reward mode. It is worth noting that the difference between the sparse reward mode and the dense reward mode is that the calculation of the dense reward mode is performed in real time; and in the sparse reward mode, since it is for discrete execution steps, the calculation is performed step by step, the rewards of each step are calculated, and finally the corresponding reward score is obtained.

[0097] Step S140: training the initial agent using the reward score, and iteratively optimizing the initial agent to meet a preset convergence condition.

[0098] Step S150: Mark the initial agent after training as an agent and output.

[0099] In an embodiment, the initial agent is trained by using the reward score, and the initial agent is iteratively optimized to meet a preset convergence condition, including: combining the assembly action and the corresponding joint state, the reward score, and the completion mark to generate a sample, the completion mark being used to indicate whether the assembly action completes the assembly; aggregating all samples to obtain a training sample set; obtaining a preset training network, the training network being constructed by using a deep reinforcement learning algorithm or an actor-critic algorithm; inputting the training sample set into the training network, and adjusting the policy parameter of the initial agent by policy gradient update; in each iteration, performing the assembly action according to the current policy and collecting new samples, and continuously updating the training sample set; iteratively training until the policy parameter meets the preset convergence condition, the preset convergence condition including at least one of the following: a change amount of the policy parameter being less than a preset threshold, a change rate of a mean value of the reward score within a preset training period being less than a preset threshold, and a number of iterations reaching a preset upper limit.

[0100] In an embodiment, the initial agent is trained, that is, it needs to learn how to more quickly and accurately process the output to obtain an assembly action that can quickly and accurately complete the assembly requirement according to the robot state and task information. Training needs to use samples, and a single sample is composed of a four-tuple of an assembly action and a corresponding joint state combination, a reward score, and a completion mark. The assembly action is an action performed; the joint state combination is the state of each joint of the robot; the reward score is a value obtained by the process described above; and the completion mark is used to determine whether the assembly action completes the assembly, including success or failure. All samples are aggregated to obtain a sample set.

[0101] A preset training network is obtained, and the training network is constructed by using a deep reinforcement learning algorithm (Deep Q-Network, DQN) or an actor-critic algorithm (Actor-Critic, AC). Different algorithms will use different training processes.

[0102] When the AC algorithm is used, two MLP (Multi-Layer Perceptron) networks are needed, which are used as a policy network and a value network, respectively. Sharing the operation of the feature layer of the two networks is beneficial to improving the efficiency of parameter adjustment, reducing the redundancy of feature extraction, and reducing overfitting.

[0103] When the DQN algorithm is used, the training process is also two networks, which are a main network and a target network. However, only one MLP network is needed, and the target value is calculated by copying the main network as the target network. This double-network structure helps to solve the problems of unstable training and estimation bias in reinforcement learning.

[0104] When selecting a reinforcement learning algorithm for the discrete action space, the loss function of the DQN is selected as the SmoothL1Loss, and the formula is as follows.

[0105] (14) In the above formula, is the predicted value, is the target value, is an adjustable hyperparameter, which can be set as δ = 1 in a preferred embodiment. Using this loss function helps to improve the robustness of the model to outliers and noise, maintain the stability of the training process, and provide smooth gradients.

[0106] The pre-prepared training sample set is input into the training network. The policy parameters possessed by the initial agent are adjusted by means of the policy gradient update method. This adjustment is to make the agent's policy more in line with the training goals and requirements. In each iteration of the entire training process, the agent generates assembly actions according to the current policy, repeatedly adjusts the parameters, so that the generated assembly actions can better complete the task.

[0107] The iteration is trained until the policy parameters meet the preset convergence condition, and the preset convergence condition includes at least one of the following: the change amount of the policy parameters is less than a preset threshold, the change rate of the average reward score in a preset training period is less than a preset threshold, and the number of iterations reaches a preset upper limit.

[0108] Repeated training enables the initial agent to continuously learn the assembly contact skill. The assembly environment can also be repeatedly configured and randomized during the iteration process, so that the initial agent can better adapt to the real physical environment. The initial agent that has finally completed the training is marked as an agent and output.

[0109] Correspondingly, the application also provides an application method, i.e., a robot assembly control method, which includes steps S210-S230.

[0110] Step S210: obtaining a pre-trained agent.

[0111] In an embodiment, the training process of the agent has been described in detail in the foregoing, and will not be repeated here.

[0112] Step S220: inputting the environment data of the current time into the agent to adjust the parameters of the agent and obtain an implemented agent.

[0113] Step S230: inputting a task instruction into the implemented agent to execute an assembly action according to the policy output by the implemented agent.

[0114] In an embodiment, it can be understood that the foregoing training process is a sim and sim2real process, and the last 2real step still needs to be completed. For this purpose, real physical data is needed to adjust the parameters of the agent to adapt to the real environment. First, the current environmental data is obtained, including but not limited to the joint state of the robot, the stiffness, damping, and friction of the assembled environment, etc., which are input into the trained agent to adjust the parameters of the agent, so that the agent can better simulate the real environment, and the implemented agent, also known as the assembly control model, is obtained and configured in the corresponding hardware.

[0115] The physical robot assembly system includes the above-mentioned components and a robot system, which includes a hardware part and a software part. The hardware part is composed of a robot (6 degrees of freedom), a lower computer (robot control cabinet), and an upper computer (personal PC). The software part includes the agent described above. After establishing the kinematic model according to the robot parameters, the communication mechanism as shown in Figure 4 is used to realize the communication between the industrial robot, the six-axis force sensor, the upper computer, and the assembly control model (i.e., the implemented agent). The joint data is transmitted to the PC upper computer through the ROS message type sensor_msgs / JointState, and then the pose is calculated through the forward kinematics of the PC upper computer. The force / torque information of the robot end is obtained based on the ROS driver package of the six-axis force sensor and is transmitted in the message type geometry_msgs / Wrench. The robot data and sensor data are transmitted to the PC upper computer as state information, and the upper computer transmits the state information into the skill model. The skill model outputs the corresponding assembly strategy, i.e., the robot pose increment, according to the input, and then the robot joint angle is calculated through the inverse kinematics of the PC upper computer. The PC upper computer transmits the joint angle to the industrial robot through the message type sensor_msgs / JointState, and then controls the robot movement. In this way, the assembly action strategy is updated in real time according to the robot data and sensor data, and finally the robot automatic assembly is realized.

[0116] According to the physical assembly platform, the robot control panel is opened using the teach pendant, the automatic motion mode is set, the ROS communication file “ros_sever” is started, and then the ROS related nodes are started by running the roslaunch file on the PC side, the robot is initialized, the shaft hole relative pose is randomized, and the current state information is transmitted / updated. Then the corresponding task instruction is issued, and the assembly strategy is obtained through the implemented agent. The robot will automatically execute the assembly strategy. After the execution of the strategy, it is judged whether the threshold is exceeded and whether the assembly is successful, the assembly data is updated, and then the assembly strategy is updated until the assembly is completed.

[0117] In addition, the application adopts a reinforcement learning algorithm based on SimReal to train the assembly control model, which is now trained in simulation, and then migrated to the actual assembly environment. There can also be a corresponding alternative solution, which directly collects data and trains the model in the actual environment through traditional assembly control methods (impedance control, force / position hybrid control), assembly control methods based on imitation learning. And in the original scheme, the end pose incremental control can also be replaced by joint angle incremental control. However, the alternative solution is less effective and is not the preferred embodiment, but is an optional solution for demonstration.

[0118] The robot assembly training method provided by the application comprises the following steps: constructing an initial agent, generating a preset number of joint state combinations based on the initial agent; in a simulation environment, controlling a simulated robot to perform an assembly action according to the joint state combinations; determining a reward mode, scoring the performed assembly action based on the reward mode to obtain a reward score; training the initial agent using the reward score, and iteratively optimizing the initial agent to satisfy a preset convergence condition; and marking the trained initial agent as an agent and outputting it. The robot assembly control method provided comprises the following steps: obtaining a pre-trained agent, which is trained by the robot assembly training method provided in the foregoing; inputting environment data of a current time into the agent to adjust the agent, and obtaining an implemented agent; inputting a task instruction into the implemented agent, and performing an assembly action according to the strategy output by the implemented agent. Therefore, the application can train an adaptive assembly control model by using dynamic physical parameter randomization, which can adapt to more complex and variable assembly environments. In addition, during the training process, the agent is trained differently using different reward modes, so that the agent has stronger generalization ability and decision accuracy when facing different assembly tasks. In addition, the method provided by the application can automatically learn an assembly strategy based on a simulation environment, can be trained through a large amount of virtual data, and can accelerate the training and shorten the development cycle. In addition, the technical solution of the application has stronger environmental adaptability. Existing technologies usually use fixed control parameters, mainly rely on pre-programmed fixed trajectory control, PID force control strategy, force / position hybrid control, or vision-based servo control, and have poor environmental adaptability. The application trains the model by using dynamic physical parameter randomization, which can adapt to more complex and variable assembly environments. In terms of development efficiency, existing technologies consume time for manual parameter adjustment and data collection, and there is a certain safety risk in the experimental process. The application can automatically learn an assembly strategy based on a simulation environment, can be trained through a large amount of virtual data, and can accelerate the training and shorten the development cycle.

[0119] Figure 5 An internal structure diagram of a computer device in an embodiment is shown. The computer device can be a terminal or a server. As shown in the figure, the computer device comprises a processor 1, a memory 2, a bus 3, an input device 4, and an output device 5. The processor 1, the memory 2, the bus 3, the input device 4, and the output device 5 are connected through the bus 3. The processor 1 is configured to execute the instructions stored in the memory 2 to realize the functions of the computer device. The memory 2 is configured to store the instructions executed by the processor 1. The input device 4 is configured to receive the instructions input by a user. The output device 5 is configured to output the results of the instructions executed by the processor 1. Figure 5As shown, the device comprises a processor 310 and a memory 311 storing a computer program; wherein, Figure 5 The processor 310 in the description is not used to refer to the number of processors 310 being one, but is only used to refer to the positional relationship of the processor 310 relative to other devices. In actual application, the number of processors 310 can be one or more; similarly, Figure 5 The memory 311 in the description also has the same meaning, that is, it is only used to refer to the positional relationship of the memory 311 relative to other devices. In actual application, the number of memories 311 can be one or more. When the processor 310 runs the computer program, the method applied to the above device is implemented.

[0120] The device can further comprise at least one network interface 312. Various components in the device are coupled together through a bus system 313. It can be understood that the bus system 313 is used to realize the connection communication between the components. The bus system 313 includes not only a data bus, but also a power supply bus, a control bus and a status signal bus. However, in order to clearly illustrate, various buses are marked as the bus system 313 in the Figure 5

[0121] ​The memory 311 can be a volatile memory or a nonvolatile memory, and can include both a volatile and a nonvolatile memory. The nonvolatile memory can be a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a ferroelectric random access memory (FRAM), a flash memory, a magnetic memory, an optical memory, or a compact disc read-only memory (CD-ROM). The magnetic memory can be a magnetic disk memory or a magnetic tape memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM can be used, such as a Static Random Access Memory (SRAM), a Synchronous Static Random Access Memory (SSRAM), a Dynamic Random Access Memory (DRAM), a Synchronous Dynamic Random Access Memory (SDRAM), a Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), an Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), a Sync Link Dynamic Random Access Memory (SLDRAM), or a Direct Rambus Random Access Memory (DRRAM).The memory 311 described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable type of memory.

[0122] The memory 311 in the embodiments of the present application is configured to store various types of data to support the operation of the device. Examples of the data include: any computer programs for operating on the device, such as operating systems and application programs; contact data; phonebook data; messages; pictures; videos; and the like. The operating system contains various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks. The application programs can contain various application programs, such as a Media Player, a Browser, and the like, for implementing various application services. Here, the program for implementing the method of the embodiments of the present application can be contained in the application programs.

[0123] Based on the same inventive concept as the foregoing embodiments, the present embodiment also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer readable storage medium can be a ferromagnetic random access memory (FRAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, a compact disc read-only memory (CD-ROM), or the like. The computer readable storage medium can also be various devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, and the like. The computer program stored in the computer readable storage medium is run by a processor to implement the above method. The specific step flow implemented by the computer program when executed by the processor will be described in the embodiments of the present application shown in the description, which will not be described here again. Figure 1 The description of the embodiments shown in the foregoing description will not be described here again.

[0124] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0125] In this document, the terms "comprise", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0126] The above description is only specific embodiments of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered by the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A robot kitting training method, characterized by, The method comprises the following steps: constructing an initial agent, generating a preset number of joint state combinations based on the initial agent; controlling a simulated robot to perform an assembly action according to the joint state combinations in a simulation environment; determining a reward mode, scoring the performed assembly action based on the reward mode to obtain a reward score; training the initial agent using the reward score, and iteratively optimizing the initial agent to meet a preset convergence condition; marking the trained initial agent as an agent and outputting.

2. The robot pick training method of claim 1, wherein, The generating a preset number of joint state combinations based on the initial agent comprises: obtaining design parameters, including the number of joints of the robot, the motion range and the degree of freedom configuration; randomly generating a preset number of to-be-screened joint state combinations according to the design parameters; performing feasibility verification on the to-be-screened joint state combinations; marking the to-be-screened joint state combinations that pass the feasibility verification as the joint state combinations.

3. The robot pick-and-place training method of claim 1, wherein, The controlling a simulated robot to perform an assembly action according to the joint state combinations comprises: initializing the simulated robot and randomizing the pose of a target workpiece, which is the target of the robot performing a task; generating corresponding control instructions according to the joint state combinations to drive the end effector of the simulated robot to move towards the target workpiece; judging whether the simulated robot successfully performs the assembly action; if yes, saving the assembly action; if no, re-performing the assembly action until a preset threshold of the number of attempts is reached; if the threshold is reached and the assembly action is still not successfully performed, marking the joint state combination as invalid.

4. The robot pick-and-place training method of claim 3, wherein, The initializing the simulated robot and randomizing the pose of a target workpiece comprises: setting an elastic body at the target workpiece, the elastic body being used to connect the target workpiece and the simulated robot; applying a random disturbance force to the elastic body, the random disturbance force including at least one of stiffness, damping, friction coefficient and direction variable.

5. The robot pick-and-place training method of claim 1, wherein, The determining a reward mode comprises: obtaining task information performed by the initial agent; when the task information meets a complex task type and the number of steps of the assembly action is greater than a preset threshold of the number of steps, determining that the reward mode is a dense reward mode; when the task information meets a simple task type or the number of steps of the assembly action is less than or equal to the preset threshold of the number of steps, determining that the reward mode is a sparse reward mode.

6. The robot pick-and-place training method of claim 1, wherein, When the reward mode is the dense reward mode, The scoring the performed assembly action based on the reward mode comprises: obtaining a real-time pose of the assembly action; determining a real-time coordinate corresponding to the real-time pose, obtaining a target coordinate of a target workpiece; calculating an error value between the real-time coordinate and the target coordinate, and determining a distance reward according to the error value; calculating a pose deviation and determining a success reward according to a preset ideal pose and the real-time pose; obtaining the number of steps of performing the assembly action, and determining a step reward according to the number of steps; determining the number of times of exceeding a limit state in the process of performing the assembly action, and determining a penalty reward according to the number of times of exceeding the limit state; The distance reward, the success reward, the step reward and the penalty reward are weighted and summed to generate a reward score in the dense reward mode.

7. The robotic pick training method of claim 1, wherein, When the reward mode is a sparse reward mode, The assembling action performed is scored based on the reward mode, including: An end pose of the assembling action is obtained; It is determined whether the end pose meets a preset ideal pose, and if not, a preset penalty value is given, and if so, no penalty value is given. The reward value and the penalty value are added to generate a reward score in the sparse reward mode. The initial agent is trained using the reward score, and the initial agent is iteratively optimized to meet a preset convergence condition, including:

8. The robotic pick training method of claim 1, wherein, The assembling action and the corresponding joint state combination, the reward score and a completion flag are combined to generate a sample, the completion flag being used to indicate whether the assembling action completes assembly, and all the samples are aggregated to obtain a training sample set; A preset training network is obtained, the training network being constructed using a deep reinforcement learning algorithm or an actor-critic algorithm; The training sample set is input into the training network, and the policy parameters of the initial agent are adjusted through policy gradient update, in each iteration, an assembling action is performed according to the current policy and new samples are collected, and the training sample set is continuously updated; The iterative training is performed until the policy parameters meet a preset convergence condition, the preset convergence condition including at least one of the following: a change amount of the policy parameters being less than a preset threshold, a change rate of a mean value of the reward score within a preset training period being less than a preset threshold, and an iteration number reaching a preset upper limit. The method comprises the following steps:

9. A robot fitting control method characterized by, An agent is obtained, the agent being trained by the robot assembly training method of any one of claims 1 to 8; Environment data of a current time is input into the agent to adjust the agent, and an implementation agent is obtained; A task instruction is input into the implementation agent, and an assembling action is performed according to the policy output by the implementation agent. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the robot assembly training method of any one of claims 1 to 8 and / or the robot assembly control method of claim 9.

10. A computer-readable storage medium, characterized in that, ​

Images