Robot assembly training method, robot assembly control method, and computer-readable storage medium

Abstract

Embodiments of the present application disclose a robot assembly training method, a robot assembly control method and a computer readable storage medium. The training method comprises: constructing an initial agent, generating a preset number of joint state combinations based on the initial agent; in a simulation environment, controlling a simulated robot to perform an assembly action according to the joint state combinations; determining a reward mode, scoring the performed assembly action based on the reward mode to obtain a reward score; and training the initial agent using the reward score. The control method comprises: obtaining a trained agent; inputting environment data of a current time into the agent to adjust the agent, and obtaining an implemented agent; and inputting a task instruction into the implemented agent, and performing an assembly action according to a strategy output by the implemented agent. Therefore, the present application can effectively improve the adaptability of the assembly robot and accurately complete the assembly action in more application scenarios.

Description

Technical Field

[0001] This application belongs to the field of robot control technology, and in particular relates to a robot assembly training method, a robot assembly control method, and a computer-readable storage medium. Background Technology

[0002] Assembly is one of the most common production activities worldwide, and the automation of robotic assembly processes is of great significance for saving production costs and shortening production cycles. Thanks to advancements in technologies such as precision machining, precision pose calibration, machine vision, and visual servo control, the assembly of many products is now automated. However, the assembly of many products still often requires the control of assembly forces, and the degree of control over these forces directly determines the assembly quality.

[0003] The difficulty in controlling contact forces during assembly lies primarily in the numerous contacts encountered during the assembly process. This necessitates repeated adjustments to the relative positions of components, generating contact forces and torques of varying directions and magnitudes. These forces are multidirectional, transient, and random. Due to the complex nonlinear relationships involved, the changes in contact forces and torques during assembly are uncertain. This often renders traditional rules and pre-programmed methods inadequate, increasing the difficulty of assembly. Using machine vision to control the assembly process may lead to abrupt changes in contact forces due to visual feature occlusion and positioning errors, increasing the risk of assembly failure. Existing assembly methods can only effectively address assembly needs in specific scenarios and are ill-suited to complex and variable contact states and force control precision requirements. Therefore, improving the adaptability of robots to complex contact states during assembly is a pressing technical problem that needs to be solved by those skilled in the art.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] The purpose of this application is to provide a robot assembly training method, a robot assembly control method, and a computer-readable storage medium, which can effectively improve the adaptability of assembly robots and enable them to accurately complete assembly actions in more application scenarios.

[0006] To achieve the above objectives:

[0007] In a first aspect, embodiments of this application provide a robot assembly training method, comprising the following steps: constructing an initial agent, generating a preset number of joint state combinations based on the initial agent; controlling a simulated robot to perform assembly actions in a simulation environment according to the joint state combinations; determining a reward mode, scoring the performed assembly actions based on the reward mode to obtain a reward score; training the initial agent using the reward score, iteratively optimizing the initial agent until a preset convergence condition is met; and marking the trained initial agent as an agent and outputting it.

[0008] In an optional embodiment of this application, generating a preset number of joint state combinations based on an initial intelligent agent includes: obtaining design parameters, including the number of robot joints, range of motion, and degree of freedom configuration; randomly generating a preset number of joint state combinations to be screened according to the design parameters; performing feasibility verification on the joint state combinations to be screened; and marking the joint state combinations to be screened that pass the feasibility verification as joint state combinations.

[0009] In an optional embodiment of this application, controlling a simulated robot to perform an assembly action based on a joint state combination includes: initializing the simulated robot and randomizing the pose of the target workpiece, wherein the target workpiece is the target of the robot's task; generating corresponding control commands based on the joint state combination to drive the simulated robot's end effector to move toward the target workpiece; determining whether the simulated robot has successfully performed the assembly action; if successful, saving the assembly action; if unsuccessful, re-executing the assembly action until a preset threshold number of attempts is reached; if the threshold is reached but the attempt is still unsuccessful, marking the joint state combination as invalid.

[0010] In an optional embodiment of this application, initializing the simulated robot and randomizing the pose of the target workpiece includes: setting an elastic body at the target workpiece, the elastic body being used to connect the target workpiece and the simulated robot; applying a random perturbation force to the elastic body, the random perturbation force including at least one of stiffness, damping, friction coefficient and direction variable.

[0011] In an optional embodiment of this application, determining the reward mode includes: obtaining task information executed by the initial intelligent agent; when the task information conforms to a complex task type and the number of steps in the assembly action is greater than a preset step threshold, determining the reward mode as a dense reward mode; when the task information conforms to a simple task type, or the number of steps in the assembly action is less than or equal to a preset step threshold, determining the reward mode as a sparse reward mode.

[0012] In an optional embodiment of this application, when the reward mode is a dense reward mode, the assembly action is scored based on the reward mode, including: obtaining the real-time pose of the assembly action; determining the real-time coordinates corresponding to the real-time pose and obtaining the target coordinates of the target workpiece; calculating the error value between the real-time coordinates and the target coordinates, and determining the distance reward based on the error value; calculating the posture deviation and determining the success reward based on the preset ideal pose and the real-time pose; obtaining the number of execution steps of the assembly action and determining the step reward based on the number of execution steps; determining the number of out-of-limit states during the execution of the assembly action and determining the penalty reward based on the number of out-of-limit states; and weighted summing the distance reward, success reward, step reward, and penalty reward to generate a reward score in the dense reward mode.

[0013] In an optional embodiment of this application, when the reward mode is a sparse reward mode, the assembly action is scored based on the reward mode, including: obtaining the end-effector pose of the assembly action; determining whether the assembly task is completed based on the end-effector pose; if completed, a preset reward value is given; if not completed, no reward is given; determining whether the end-effector pose conforms to a preset ideal pose; if not, a preset penalty value is given; if conforms, no penalty value is given; and adding the reward value and the penalty value to generate a reward score in the sparse reward mode.

[0014] In an optional embodiment of this application, the initial agent is trained using reward scores, and the initial agent is iteratively optimized until a preset convergence condition is met. This includes: combining assembly actions and corresponding joint states, reward scores, and completion markers to generate samples, where the completion markers indicate whether the assembly action is completed; summarizing all samples to obtain a training sample set; obtaining a preset training network, which is constructed using a deep reinforcement learning algorithm or an actor-critic algorithm; inputting the training sample set into the training network and adjusting the policy parameters of the initial agent through policy gradient updates; in each iteration, executing assembly actions according to the current policy and collecting new samples to continuously update the training sample set; iteratively training until the policy parameters meet the preset convergence condition, which includes at least one of the following: the change in policy parameters is less than a preset threshold, the rate of change of the average reward score within a preset training period is less than a preset threshold, and the number of iterations reaches a preset upper limit.

[0015] Secondly, embodiments of this application provide a robot assembly control method, including the following steps: acquiring a pre-trained intelligent agent, which is trained using the robot assembly training method provided above; inputting the acquired environmental data at the current moment into the intelligent agent, adjusting the parameters of the intelligent agent to obtain an implementing intelligent agent; inputting task instructions into the implementing intelligent agent, and executing assembly actions according to the strategy output by the implementing intelligent agent.

[0016] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described robot fitting training method and / or robot fitting control method.

[0017] The embodiments of this application have the following beneficial effects:

[0018] The robot assembly training method provided in this application includes the following steps: constructing an initial agent, generating a preset number of joint state combinations based on the initial agent; controlling the simulated robot to perform assembly actions according to the joint state combinations in a simulated environment; determining a reward mode, scoring the executed assembly actions based on the reward mode to obtain a reward score; training the initial agent using the reward score, iteratively optimizing the initial agent until a preset convergence condition is met; marking the trained initial agent as an agent and outputting it. The provided robot assembly control method includes the following steps: obtaining a pre-trained agent, which is trained using the robot assembly training method described above; inputting the acquired environmental data at the current moment into the agent, tuning the agent's parameters to obtain an implementing agent; inputting task instructions into the implementing agent, and executing assembly actions according to the strategy output by the implementing agent. Therefore, this application can train an adaptive assembly control model by using dynamic physical parameter randomization, which can adapt to more complex and variable assembly environments. Furthermore, during training, different reward modes are used to differentiate the training of the agent, enabling the agent to have stronger generalization ability and decision-making accuracy when facing different assembly tasks. Furthermore, the method provided in this application can automatically learn assembly strategies based on a simulation environment, can be trained using a large amount of virtual data, and can accelerate training, thereby shortening the development cycle.

[0019] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it according to the contents of the specification, and to make the above and other objects, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a robot fitting and training method provided in one embodiment.

[0022] Figure 2 A schematic diagram of the assembly process is provided for one embodiment.

[0023] Figure 3 This is a flowchart illustrating a robot assembly control method provided in one embodiment.

[0024] Figure 4 This is a schematic diagram of the communication mechanism in a robot fitting control method provided in one embodiment.

[0025] Figure 5 This is a schematic block diagram of the structure of a computer device provided in one embodiment. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements.

[0027] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0028] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, can be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive, or mean any one or any combination thereof. Therefore, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C". Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0029] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0030] It should be noted that step designations such as S110 and S120 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S120 first and then S110, etc., but these should all be within the protection scope of this application.

[0031] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0032] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0033] To facilitate understanding of this application, the following explanations are provided for the terms and technical objects that may be used in this application:

[0034] A robot is a device that performs assembly tasks, typically an industrial robot or a collaborative robot. It has multiple degrees of freedom, allowing it to flexibly grasp, move, and place parts.

[0035] Adaptive refers to the ability of a robot system to perceive changes (uncertainty) in the environment and adjust its actions and strategies in real time to cope with these changes, thereby successfully completing the assembly task.

[0036] Assembly control refers to the control technology for robots to perform the specific process of "assembly". The core goal is to enable robots to successfully combine two or more parts with the correct position (x, y, z), posture / direction (rx, ry, rz) and force (force / torque: Fx, Fy, Fz, Tx, Ty, Tz).

[0037] In actual robot assembly processes, various factors lead to contact-rich phenomena such as friction and collisions, involving complex nonlinear contact dynamics. The key to solving these problems lies in developing assembly motion strategies that enable the robot to effectively cope with contact force disturbances during assembly. This requires measuring contact forces and torques using force / torque sensors and dynamically adjusting the end effector pose of the robotic arm to ensure smooth assembly. Common assembly contact force control methods fall into two main categories. One is assembly control based on accurate models, belonging to traditional assembly control methods, including impedance control and force / position hybrid control. The other is learning-based assembly control, belonging to intelligent assembly control methods, including imitation learning and deep reinforcement learning. Furthermore, some learning-based methods require integration with Sim2Real.

[0038] However, existing technologies have the following drawbacks: 1. Precise Model: Assembly control methods based on precise models treat the assembly process as a motion process constrained by geometry and the environment. They employ a series of transformations to describe the contact state and model the kinematic and dynamic parameters involved in the assembly process to adjust the motion strategy for the assembly contact process. However, the assembly process involves many parameters, making modeling complex and requiring extensive experimental verification. Furthermore, traditional robot assembly tasks often rely heavily on manual programming and hard coding, resulting in models that lack versatility and fail when the scenario changes slightly.

[0039] 2. Imitation Learning: Imitation learning is an effective method for learning assembly skills, but it is affected by factors such as the teaching level and personal habits. The teaching data contains many uncertainties such as drift, error, and noise. Imitation learning requires a large amount of data and time to train the robot, and requires repeated training and adjustment. This limits the widespread application of this method to some extent, and it is also difficult to use for learning general assembly skills.

[0040] 3. Deep Reinforcement Learning: Deep reinforcement learning has achieved considerable success in this field. However, its assembly performance is affected by factors such as sampling efficiency, security, and consistency between virtual and real data. These factors limit the further application of reinforcement learning methods in practical assembly. Therefore, many researchers have turned their attention to Sim2Real, specifically developing more robust policies or researching more realistic simulation environments. That is, simulating first, then transferring the results.

[0041] 4. Sim2Real: Most current research is based on simulators such as PyBullet, Mujoco and IsaacSim. However, when faced with assembly tasks with rich contact, it is usually necessary to collect visual and force samples at the same time, or to collect simulated and real samples at the same time, or only a single assembly task can be achieved.

[0042] To overcome the technical deficiencies of existing technologies, this application proposes a robot assembly training method and a robot assembly control method. For a clear description of the method provided in this embodiment, please refer to... Figures 1-4 .

[0043] This application provides a robot assembly training method that constructs a Sim2Real-based assembly contact skill learning framework (deep reinforcement learning + dynamic physical parameter randomization + communication architecture of the robot assembly system). This framework uses deep reinforcement learning to train robot assembly skills. First, it learns the mapping relationship between assembly contact states and assembly action adjustments in a simulated environment, obtaining a force-controlled assembly skill model. Then, it utilizes the Sim2Real method to enhance the generalization ability of the skill model, achieving Sim2Real transfer of assembly skills. Finally, the trained skill model is applied to an actual industrial robot, enabling it to effectively perform assembly tasks in a real-world environment.

[0044] The overall solution mainly consists of three parts: assembly skill training based on a simulated environment (Sim), skill model transfer based on physical parameter randomization (Sim2Real), and actual assembly control based on the skill model (To Real). By establishing a state-action-reward (SAR) closed-loop learning mechanism for the assembly process, end-to-end skill transfer from simulated training to real-world deployment is achieved.

[0045] In the Sim simulation training phase, deep reinforcement learning, data acquisition, and model training are employed. Within this framework, a simulated assembly environment is first constructed, then assembly states, actions, and rewards are designed, a reinforcement learning framework is built, and a deep reinforcement learning model is trained in the simulation environment to find the mapping relationship and patterns between contact states and action adjustments.

[0046] In the Sim2Real simulation phase, physical-based dynamic randomization is employed. Specifically, domain randomization methods can be used to construct a dynamically randomized agent training process based on the stiffness, damping, and friction parameters of the assembly environment, thereby achieving effective model transfer.

[0047] In the "To Real" stage, after model training is complete, a communication system is built using the ROS framework between the actual robot, the six-dimensional force sensor, and the model. A reinforcement learning framework is then loaded, and real-time environmental information is used as the actual state input. The model then outputs a real action, which is transmitted to the robot through the communication system, and the robot is controlled to execute the action, thus completing the Sim2Real transfer of assembly skills.

[0048] Step S110: Construct an initial agent and generate a preset number of joint state combinations based on the initial agent.

[0049] In one embodiment, generating a preset number of joint state combinations based on an initial intelligent agent includes: acquiring design parameters, including the number of robot joints, range of motion, and degree of freedom configuration; randomly generating a preset number of joint state combinations to be screened according to the design parameters; performing feasibility verification on the joint state combinations to be screened; and marking the joint state combinations to be screened that pass the feasibility verification as joint state combinations.

[0050] In one implementation, a virtual simulation environment is constructed, such as a simulated assembly environment built based on the Mujoco simulation engine. Models of robots, force / torque sensors, and assembly components are loaded into this simulation environment, which can also include environmental physical parameters such as gravity, friction coefficient, damping, and stiffness. Reinforcement learning typically uses Markov Decision Processes (MDPs) in interactive loops where actions are taken to change the state and maximize reward. Based on MDPs, this innovation treats the robot, force / torque sensors, and assembly components in the assembly environment as an assembly system and designs a corresponding robot assembly control method.

[0051] An initial intelligent agent is constructed, which can be a decision-making unit responsible for outputting the next assembly control command based on the current assembly state. It outputs action A by inputting state S. State S can be determined by joint state combinations. Joint state combinations can be obtained by acquiring the robot's design parameters, including the number of joints, range of motion, and degree of freedom configuration. Based on the robot arm's degrees of freedom and joint constraints, Monte Carlo simulation is used to generate a large number of joint state combinations, each encompassing the pose states of each joint of the robot arm. For example, for a 7-DOF robot arm, 10 joint state combinations are randomly sampled and generated within the joint constraints. 7 The combination of joint states is called the joint state combination to be screened.

[0052] Feasibility verification of the joint state combinations to be screened can be performed. Specifically, the accessibility and collision-free nature of the joint state combinations to be screened in physical space can be determined, invalid postures caused by singular configurations or self-collisions can be eliminated, and valid states that meet the dynamic constraints and task requirements can be retained.

[0053] The process involves obtaining self-collision constraints for joint state combinations of the robotic arm, and identifying unreachable end-effector pose data corresponding to joint state combinations that do not meet these constraints from a pre-defined number of combinations. Self-collision constraints are then obtained, such as joint torque limits, obstacle avoidance requirements, or path continuity. Alternatively, based on the robot / robotic arm's URDF file, it can be determined whether the generated joint state will collide with itself; poses that experience self-collision are marked as unreachable. Finally, joint state combinations that do not meet these self-collision constraints are marked as failing verification.

[0054] The task workspace for the robotic arm under the current task is obtained. The task workspace is the neighborhood space surrounding the target object corresponding to the current task. End-effector pose data not within the neighborhood space are identified as unreachable end-effector pose data. The task workspace is the neighborhood space surrounding the target object corresponding to the current task being performed by the robotic arm, which is also the destination the robotic arm needs to reach. Joint state combinations not within the neighborhood space are marked as failing verification.

[0055] Obtain obstacle point clouds from the robotic arm's external environment. Identify unreachable end-effector pose data corresponding to joint state combinations where the robotic arm point cloud (reconstructed based on design parameters) collides with the obstacle point clouds. The obstacle point cloud describes the distribution of static obstacles, dynamic interference areas, and spatial constraints in the robotic arm's working environment. Through the obstacle point cloud, pose data where the robotic arm may collide with the working environment can be determined. Joint state combinations to be screened are marked as failing validation.

[0056] The joint state combinations that pass the feasibility verification are marked as joint state combinations. Specifically, a joint state combination refers to the assembly state information input to the agent using multimodal parameters, which may include the robot's end-effector position and orientation. Force / Torque Robot joint angle information The mathematical expression for the combination of joint states can be found in the following formula.

[0057] (1)

[0058] Step S120: In the simulation environment, control the simulated robot to perform assembly actions according to the joint state combination.

[0059] In one embodiment, controlling a simulated robot to perform an assembly action based on a joint state combination includes: initializing the simulated robot and randomizing the pose of a target workpiece, where the target workpiece is the target of the robot's task; generating corresponding control commands based on the joint state combination to drive the simulated robot's end effector to move toward the target workpiece; determining whether the simulated robot has successfully performed the assembly action; if successful, saving the assembly action; if unsuccessful, re-executing the assembly action until a preset threshold number of attempts is reached; if the threshold is reached but the attempt is still unsuccessful, marking the joint state combination as invalid.

[0060] In one embodiment, in the simulation, the simulated robot is initialized, and the position of the target workpiece is randomized. The target workpiece is the objective for the robot to perform its task. Randomization involves randomly perturbing the initial position and orientation of the target workpiece each time to simulate positioning errors and tooling deviations in a real assembly scenario. The randomization operation will be explained in detail later; here, we will focus on the simulated assembly action.

[0061] Based on the joint state combinations, corresponding control commands are generated to drive the simulated robot's end effector to move towards the target workpiece. As mentioned earlier, the joint state combinations serve as the initial state input for the agent, which will then generate assembly actions based on the processed output. To adapt to robot position control applications, this application uses a robot kinematics model and employs the robot's end effector position... and posture The increment serves as the discrete / continuous action space for the agent. Therefore, an agent that takes a combination of joint states as input can obtain the corresponding assembly action, which can be referred to in the following formula.

[0062] (2)

[0063] Based on the assembly actions, the control commands for the robot can be determined. These control commands are then used to control the robot to simulate and complete the assembly task. For ease of understanding, the assembly actions can be referenced... Figure 2 As shown, in the robot's automated assembly task, based on the contact state of the assembled parts, the automated assembly process can be generally divided into four stages: 1. Shaft-hole approach state; 2. Shaft-hole initial contact state, requiring adjustment of the relative position of the shaft and hole; 3. Insertion and adjustment process. For single-shaft-hole assembly with a large gap, the shaft and hole can be directly inserted after adjusting the relative pose; however, for single-shaft-hole assembly with a small gap, parallel assembly of multiple shafts and holes, and complex assembly trajectories, dynamic pose adjustment is also required during this process; 4. Completion of shaft-hole assembly.

[0064] Since stage 1 can be quickly located visually, the current difficulty in automated robot assembly lies mainly in the contact process of assembly, that is, the subsequent three stages, which require repeated adjustments to the relative positions between parts.

[0065] In actual robot assembly processes, various factors lead to contact-rich phenomena such as friction and collisions, involving complex nonlinear contact dynamics. The key to solving these problems lies in developing assembly motion strategies that enable the robot to effectively cope with contact force disturbances during assembly. This requires measuring contact forces and torques using force / torque sensors and dynamically adjusting the end effector pose of the robotic arm to ensure successful assembly. In other words, simulated assembly actions are not guaranteed to complete the assembly task and there is a possibility of failure.

[0066] After each assembly simulation, it is determined whether the simulated robot has successfully executed the assembly action. The success of the assembly action can be determined by the relative pose deviation between the end effector and the target workpiece and the change in contact force. When the pose deviation is less than a set threshold and the contact force tends to stabilize, the assembly is considered successful at this stage.

[0067] If successful, the assembly action is saved. This save also saves the number of steps required to complete the assembly task and a success flag. The assembly actions and corresponding joint state combinations will then be analyzed and learned to complete the training of the initial agent. After saving, the simulation of the next joint state combination continues.

[0068] If the attempt fails, the assembly action is repeated until a preset threshold of attempts is reached. If the attempt still fails after reaching the threshold, the joint state combination is marked as invalid. It is worth noting that regardless of success or failure, the joint state combination and its corresponding assembly action are saved. Successful combinations are used as positive examples for learning, while failed combinations are used as negative examples, thereby improving the training effect.

[0069] In one embodiment, initializing the simulated robot and randomizing the pose of the target workpiece includes: setting an elastic body at the target workpiece, the elastic body being used to connect the target workpiece and the simulated robot; applying a random perturbation force to the elastic body, the random perturbation force including at least one of stiffness, damping, friction coefficient and direction variable.

[0070] In one embodiment, the robot assembly environment constructed and simulated above is entirely rigid. However, in the real environment, some components are flexible, such as the bottom of the force sensor, which is made of flexible material. Therefore, the force / torque acquired by the agent in the simulated environment always deviates from that in the real environment. The assembly state in the simulated environment is difficult to match with that in the real environment, and assembly failure is easily caused when the assembly skills are transferred from the simulated environment to the real environment.

[0071] Therefore, in the simulation environment, this application constructs a sensor with a flexible material. An elastic body is added between the sensor and the workpiece through a simulation platform, connecting them. This elastic body is randomized each time a new joint state combination is simulated. The randomization process can be achieved by setting physical parameters such as random perturbation force (e.g., stiffness k, damping c, friction f), allowing the workpiece to rotate to different degrees and in different directions, thus enabling the sensor to obtain different levels of force / torque (F / T) parameters. An assembly skill training method based on dynamic randomization of physical parameters is designed to adapt to the mismatch between physical parameters in the simulated assembly environment and the actual assembly environment.

[0072] In actual assembly environments, the physical parameters of the overall assembly system are fixed, but due to various factors, it is difficult to find the actual physical parameters. This application eliminates the need for manual searching of physical parameter values; it only requires setting reasonable parameter thresholds. k max , k min , c max , c min , f max , f min In a simulated environment, assembly control skills are trained using big data, with the physical parameters of the assembly environment updated each round to achieve generalization of the assembly skills and transfer from simulation to reality. By incorporating an elastomer for randomization each round, a Sim2Real process is implemented, improving the model's robustness to adapt to changes in sensors or workpiece materials.

[0073] Step S130: Determine the reward mode, score the performed assembly action based on the reward mode, and obtain the reward score.

[0074] In one embodiment, determining the reward mode includes: acquiring task information executed by the initial agent; when the task information matches a complex task type and the number of steps in the assembly action is greater than a preset step threshold, determining the reward mode as a dense reward mode; when the task information matches a simple task type, or the number of steps in the assembly action is less than or equal to a preset step threshold, determining the reward mode as a sparse reward mode.

[0075] In one embodiment, the reward design can be based on information such as the robot end-effector position and the force / torque generated during assembly, as shown in the following formula.

[0076] (3)

[0077] In the above formula, Total reward, including distance reward Success Rewards Step rewards and punishments and rewards Among them, distance reward Step rewards and punishments and rewards This is an instant reward, calculated in real-time; success reward. The success of the assembly task, as simulated earlier, determines whether it was completed. Success grants a positive reward, while failure incurs a negative penalty. (Distance Reward) Specifically, the distance between the robot's end effector and the target workpiece is determined. When they are close, a positive reward is given in real time; when they are far apart, a negative reward is given in real time. (Step reward) Rewards are given based on the number of steps the robot takes to complete a task, encouraging it to complete the task quickly with fewer steps. When the number of steps exceeds a certain threshold, a negative value is applied as a penalty. (Penalty / Reward) Generally, penalties are imposed for whether the simulated assembly actions during the assembly process of the intelligent agent exceed the preset force / torque / position thresholds. If they exceed the thresholds, a preset negative value is given; otherwise, the value is 0. and As a penalty weight, both are set to 1 in dense reward modes and 0 in sparse reward modes. Furthermore, distance reward varies depending on the mode. and success rewards The calculation methods are also different, which will be described in detail later.

[0078] In addition, there are different reward patterns: dense reward patterns and sparse reward patterns. Dense reward patterns are for complex tasks using continuous action spaces, while sparse reward patterns are for simple tasks using discrete action spaces. Therefore, the corresponding reward pattern needs to be determined before calculating the reward. This can be done by obtaining the task information executed by the initial agent. When the task information matches the complex task type and the number of assembly steps is greater than a preset step threshold, the reward pattern is determined to be dense reward pattern; when the task information matches the simple task type, or the number of assembly steps is less than or equal to the preset step threshold, the reward pattern is determined to be sparse reward pattern. The number of assembly steps is directly obtained from the simulated assembly actions.

[0079] If the task information indicates that the execution of the corresponding assembly action involves action strategy selection, parameter adjustment, or retry decision based on real-time perception information, the task is classified as a complex task and thus adopts a dense reward mode. Conversely, if the task information only involves the execution of a preset trajectory and does not require strategy adjustment based on perception feedback, it is classified as a simple task and adopts a sparse reward mode.

[0080] In addition, the specific reward calculation methods for dense reward mode and sparse reward mode also differ.

[0081] In one embodiment, when the reward mode is a dense reward mode, the assembly action is scored based on the reward mode, including: acquiring the real-time pose of the assembly action; determining the real-time coordinates corresponding to the real-time pose and acquiring the target coordinates of the target workpiece; calculating the error value between the real-time coordinates and the target coordinates, and determining the distance reward based on the error value; calculating the posture deviation and determining the success reward based on the preset ideal pose and the real-time pose; acquiring the number of execution steps of the assembly action and determining the step reward based on the number of execution steps; determining the number of out-of-limit states during the execution of the assembly action and determining the penalty reward based on the number of out-of-limit states; and weighted summing the distance reward, success reward, step reward, and penalty reward to generate a reward score in the dense reward mode.

[0082] In one implementation, the dense reward mode means that the reward is calculated and assigned in real time during the assembly process. Therefore, the end pose of the assembly action obtained at this time is real-time, referred to as the real-time pose. The real-time coordinates corresponding to the real-time pose are determined, and the target coordinates of the target workpiece are obtained. Both the real-time coordinates and the target coordinates include three sub-quantities, corresponding to the position above the hole center, the hole center position, and the hole bottom center position, respectively. The hole refers to the hole structure that needs to be inserted to complete the assembly action, and similar positioning holes or mating holes. The deviations between the real-time coordinates and the target coordinates in the three sub-quantities are calculated, and finally, a weighted sum is obtained to obtain the distance reward. The calculation method can be found in the following formula.

[0083] (4)

[0084] In the above formula, , , These are the preset weighting coefficients. , and These are the weight values ​​corresponding to the three sub-quantities. The calculation method for the real-time deviation between the robot end effector and the surface above the hole in three directions can be referred to in the following formula.

[0085] (5)

[0086] In the above formula, , and The preset top deviation weight. These are real-time coordinates, obtained in real time. The coordinates of the center of the top of the hole in the target coordinate system are preset values ​​that can be obtained directly.

[0087] The real-time deviations between the robot's end effector and the center of the hole in three directions can be calculated using the following formula.

[0088] (6)

[0089] In the above formula, , and The preset center deviation weight. The coordinates of the hole's center position in the target coordinate system can be obtained directly.

[0090] The real-time deviations in three directions corresponding to the position of the robot's end effector and the bottom of the hole can be calculated using the following formula.

[0091] (7)

[0092] In the above formula, , and The preset bottom deviation weight is used. The coordinates of the hole's center position in the target coordinate system can be obtained directly.

[0093] Success reward The calculation is performed using the preset ideal pose and the real-time pose, with reference to the following formula.

[0094] (8)

[0095] In the above formula, The preset initial success reward, This is the preset reward coefficient. Reward Coefficient An update will be performed after each successful assembly, following the rules below.

[0096] (9)

[0097] In the above formula, This represents the end-effector depth in the real-time pose. The target depth is defined as the depth within the ideal pose. When the end depth exceeds the target depth, the target workpiece is considered to have been inserted into the corresponding assembly hole, thus completing the assembly. Success or failure can also be determined directly using success or invalid labels obtained during simulated assembly.

[0098] Obtain the number of steps taken to perform the assembly action, and determine the step reward based on the number of steps taken. Step Reward The calculation method is shown in the following formula.

[0099] (10)

[0100] In the above formula, It is the real-time execution step count. The maximum number of steps to attempt per round is set to the preset number of steps. According to the above formula, the step reward... A negative number means that a penalty is imposed during the real-time reward process, thereby stimulating the agent to complete the assembly as quickly as possible.

[0101] Punishment and Reward The calculation method is shown in the following formula.

[0102] (11)

[0103] In the above formula, The penalty weight is a preset negative number. The above formula represents the number of times the assembly process exceeds the limit, with each instance receiving a corresponding penalty weight. The threshold values ​​for exceeding the limit set for the robot include, but are not limited to, force, torque, or position.

[0104] Ultimately, the distance to the reward will be... Success Rewards Step rewards and punishments and rewards The total reward is obtained by weighted summation according to the calculation method shown in formula (3). The total reward This is considered a reward score.

[0105] In one embodiment, when the reward mode is a sparse reward mode, the assembly action is scored based on the reward mode, including: obtaining the end-effector pose of the assembly action; determining whether the assembly task is completed based on the end-effector pose; if completed, a preset reward value is given; if not completed, no reward is given; determining whether the end-effector pose conforms to a preset ideal pose; if not, a preset penalty value is given; if conforms, no penalty value is given; and adding the reward value and the penalty value to generate a reward score under the sparse reward mode.

[0106] In one implementation, for the sparse reward pattern, referring to formula (3), a distance reward is also included. and success rewards However, the calculation methods are completely different. Among them, distance reward... If the value is negative, it is referred to as the penalty value. The specific calculation formula is shown in the following formula.

[0107] (12)

[0108] In the above formula, This is the preset penalty value; The ideal pose for the end effector is determined by ensuring it conforms to the preset values. Specifically, during execution, the end effector's pose continuously advances towards the ideal pose, meaning all executed steps are valid and the end effector's pose gradually approaches the ideal pose. Otherwise, a penalty value is applied.

[0109] For success rewards In sparse reward models, this is called the reward value. The calculation method is shown in the following formula.

[0110] (13)

[0111] In the above formula, The preset reward value; This indicates that when the depth at the end is greater than the target depth, the target workpiece is considered to have been inserted into the corresponding assembly hole, thus the assembly is considered complete. Success or failure can also be directly determined using success or invalid labels obtained from the simulated assembly.

[0112] Finally, the reward and penalty values ​​in the sparse reward mode are added together to generate the reward score for the sparse reward mode. It is worth noting that the calculation difference between the sparse reward mode and the dense reward mode is that the calculation in the dense reward mode is performed in real time; while in the sparse reward mode, since it is for discrete execution steps, the calculation is performed step by step, calculating the reward for each step, and finally summing them to obtain the corresponding reward score.

[0113] Step S140: Train the initial agent using reward scores and iteratively optimize the initial agent until it meets the preset convergence conditions.

[0114] Step S150: Mark the initial agent that has completed training as an agent and output it.

[0115] In one embodiment, an initial agent is trained using reward scores, and the initial agent is iteratively optimized until a preset convergence condition is met. This includes: combining assembly actions and corresponding joint states, reward scores, and completion markers to generate samples, where the completion markers indicate whether the assembly action is completed; summarizing all samples to obtain a training sample set; acquiring a preset training network, which is constructed using a deep reinforcement learning algorithm or an actor-critic algorithm; inputting the training sample set into the training network and adjusting the policy parameters of the initial agent through policy gradient updates; in each iteration, executing assembly actions according to the current policy and collecting new samples to continuously update the training sample set; iteratively training until the policy parameters meet a preset convergence condition, which includes at least one of the following: the change in policy parameters is less than a preset threshold, the rate of change of the mean reward score within a preset training period is less than a preset threshold, and the number of iterations reaches a preset upper limit.

[0116] In one implementation, the initial agent is trained, meaning it needs to learn how to process outputs more quickly and accurately based on robot state and task information to produce assembly actions that can rapidly and accurately complete the assembly requirements. Training requires samples; a single sample consists of a quadruple of assembly action and corresponding joint state combination, reward score, and completion marker. The assembly action is the performed action; the joint state combination is the state of each joint of the robot; the reward score is the value obtained from the previously calculated process; and the completion marker determines whether the assembly action was completed, including success or failure. All samples are then aggregated to obtain a sample set.

[0117] Obtain a pre-defined training network, which is constructed using either a Deep Q-Network (DQN) algorithm or an Actor-Critic (AC) algorithm. Different algorithms will employ different training processes.

[0118] When using the AC algorithm, two MLP (Multi-Layer Perceptron) networks are required, serving as the policy network and the value network respectively. Sharing feature layers between the two networks improves the efficiency of parameter tuning, reduces redundancy in feature extraction, and mitigates overfitting.

[0119] When using the DQN algorithm, the training process also involves two networks: a main network and a target network. However, only one MLP network is needed. The target value is calculated by replicating the main network as the target network. This dual-network structure helps to solve problems such as training instability and estimation bias in reinforcement learning.

[0120] When choosing a reinforcement learning algorithm for discrete action spaces, DQN selects SmoothL1Loss as its loss function, as shown in the formula below.

[0121] (14)

[0122] In the above formula, It is a predicted value. It is the target value. It is an adjustable hyperparameter, which can be set to δ=1 in a preferred implementation. Using this loss function helps improve the model's robustness to outliers and noise, maintains the stability of the training process, and provides smooth gradients.

[0123] A pre-prepared training sample set is input into the training network. The initial policy parameters of the agent are adjusted using policy gradient updates. This adjustment aims to better align the agent's policy with the training objectives and requirements. In each iteration of the training process, the agent generates assembly actions based on the currently adopted policy, iteratively tuning parameters to ensure the generated assembly actions better accomplish the task.

[0124] Iterative training continues until the policy parameters meet preset convergence conditions. These preset convergence conditions include at least one of the following: the change in policy parameters is less than a preset threshold, the rate of change of the average reward score within a preset training period is less than a preset threshold, and the number of iterations reaches a preset upper limit.

[0125] Repeated training allows the initial agent to continuously learn assembly contact skills. During the iteration process, the assembly environment can be repeatedly configured and randomized, enabling the initial agent to better adapt to the real physical environment. The final trained initial agent is then labeled as the agent and output.

[0126] Correspondingly, this application also provides an application method, namely a robot assembly control method, including steps S210 to S230.

[0127] Step S210: Obtain the pre-trained agent.

[0128] In one implementation, the training process for the agent has been described in detail above and will not be repeated here.

[0129] Step S220: Input the acquired environmental data at the current moment into the agent, adjust the parameters of the agent, and obtain the implementing agent.

[0130] Step S230: The task instruction is input into the implementing agent, and the assembly action is executed according to the strategy output by the implementing agent.

[0131] In one implementation, it is understood that the training process described above involves both sim and sim2real steps, but a final 2real step is also required. This necessitates acquiring real-world physical data to tune the agent's parameters and adapt it to the real environment. First, environmental data at the current moment is acquired, including but not limited to the robot's joint states, the stiffness, damping, and friction of the assembly environment, etc. This data is input into the trained agent to tune its parameters, enabling it to better simulate the real environment. This results in the implemented agent, also known as the assembly control model, which is then configured in the corresponding hardware.

[0132] The physical robot assembly system includes the aforementioned components and the robot system. The robot system comprises hardware and software. The hardware consists of three parts: the robot (6 degrees of freedom), the lower-level computer (robot control cabinet), and the upper-level computer (personal PC). The software includes the intelligent agent described earlier. After establishing a kinematic model based on the robot parameters, then based on... Figure 4 The communication mechanism shown enables communication between the industrial robot, the six-dimensional force sensor, the host computer, and the assembly control model (i.e., the implementing agent). Joint data is transmitted to the host computer via the ROS message type `sensor_msgs / JointState`, and the robot's pose is then calculated using the robot's forward kinematics on the host computer. The ROS driver package based on the six-dimensional force sensor acquires the force / torque information of the robot's end effector and transmits it via the message type `geometry_msgs / Wrench`. Robot data and sensor data are transmitted as state information to the host computer, which then passes this state information to the skill model. Based on the input, the skill model outputs the corresponding assembly strategy, i.e., the robot's pose increment. The robot's joint angles are then calculated using the robot's inverse kinematics on the host computer. The host computer also transmits the joint angles to the industrial robot via the message type `sensor_msgs / JointState`, and then controls the robot's movement. This process is repeated, updating the assembly action strategy in real time based on robot and sensor data, ultimately achieving automated robot assembly.

[0133] Based on the physical assembly platform, use a teach pendant to open the robot control panel, set the automatic motion mode, and launch the ROS communication file "ros_server". Then, run the roslaunch file on the PC to start the relevant ROS nodes, initialize the robotic arm, randomize the relative poses of the axes and holes, and transmit / update the current state information. Then, issue the corresponding task instructions, and obtain the assembly strategy through the implementing agent. The robot will automatically execute this assembly strategy. After execution, it will determine whether a threshold has been exceeded, whether the assembly was successful, update the assembly data, and then update the assembly strategy until assembly is complete.

[0134] Furthermore, this application employs a SimReal-based reinforcement learning algorithm to train the assembly control model. The model is trained in simulation and then transferred to a real assembly environment. Alternative solutions exist, such as using traditional assembly control methods (impedance control, force-position hybrid control) or imitation learning-based assembly control methods to directly acquire data and train the model in the actual environment. Additionally, the original solution's end-effector pose incremental control can be replaced with joint angle incremental control. However, these alternative solutions are less effective and are not considered preferred implementations, but rather optional solutions for demonstration purposes.

[0135] The robot assembly training method provided in this application includes the following steps: constructing an initial agent, generating a preset number of joint state combinations based on the initial agent; controlling the simulated robot to perform assembly actions according to the joint state combinations in a simulated environment; determining a reward mode, scoring the executed assembly actions based on the reward mode to obtain a reward score; training the initial agent using the reward score, iteratively optimizing the initial agent until a preset convergence condition is met; marking the trained initial agent as an agent and outputting it. The provided robot assembly control method includes the following steps: obtaining a pre-trained agent, which is trained using the robot assembly training method described above; inputting the acquired environmental data at the current moment into the agent, tuning the agent's parameters to obtain an implementing agent; inputting task instructions into the implementing agent, and executing assembly actions according to the strategy output by the implementing agent. Therefore, this application can train an adaptive assembly control model by using dynamic physical parameter randomization, which can adapt to more complex and variable assembly environments. Furthermore, during training, different reward modes are used to differentiate the training of the agent, enabling the agent to have stronger generalization ability and decision-making accuracy when facing different assembly tasks. Furthermore, the method provided in this application can automatically learn assembly strategies based on a simulation environment. It can be trained using a large amount of virtual data, which accelerates the training process and shortens the development cycle. In addition, the technical solution of this application has stronger environmental adaptability. Existing technologies typically use fixed control parameters, mainly relying on pre-programmed fixed trajectory control, PID force control strategies, force / position hybrid control, or vision-based servo control, resulting in poor environmental adaptability. This application, however, uses dynamic physical parameter randomization to train the model, enabling it to adapt to more complex and variable assembly environments. Regarding development efficiency, existing technologies are time-consuming in terms of manual parameter tuning and data acquisition, and pose certain safety risks during experimentation. This application can automatically learn assembly strategies based on a simulation environment, can be trained using a large amount of virtual data, which accelerates the training process and shortens the development cycle.

[0136] Figure 5 An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 5 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 5 The processor 310 shown in the diagram does not indicate that there is only one processor 310, but only indicates the positional relationship of the processor 310 relative to other devices. In practical applications, there can be one or more processors 310; similarly, Figure 5The memory 311 illustrated herein has the same meaning, that is, it is only used to indicate the positional relationship of memory 311 relative to other devices. In practical applications, there can be one or more memories 311. When the processor 310 runs the computer program, the method applied to the above-mentioned device is implemented.

[0137] The device may also include at least one network interface 312. The various components of the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to implement communication between these components. In addition to a data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general designated all buses as Bus System 313.

[0138] The memory 311 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 311 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0139] The memory 311 in this embodiment of the invention is used to store various types of data to support the operation of the device. Examples of this data include: any computer programs used to operate on the device, such as operating systems and applications; contact data; phonebook data; messages; pictures; videos, etc. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications, such as media players, browsers, etc., used to implement various application services. Here, the program implementing the method of this embodiment of the invention can be included in the application.

[0140] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program. The computer-readable storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc. When the computer program stored in the computer-readable storage medium is run by a processor, it implements the above method. For the specific steps implemented when the computer program is executed by the processor, please refer to [link to relevant documentation]. Figure 1 The description of the illustrated embodiments will not be repeated here.

[0141] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0143] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot assembly training method, characterized in that, Includes the following steps: Construct an initial intelligent agent, and generate a preset number of joint state combinations based on the initial intelligent agent; In a simulated environment, the simulated robot is controlled to perform assembly actions based on the joint state combination; Determine the reward mode, and score the performed assembly action based on the reward mode to obtain a reward score; The method for determining the reward mode includes: acquiring task information executed by the initial agent; determining the reward mode as a dense reward mode when the task information matches a complex task type and the number of steps in the assembly action is greater than a preset step threshold; and determining the reward mode as a sparse reward mode when the task information matches a simple task type or the number of steps in the assembly action is less than or equal to a preset step threshold. When the reward mode is a dense reward mode, scoring the assembly action based on the reward mode includes: acquiring the real-time pose of the assembly action; determining the real-time coordinates corresponding to the real-time pose, and acquiring the target coordinates of the target workpiece; calculating the error value between the real-time coordinates and the target coordinates, and determining a distance reward based on the error value; calculating the posture deviation and determining a success reward based on a preset ideal pose and the real-time pose; acquiring the number of execution steps of the assembly action, and determining a step reward based on the number of execution steps; determining the number of out-of-limit states during the execution of the assembly action, and determining a penalty reward based on the number of out-of-limit states; and weighted summing the distance reward, the success reward, the step reward, and the penalty reward to generate a reward score under the dense reward mode. When the reward mode is a sparse reward mode, the step of scoring the assembly action based on the reward mode includes: obtaining the end-effector pose of the assembly action; determining whether the assembly task is completed based on the end-effector pose; if completed, giving a preset reward value; if not completed, not giving any reward; determining whether the end-effector pose conforms to a preset ideal pose; if not, giving a preset penalty value; if conforms, not giving a penalty value; adding the reward value and the penalty value to generate a reward score under the sparse reward mode. The initial agent is trained using the reward score, and the initial agent is iteratively optimized until a preset convergence condition is met. The initial agents that have completed training are labeled as agents and output.

2. The robot assembly and training method as described in claim 1, characterized in that, The generation of a preset number of joint state combinations based on the initial agent includes: Obtain design parameters, including the number of robot joints, range of motion, and degree of freedom configuration; A preset number of joint state combinations to be screened are randomly generated based on the design parameters; The feasibility of the proposed joint state combinations to be screened is verified. The joint state combinations that pass the feasibility verification are marked as the joint state combinations.

3. The robot assembly and training method as described in claim 1, characterized in that, The step of controlling the simulated robot to perform assembly actions based on the joint state combination includes: The simulated robot is initialized, and the pose of the target workpiece is randomized, wherein the target workpiece is the target of the robot's task. Based on the joint state combination, corresponding control commands are generated to drive the simulated robot end effector to move toward the target workpiece; Determine whether the simulated robot successfully performed the assembly action; If successful, save the assembly action; If the attempt fails, the assembly process is repeated until a preset threshold number of attempts is reached; if the attempt still fails after reaching the threshold, the joint state combination is marked as invalid.

4. The robot fitting and training method as described in claim 3, characterized in that, The initialization of the simulated robot and randomization of the target workpiece pose include: An elastic body is provided at the target workpiece, the elastic body being used to connect the target workpiece and the simulated robot; A random disturbance force is applied to the elastic body, the random disturbance force including at least one of stiffness, damping, friction coefficient and direction variable.

5. The robot assembly and training method as described in claim 1, characterized in that, The step of training the initial agent using the reward score and iteratively optimizing the initial agent until it meets a preset convergence condition includes: The assembly action and the corresponding joint state, the reward score, and the completion mark are combined to generate a sample, where the completion mark indicates whether the assembly action has been completed; all the samples are then aggregated to obtain a training sample set. Obtain a preset training network, which is constructed using a deep reinforcement learning algorithm or an actor-critic algorithm; The training sample set is input into the training network, and the policy parameters of the initial agent are adjusted by updating the policy gradient. In each iteration, the assembly action is performed according to the current policy and new samples are collected to continuously update the training sample set. The training is iterated until the policy parameters meet the preset convergence conditions. The preset convergence conditions include at least one of the following: the change in the policy parameters is less than a preset threshold, the rate of change of the average reward score within a preset training period is less than a preset threshold, and the number of iterations reaches a preset upper limit.

6. A robot assembly control method, characterized in that, Includes the following steps: A pre-trained intelligent agent is obtained, wherein the intelligent agent is trained by the robot fitting training method according to any one of claims 1 to 5; The acquired environmental data at the current moment is input into the intelligent agent, and the parameters of the intelligent agent are adjusted to obtain the implementing intelligent agent; The task instructions are input into the implementing agent, and the assembly action is executed according to the strategy output by the implementing agent.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the robot fitting training method as described in any one of claims 1 to 5, and / or the robot fitting control method as described in claim 6.

Images