Robot rapid jack inserting method, device and equipment and storage medium

By using distributed maximum entropy reinforcement learning algorithm in the simulation environment, and switching to the trained controller for jacking in the real environment, the problem that the robot jack method is difficult to adapt to task changes and environmental disturbances is solved, and the robot jacking method is realized. Fast and low-cost jack operation.

CN118990506APending Publication Date: 2024-11-22JIHUA LAB

Patent Information

Application Number
CN202411353157.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-26
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing robot jack methods are difficult to adapt to task changes and environmental disturbances, and the data acquisition cost is high. Traditional mechanical analysis methods rely on precise modeling and have high contact force. Deep learning methods are difficult to apply in real scenarios, which can easily cause damage to workpieces and equipment.

Method used

Use distributed maximum entropy reinforcement learning algorithm to train the robot high-level controller in a simulation environment to obtain the trained controller, and move the robot to the target hole position through the environmental camera and the open loop planner in the real environment , switch to the trained controller for jack task.

Benefits of technology

The robot can quickly adapt to the jack needs of various types of terminals, reduce the amount of training data in real scenes, reduce the requirements of the environmental camera, and complete the jack actions under noisy conditions, avoiding damage to workpieces and equipment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118990506A_ABST
    Figure CN118990506A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of intelligent assembly of robots, and discloses a quick jack inserting method, device and equipment for a robot and a storage medium. The method comprises the following steps: in a simulation environment, performing terminal jack training on a high-level controller of a robot by using a distributed maximum entropy reinforcement learning algorithm to obtain a trained controller; in a real environment, a target hole site is photographed through an environment camera, and an open-loop planner is used for moving a robot end effector to the position above the target hole site; and switching the robot to the trained controller, and controlling the end effector to process the terminal jack task through the trained controller. The method does not depend on establishment of an object contact force model, and can quickly adapt to jack requirements of various types of terminals; compared with a deep learning method, the method has the advantages that the real scene training data volume required in the jack inserting process can be reduced, the requirement for an environment camera is low, common shielding and other conditions which are not beneficial to visual observation in the jack inserting process can be overcome, and the jack inserting action is completed under the noise condition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of robot intelligent assembly, and in particular to a robot quick jacking method, device, equipment and storage medium. Background Art

[0002] The plug-in action is an assembly production process seen in industrial production, usually completed by workers manually. Special-shaped terminals in the production of information appliances (3C) include power terminals, antenna terminals, flexible printed circuit terminals, etc. For robot arm assembly, terminal bending not only requires accurate alignment, but also requires moderate buckling force. Excessive buckling force will damage the parts, and too little buckling force will cause assembly failure.

[0003] Existing jacking methods can be divided into two categories: traditional mechanical analysis methods and deep learning methods. In traditional methods that rely on mechanical analysis, the parameters of the controller must be designed for each assembly stage by accurately modeling the robot, actuator, and terminal. However, this approach that relies entirely on mechanical models not only makes it difficult for the robot to adapt to changes in tasks and environmental disturbances, but also often requires a larger contact force to meet the requirements of low fault tolerance. Since the detection accuracy of the checkerboard during the jacking process directly affects the accuracy of the camera calibration results, the application of deep learning-based methods in real scenarios is still very difficult. In addition, since the robot needs to interact with the environment for trial and error, especially the jacking process needs to deal with complex and changeable contact force changes, if the algorithm is trained directly on the physical robot, it is easy to cause damage to the workpiece and equipment, and has a high data acquisition cost.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a robot quick jacking method, device, equipment and storage medium, aiming to solve the problem that the existing robot jacking method is difficult to adapt to task changes and environmental disturbances, as well as the high cost of data acquisition.

[0006] The first aspect of the present invention provides a robot quick jacking method, comprising: in a simulation environment, using a distributed maximum entropy reinforcement learning algorithm to train the robot's high-level controller for terminal jacking to obtain a trained controller; in a real environment, taking a picture of a target hole position through an environmental camera, and using an open-loop planner to move the robot's end effector to above the target hole position; switching the robot to the trained controller, and controlling the end effector to process the terminal jacking task through the trained controller.

[0007] Optionally, in a first implementation of the first aspect of the present invention, in a simulation environment, a distributed maximum entropy reinforcement learning algorithm is used to perform terminal jack training on a high-level controller of the robot to obtain a trained controller, comprising the following steps: the high-level controller of the robot includes a discriminator network and an actor network, both the discriminator network and the actor network use a three-layer MLP network, each layer of the MLP network contains 256 neurons and a Gaussian error linear unit; in a simulation environment, the same robot and terminal to be connected as in a real environment are used, and the robot end effector is set to hold the terminal above the target of the terminal seat to be connected to start jack training; the high-level controller is trained by a distributed maximum entropy reinforcement learning algorithm to obtain the optimal action strategy, that is, let the value function:

[0008] Maximize to obtain the trained controller, where represents mathematical expectation, s represents state, a represents action, represents the policy entropy, r represents the single-step reward, γ represents the discount factor, t represents the time, π represents the policy function, and α represents the temperature parameter, which is used to characterize the policy entropy. The relative importance of the term relative to the single-step reward r; through the two steps of strategy evaluation and strategy update, it is iterated to find the one that satisfies the formula J π The optimal strategy π and its corresponding optimal value function J π The conditions of , where the policy evaluation gradient is:

[0009]

[0010] The policy update gradient is:

[0011] Among them, θ is the parameter of the discriminator network, Φ is the parameter of the actor network, represents the sampled experience replay cache, and s' represents the next state reached by action a starting from state s.

[0012] Optionally, in a second implementation of the first aspect of the present invention, the state value s of the robot end effector at time t is t The pose of the robot end effector and the force twist W, namely Among them, the posture of the robot end effector is a set of 6-dimensional vectors, which represents the coordinates of the current position of the robot end effector in the coordinate system constructed by the target position. The force torque W is composed of the three-dimensional force f and the three-dimensional torque τ on the robot end effector; the action value a of the robot end effector at time t is t Set to the change in the actuator's target pose, i.e. at =(Δx, Δy, Δz, Δrx, Δry, Δrz).

[0013] Optionally, in a third implementation of the first aspect of the present invention, the reward function r t The design is as follows:

[0014]

[0015] Optionally, in a fourth implementation of the first aspect of the present invention, the robot is switched to a trained controller, and the trained controller is used to control the end effector to process the terminal jack task, including the steps of: switching the robot to the trained controller, and generating an action a by the trained controller; t is the change in the target position of the robot actuator; when the trained controller outputs action a at a frequency of 25 Hz t When the value is, let the current trajectory point be x t , then the target motion trajectory point at the next moment will be updated to x t+1 =x t +a t , the underlying controller runs at 1kHz, so when x t and a t During the update time interval, the bottom-level controller will perform 1000 / 25=40 calculations to obtain the robot joint accelerations and perform inverse kinematics calculations to obtain the joint torques τ, controlling the robot's motion from x t Movement to x t+1 ,in, τ is the joint space force or torque; x t is the desired position; T is the kinematic equation, is the transformation matrix from the base coordinate system to the end coordinate system; q is the joint position vector; J is the Jacobian matrix; K pp is the proportional coefficient of the position channel; K pi is the integral coefficient of the position channel; is the expected speed; is the joint velocity vector; K pd is the proportional coefficient of the velocity channel; C p is the selection matrix for position and velocity channels.

[0016] The second aspect of the present invention provides a robot quick jacking device, including: a simulation training module, which is used to use a distributed maximum entropy reinforcement learning algorithm to perform terminal jack training on the robot's high-level controller in a simulation environment to obtain a trained controller; a moving module, which is used to take pictures of the target hole position through an environmental camera in a real environment, and use an open-loop planner to move the robot's end effector above the target hole position; and a jacking module, which is used to switch the robot to the trained controller, and control the end effector to process the terminal jacking task through the trained controller.

[0017] The third aspect of the present invention provides a robot quick jack device, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the computer-readable instructions in the memory so that the robot quick jack device executes the various steps of the robot quick jack method as described above.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium, in which computer-readable instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the various steps of the robot quick jacking method as described above.

[0019] Beneficial effects: The present invention uses the distributed maximum entropy reinforcement learning algorithm DSAC in a simulation environment to train the robot to plug in holes, and uses the same robot and terminals to be connected as in the real environment. Then, in the real environment, first take a photo of the target hole position through the environmental camera, use the open-loop planner to move the robot end above the target position, and then switch to the trained DSAC controller to handle the plugging task. The present invention provides a robot fast plugging method with small sample characteristics. Compared with traditional mechanical analysis methods, this method does not rely on establishing an object contact force model, and can quickly adapt to the plugging needs of various types of terminals; compared with other deep learning methods, it can greatly reduce the amount of real scene training data required for the plugging process, has lower requirements for environmental cameras, and can overcome common occlusions in the plugging process that are not conducive to visual observation, and complete the plugging action under noisy conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a robot quick jacking method provided in an embodiment of the present invention.

[0021] Figure 2 A schematic diagram of a robot quick jacking method provided in an embodiment of the present invention.

[0022] Figure 3 A schematic diagram of the starting position of the robot during simulation training provided by an embodiment of the present invention.

[0023] Figure 4 A schematic diagram of the jack contact of a robot during simulation training provided by an embodiment of the present invention.

[0024] Figure 5 This is a schematic structural diagram of the robot quick jack device provided by the present invention.

[0025] Figure 6 This is a schematic structural diagram of the robot quick jack device provided by the present invention. DETAILED DESCRIPTION

[0026] Embodiments of the present invention provide a robot quick jacking method, apparatus, device and storage medium. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] See also Figure 1 , Figure 1 A flow chart of a robot quick jacking method provided by the present invention, as shown in the figure, comprises the following steps:

[0028] S10, in a simulation environment, using a distributed maximum entropy reinforcement learning algorithm to perform terminal jack training on a high-level controller of the robot to obtain a trained controller;

[0029] Specifically, in the simulation training phase, the same robots and terminals to be connected as those in the real environment are used in the simulation environment. Figure 2 As shown, the robot includes a high-level controller and a low-level controller (i.e., a force-position hybrid controller) electrically connected to the high-level controller. The high-level controller of the robot includes a discriminator network and an actor network. Both the discriminator network and the actor network use a three-layer MLP network, and each layer of the MLP network contains 256 neurons and a Gaussian error linear unit.

[0030] In this embodiment, the discriminator network and the actor network are tightly integrated in the high-level controller to form a closed-loop control system; data flows into the discriminator network from the robot sensor or other data sources, and is output to the actor network after processing. The actor network generates control commands based on this information and drives the robot to perform corresponding actions through the actuator; at the same time, the current state information of the robot may also be fed back to the discriminator network and the actor network to achieve more accurate control and decision-making. This embodiment uses the MLP network as the basic structure of the discriminator network and the actor network, which has strong nonlinear modeling and generalization capabilities and can handle complex control tasks; each layer contains 256 neurons, which provides sufficient capacity to learn complex feature representations and control strategies; Gaussian error linear units are used as activation functions to help increase the nonlinear ability and stability of the network.

[0031] During training, first set the robot end effector to hold the terminal at a predetermined distance above the target terminal seat to be connected and start the jack training, such as Figure 3 As shown. Since this scheme only considers the plugging process and not the robot grasping process, it is assumed that the robot end effector starts with the terminal holding state. In the simulator, if the six-degree-of-freedom posture of the terminal is satisfied, that is, the distance between the center point in the x, y, and z directions and the target is less than 0.1 cm, and rx, ry, and rz are all less than 0.1 rad, the plugging is considered successful. If the target posture is not reached after 200 calculation steps, the plugging is considered unsuccessful.

[0032] In this embodiment, the high-level controller is simulated and trained by a distributed maximum entropy reinforcement learning algorithm to obtain the best action strategy, that is, the value function is:

[0033] Maximize to obtain a trained controller, through which the trained controller achieves Figure 4 The jacking process shown, where represents mathematical expectation, s represents state, a represents action, represents the policy entropy, r represents the single-step reward, γ represents the discount factor, t represents the time (discrete moment, i.e. t = 0, 1, 2, ... ∞), π represents the policy function, and α represents the temperature parameter, which is used to characterize the policy entropy The relative importance of the term to the single-step reward r. Compared with the traditional reinforcement learning algorithm, this value function adds policy entropy The term improves the exploration efficiency of the policy function π. In this embodiment, when the value function is maximized, it means that the reinforcement learning algorithm has reached the optimal solution in theory, that is, the optimal high-level controller is found, which does not mean that the terminal is successfully installed. Even the optimal controller may fail to install, but the optimal controller has a higher success rate than the non-optimal one during installation.

[0034] In this embodiment, the policy entropy Can be written as in, Represents the feasible domain of action a.

[0035] Through the two steps of strategy evaluation and strategy update, we iteratively search for the solution that satisfies the formula J π The optimal strategy π and its corresponding optimal value function J π The conditions of , where the policy evaluation gradient is:

[0036] The symbol on the left side of the formula represents the direction of gradient update, which is optimized by updating the parameters θ of the discriminator critic network. π The evaluation value J z (θ), where z(φ) represents a Gaussian distribution;

[0037] The policy update gradient is:

[0038] By updating the parameters Φ of the actor network, the value function J is optimized. π The evaluation value J x (φ).

[0039] The state value s of the robot end effector at time t t The pose of the robot end effector and the force twist W, namely Among them, the posture of the robot end effector is a set of 6-dimensional vectors, which represents the coordinates of the current position of the robot end effector in the coordinate system constructed by the target position. Therefore, ideally, the algorithm should make the actuator posture approach the target posture, that is, make the 6-dimensional vector approach 0. In specific implementation, it is necessary to add Gaussian noise to the target posture to simulate the uncertainty in the real scene; the force spinor W is composed of the 3D force f and 3D torque τ exerted on the robot end effector.

[0040] Furthermore, since operations in real environments are often accompanied by various delay phenomena, such as sensor perception delay, control strategy calculation delay, etc., these delay phenomena are often ignored or simplified in virtual simulation training environments, resulting in large differences between training results and actual applications. This embodiment achieves virtual-real migration through time delay modeling and historical observation sequence technology, which can reduce the difference between the simulated environment and the real environment and improve the consistency of training results.

[0041] Specifically, time delay modeling is to add policy calculation delay and sensor perception delay to simulation training to simulate the delay phenomenon in the real environment. This modeling method helps to make the training process in the virtual environment closer to the operating conditions of the real environment, thereby improving the effectiveness and reliability of the training results. In the experiment, the delay modeling was set to obey the uniform distribution of the interval [0.015s, 0.025s], which means that in each simulation training, the delay of policy calculation and sensor perception will randomly change within this range to simulate the diversity and uncertainty of delay in the real environment. This random change helps to enhance the generalization ability and robustness of the training results.

[0042] Time delay modeling focuses on simulating the delay phenomenon in the real environment, while historical observation sequence technology focuses on using historical data to understand and analyze the system. The two can complement each other in the virtual-real migration process and jointly improve the effectiveness and reliability of the training results. Time delay modeling makes the training process closer to the real environment, while historical observation sequence technology provides rich data support, which helps to more accurately describe and predict the behavior of the system. In this embodiment, the historical observation sequence means that in the actual computing state s t When s t Represented as a sequence That is, a sequence of 10 moments from t-9 to t, and then take its mean The value function J integrated into the above training formula π When using Replace s in the formula t Can drive optimization and improvement of models or strategies.

[0043] In this embodiment, the action value a of the robot end effector at time t is t Set to the change in the actuator's target pose, i.e. a t =(Δx, Δy, Δz, Δrx, Δry, Δrz).

[0044] In this embodiment, the value function J π The reward function r t The design is as follows:

[0045]

[0046] In some specific embodiments, the simulation environment of this embodiment uses DeepMind Control Suite and uses MuJoCo as the simulation physics engine. DeepMind Control Suite is a powerful reinforcement learning agent performance evaluation benchmark developed by Google DeepMind. It is written in Python and supported by the MuJoCo physics engine, providing a series of stable, well-tested, easy-to-use and modify continuous control tasks; MuJoCo (Multi-Joint Dynamics with Contact) is a general physics engine maintained by Google DeepMind, designed to provide high-speed and accurate rigid body dynamics simulation. It is particularly suitable for modeling complex structures in interactive environments and supports a variety of complex physical interactions, such as contact forces. The implementation of the robot in this embodiment can use a 6-axis or 7-axis collaborative robot arm. In this embodiment, a 7-axis KUKA LBR iiwa is used, but the algorithm implementation is not limited to this model robot. In the simulation, the end effector is simplified to a custom clamping tooling, and in actual deployment, Robotiq 2f-85 two-claw or soft-touch flexible claw is selected as the end effector.

[0047] S20, in the real environment, taking a picture of the target hole position through the environment camera, and using the open-loop planner to move the robot end effector to the top of the target hole position;

[0048] In this embodiment, first ensure that the environmental camera is accurately installed and calibrated, and can accurately capture the location information of the target hole. After taking pictures, use image processing technology (such as edge detection, template matching, etc.) to identify the location of the target hole from the picture taken by the camera; based on the identified hole position, use an open-loop planner (such as RRT) to calculate the optimal path from the current position to the top of the target hole, and finally convert the planned path into a robot control instruction to drive the robot end effector to move to the specified position. In this embodiment, the robot can adapt to changes in different environments and workpieces by visually identifying the target position. Image processing technology improves the accuracy of target positioning and helps to improve the success rate of jacking; the open-loop planner can quickly calculate the moving path and reduce waiting time.

[0049] S30, switching the robot to a trained controller, and controlling the end effector to process the terminal jack task through the trained controller.

[0050] In this embodiment, after the robot reaches the target hole position, the control is switched from the open-loop planner to the trained controller, and the strategy of the trained controller is used to control the robot end effector to perform precise jacking operations, including adjusting posture, applying appropriate force, etc. Specifically, the action a generated by the trained controller tAs the change in the target position of the robot actuator; when the trained controller outputs action a at a frequency of 25 Hz t When the value is, let the current trajectory point be x t , then the target motion trajectory point at the next moment will be updated to x t+1 =x t +a t , the underlying controller runs at 1kHz, so when x t and a t During the update time interval, the bottom-level controller will perform 1000 / 25=40 calculations to obtain the robot joint accelerations and perform inverse kinematics calculations to obtain the joint torques τ, controlling the robot's motion from x t Movement to x t+1 ,in, τ is the joint space force or torque; x t is the desired position; T is the kinematic equation, is the transformation matrix from the base coordinate system to the end coordinate system; q is the joint position vector; J is the Jacobian matrix; K pp is the proportional coefficient of the position channel; K pi is the integral coefficient of the position channel; is the expected speed; is the joint velocity vector; K pd is the proportional coefficient of the velocity channel; C p is the selection matrix for position and velocity channels.

[0051] The present invention realizes a robot fast plugging method with small sample characteristics. Compared with the traditional mechanical analysis method, this method does not rely on establishing an object contact force model and can quickly adapt to the plugging needs of various types of terminals. Compared with other deep learning methods, the amount of real scene training data required for the plugging process can be greatly reduced, the requirements for environmental cameras are low, and it can overcome the common occlusions and other conditions that are not conducive to visual observation during the plugging process, and complete the plugging action under noisy conditions.

[0052] The above describes the robot quick jacking method in the embodiment of the present invention. The following describes the robot quick jacking device in the embodiment of the present invention. Figure 5 , an embodiment of the robot quick jack device in the embodiment of the present invention includes:

[0053] A simulation training module 10 is used to perform terminal jack training on a high-level controller of the robot using a distributed maximum entropy reinforcement learning algorithm in a simulation environment to obtain a trained controller;

[0054] The mobile module 20 is used to take a picture of the target hole position through an environmental camera in a real environment, and use an open-loop planner to move the robot end effector to the top of the target hole position;

[0055] The jack module 30 is used to switch the robot to a trained controller, and control the end effector to process the terminal jack task through the trained controller.

[0056] Based on the same idea as the method in the above-mentioned embodiment, the device provided in the present application can implement the method in the above-mentioned embodiment. For the convenience of explanation, the structural diagram of the device embodiment only shows the parts related to the embodiment of the present application. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer modules than shown in the figure, or a combination of certain modules, or different module arrangements.

[0057] Figure 5 The robot quick plug-in device in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the robot quick plug-in device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0058] Figure 6 1 is a schematic diagram of the structure of a robot quick jack device provided by an embodiment of the present invention. The robot quick jack device 100 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 11 (for example, one or more processors) and a memory 12, and one or more storage media 13 (for example, one or more mass storage devices) storing application programs 133 or data 132. Among them, the memory 12 and the storage medium 13 can be short-term storage or permanent storage. The program stored in the storage medium 13 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the robot quick jack device 100. Furthermore, the processor 11 can be configured to communicate with the storage medium 13 to execute a series of instruction operations in the storage medium 13 on the robot quick jack device 100.

[0059] The robot quick jack device 100 may also include one or more power supplies 14, one or more wired or wireless network interfaces 15, one or more input and output interfaces 16, and / or one or more operating systems 131, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 6 The device structure shown does not constitute a limitation on the robot quick jack device 100, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0060] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the robot quick jacking method.

[0061] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0062] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0063] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot quick jacking method, characterized in that: Includes steps: In the simulation environment, the distributed maximum entropy reinforcement learning algorithm is used to train the high-level controller of the robot for terminal jack insertion, and a trained controller is obtained; In the real environment, the target hole position is photographed by the environment camera, and the robot end effector is moved to the top of the target hole position using the open-loop planner; The robot is switched to a trained controller, through which the end effector is controlled to process the terminal jack task.

2. The robot quick jacking method according to claim 1, characterized in that: In the simulation environment, the distributed maximum entropy reinforcement learning algorithm is used to train the high-level controller of the robot for terminal jack insertion, and the trained controller is obtained, including the following steps: The high-level controller of the robot includes a discriminator network and an actor network, and both the discriminator network and the actor network use a three-layer MLP network, and each layer of the MLP network contains 256 neurons and a Gaussian error linear unit; In the simulation environment, the same robot and terminals to be connected as in the real environment are used, and the robot end effector is set to hold the terminal above the target terminal seat to be connected to start the jack training; The high-level controller is trained by a distributed maximum entropy reinforcement learning algorithm to obtain the best action strategy, that is, the value function is: Maximize to obtain the trained controller, where represents mathematical expectation, s represents state, a represents action, represents the policy entropy, r represents the single-step reward, γ represents the discount factor, t represents the time, π represents the policy function, and α represents the temperature parameter, which is used to characterize the policy entropy The relative importance of the term relative to the single-step reward r; Through the two steps of strategy evaluation and strategy update, we iteratively search for the solution that satisfies the formula J π The optimal strategy π and its corresponding optimal value function J π The conditions of , where the policy evaluation gradient is: The policy update gradient is: Among them, θ is the parameter of the discriminator network, Φ is the parameter of the actor network, represents the sampled experience replay cache, and s' represents the next state reached by action a starting from state s.

3. The robot quick jacking method according to claim 2, characterized in that: The state value s of the robot end effector at time t t The pose of the robot end effector and the force twist W, namely Among them, the posture of the robot end effector is a set of 6-dimensional vectors, which represents the coordinates of the current position of the robot end effector in the coordinate system constructed by the target position. The force torque W is composed of the three-dimensional force f and the three-dimensional torque τ on the robot end effector; the action value a of the robot end effector at time t is t Set to the change in the actuator's target pose, i.e. a t =(Δx,Δy,Δz,Δrx,Δry,Δrz).

4. The robot quick jacking method according to claim 3, characterized in that: The reward function r t The design is as follows:

5. The robot quick jacking method according to claim 1, characterized in that: The robot is switched to a trained controller, and the trained controller is used to control the end effector to process the terminal jack task, including the steps of: Switch the robot to the trained controller and generate the action a through the trained controller t is the change in the target position of the robot actuator; When the trained controller outputs action a at a frequency of 25 Hz t When the value is, let the current trajectory point be x t , then the target motion trajectory point at the next moment will be updated to x t+1 =x t +a t , the underlying controller runs at 1kHz, so when x t and a t During the update time interval, the bottom-level controller will perform 1000 / 25=40 calculations to obtain the robot joint accelerations and perform inverse kinematics calculations to obtain the joint torques τ, controlling the robot's motion from x t Movement to x t+1 ,in, τ is the joint space force or torque; x t is the desired position; T is the kinematic equation, is the transformation matrix from the base coordinate system to the end coordinate system; q is the joint position vector; J is the Jacobian matrix; K pp is the proportional coefficient of the position channel; K pi is the integration coefficient of the position channel; is the expected speed; is the joint velocity vector; K pd is the proportional coefficient of the velocity channel; C p is the selection matrix for position and velocity channels.

6. A robot quick jack device, characterized in that: include: A simulation training module is used to perform terminal jack training on the robot's high-level controller using a distributed maximum entropy reinforcement learning algorithm in a simulation environment to obtain a trained controller; The mobile module is used to take pictures of the target hole position through the environment camera in the real environment, and use the open-loop planner to move the robot end effector to the top of the target hole position; The jack module is used to switch the robot to a trained controller, and control the end effector to process the terminal jack task through the trained controller.

7. A robot quick jack device, characterized in that: comprising a memory and at least one processor, wherein the memory has computer-readable instructions stored therein; The at least one processor calls the computer-readable instructions in the memory to execute the various steps of the robot quick jacking method as described in any one of claims 1-5.

8. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the steps of the robot quick jacking method as described in any one of claims 1-5 are implemented.

Citation Information

Patent Citations

  • Robot intelligent grabbing method based on virtual training

    CN111079561A

  • Shaft hole assembling and inserting method based on gating circulation unit

    CN116408804A

  • Mechanical arm control system calibration method based on deep reinforcement learning, storage device and electronic equipment

    CN118123838A

  • Controlling a robot during interaction with a human

    US20240157557A1

Cited By

  • Robot assembly method, device and equipment based on vision

    CN120985681A