Robot constant force grinding control method and system based on improved SAC reinforcement learning

Abstract

The invention belongs to the technical field of automation control, and particularly discloses a robot constant force grinding control method and system based on improved SAC reinforcement learning, and the method comprises the steps: building a contact simulation model of a grinding robot on a MuJoCo platform; markov decision process modeling is carried out on the constant force grinding control process of the grinding robot, a state space, an action space and a reward function are set, and a robot constant force grinding control model is established; an impedance controller based on the position is constructed, and the grinding robot is controlled to conduct constant-force grinding based on the impedance controller; the impedance controller comprises unknown impedance parameters, and the impedance parameters comprise a proportionality coefficient and an integral coefficient; and impedance parameters in the impedance controller are solved based on improved SAC reinforcement learning, the optimal control strategy of the grinding robot is determined, and constant-force grinding control over the grinding robot is achieved based on the optimal control strategy. The method has better performance for the scene of robot constant force grinding control in a continuous state and an action space.

Description

Technical Field

[0001] The present application belongs to the field of automation control technology, and more specifically, to a robot constant force grinding control method and system based on improved SAC reinforcement learning. Background Art

[0002] The grinding force fluctuation of the grinding robot will affect the surface quality of the workpiece. It is necessary to improve the intelligence of the system through control and intelligent algorithms to achieve constant force grinding control of the robot.

[0003] The related technology provides a robot constant force control grinding method based on proximal policy optimization (PPO) reinforcement learning, which uses PPO reinforcement learning to solve the unknown parameters in the impedance controller, so that the impedance controller can be used to perform constant force grinding on the robot.

[0004] However, the performance of the PPO network is highly dependent on hyperparameters, especially the setting of trimming parameters. When faced with high-dimensional state space and continuous action space tasks, the algorithm performance is average. In addition, the PPO network requires a large amount of sample data to train the model, and the sample requirements are even more significant when faced with high-dimensional state space and continuous action space tasks. Summary of the invention

[0005] In view of the above-mentioned defects in the prior art, the present application provides a robot constant force grinding control method and system based on improved SAC reinforcement learning.

[0006] In a first aspect, the present application provides a robot constant force grinding control method based on improved SAC reinforcement learning, comprising: Establish a contact simulation model of the polishing robot on the MuJoCo platform; The Markov decision process is used to model the control process of the constant force grinding of the grinding robot, and the state space, action space and reward function are set to establish the robot constant force grinding control model; Construct a position-based impedance controller, and control the grinding robot to perform constant force grinding based on the impedance controller; the impedance controller includes unknown impedance parameters, and the impedance parameters include proportional coefficients and integral coefficients; The impedance parameters in the impedance controller are solved based on the improved SAC reinforcement learning, the optimal control strategy of the grinding robot is determined, and the constant force grinding control of the grinding robot is realized based on the optimal control strategy.

[0007] In a second aspect, the present application also provides a robot constant force grinding control system based on improved SAC reinforcement learning, comprising: The simulation module is used to build a contact simulation model of the polishing robot on the MuJoCo platform; The first modeling module is used to perform Markov decision process modeling on the control process of the constant force grinding of the grinding robot, set the state space, action space and reward function, and establish a robot constant force grinding control model; The second modeling module is used to construct a position-based impedance controller, and control the grinding robot to perform constant force grinding based on the impedance controller; the impedance controller includes unknown impedance parameters, and the impedance parameters include a proportional coefficient and an integral coefficient; The strategy determination module is used to solve the impedance parameters in the impedance controller based on the improved SAC reinforcement learning, determine the optimal control strategy of the grinding robot, and realize the constant force grinding control of the grinding robot based on the optimal control strategy.

[0008] In a third aspect, the present application also provides an electronic device, comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0009] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0010] In a fifth aspect, the present application further provides a computer program product, which, when executed on a processor, enables the processor to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0011] The robot constant force grinding control method based on improved SAC reinforcement learning provided in the present application solves the impedance parameters of the impedance controller used for robot constant force grinding control by improving SAC reinforcement learning. Compared with the deterministic strategy exploration algorithm, the SAC algorithm can retain valuable trajectories without discarding valuable actions, encourage the intelligent agent to explore all possible optimal trajectories, and the learned strategies have more possibilities; it has stronger new strategy exploration capabilities to prevent the model from converging to the local optimum prematurely; it has stronger stability and stronger adaptability, and can adapt and adjust faster in the face of uncertainty and disturbances in the environment, and has better performance for the scenario of robot constant force grinding control in continuous state and action space. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the present application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 It is one of the flow charts of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application; Figure 2 This is the second flow chart of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application; Figure 3 This is the third flow chart of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application; Figure 4 It is a structural schematic diagram of a robot constant force grinding control system based on improved SAC reinforcement learning provided in an embodiment of the present application; Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0015] Figure 1 This is one of the flow charts of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application, such as Figure 1 As shown, the method comprises at least the following steps: S101. Establish a contact simulation model of the polishing robot on the MuJoCo platform.

[0016] Specifically, Multi-Joint dynamics and Contact (MuJoCo) is a physics engine used for interaction between robots, machine learning, and simulated environments, with powerful contact dynamics modeling and simulation capabilities.

[0017] On the MuJoCo platform, you can build a contact simulation model of the polishing robot, add force sensors to it, set the robot's gravity, damping, stiffness, friction, etc., establish the relative coordinate system of each connected part according to the correct assembly relationship, and add its physical properties.

[0018] S102, modeling the control process of the constant force grinding of the grinding robot by the Markov decision process, setting the state space, action space and reward function, and establishing a robot constant force grinding control model.

[0019] Specifically, the basic framework of robot grinding constant force control is Markov decision making. The intelligent agent (robot terminal control unit) uses the current state Select Action Enter the next state , the goal is to maximize the cumulative reward. The Markov decision process modeling of the robot constant force grinding process is carried out, and the robot constant force grinding control model is established as follows:

[0020] in, To describe the state space of the environment, To describe the action space of the agent’s behavior, is the state transition probability, is the reward function, is the discount factor, Reinforcement learning given random dynamics And the reward function , the optimization goal is to find a strategy Make Maximize expectations.

[0021] In the modeling process of robot constant force grinding control, the state space, action space and reward function are set. Specifically, the state space is set as:

[0022] Set the action space to:

[0023] Set the reward function to:

[0024] in, , , and are constants, is the robot terminal velocity. The goal of reinforcement learning training is to minimize the robot terminal contact force error and update the velocity change of the robot terminal normal phase. Therefore, the error between the actual contact force and the expected contact force of the robot is used in the reward function. To reward and punish, encourage the agent to learn parameters that are more inclined to make the contact force error 0; use the robot terminal speed To reward or punish the amplitude of terminal fluctuations.

[0025] S103, constructing a position-based impedance controller, and controlling the grinding robot to perform constant-force grinding based on the impedance controller; the impedance controller includes unknown impedance parameters, and the impedance parameters include a proportional coefficient and an integral coefficient.

[0026] Specifically, the position-based impedance controller is an effective method for receiving contact force signals to track a constant desired force, especially for robots whose dynamic parameters are difficult to identify and open to a large extent.

[0027] For the impedance control of the robot end, it is difficult to ensure high contact force control accuracy due to the lack of force closed-loop feedback. In order to better stabilize the tracking force, a force closed-loop impedance controller based on proportional-integral (PI) is adopted. The impedance equation is:

[0028] in, is the inertia coefficient, is the damping coefficient, is the stiffness coefficient, The actual position of the robot end With expected position The position error, and They are The first and second derivatives of The actual contact force Expected contact force The force error, is the proportionality coefficient, is the integration coefficient.

[0029] In the PI-based impedance controller, the impedance equation includes , force closed-loop feedback is added, and PI control is used to ensure higher contact force control accuracy.

[0030] S104. Solve the impedance parameters in the impedance controller based on the improved SAC reinforcement learning, determine the optimal control strategy of the grinding robot, and implement constant force grinding control of the grinding robot based on the optimal control strategy.

[0031] Specifically, the impedance parameters of the impedance controller (proportional coefficient and the integral coefficient ) is crucial to the stability of the controller and needs to be adjusted dynamically based on the force tracking error. Using reinforcement learning to self-learn the impedance parameters of the impedance controller can autonomously discover the optimal behavior.

[0032] As a proximal policy optimization solution, the PPO reinforcement learning algorithm is suitable for robot constant force grinding control scenarios with continuous states and action spaces. However, the PPO reinforcement learning algorithm needs to resample enough sample data under the current policy every time the policy is updated, and the sample requirement is strong; at the same time, the PPO reinforcement learning algorithm tends to converge to a deterministic policy, and it is difficult to find the global optimal solution when there are multiple optimal policies or when there is noise or uncertainty in the environment; and the performance of the PPO reinforcement learning algorithm depends on the setting of hyperparameters, especially the trimming parameters. Inappropriate hyperparameter settings will lead to algorithm performance degradation or training failure.

[0033] The SAC reinforcement learning algorithm is a reinforcement learning algorithm without obvious random strategies. It combines the Actor-Critic structure, Soft Q-learning and entropy regularization technology, and is suitable for robot constant force grinding control scenarios with continuous state and action space. The SAC reinforcement learning algorithm combines Actor-Critic with different strategy updates, and has better performance on a series of continuous action benchmark tasks. Its optimal strategy equation is as follows:

[0034] in, is the temperature coefficient, which affects the weight of entropy, It is the calculation of information entropy. Compared with the deterministic strategy (PPO reinforcement learning), the maximum entropy retains valuable trajectories without discarding valuable actions, encourages the agent to explore all possible optimal trajectories, and the learned strategy has more possibilities; it has a stronger ability to explore new strategies, preventing the model from converging to the local optimum in advance; it has stronger stability and stronger adaptability, and can adapt and adjust faster in the face of uncertainty and disturbance in the environment.

[0035] Therefore, the embodiment of the present application attempts to use the Soft Actor-Critic (SAC) reinforcement learning algorithm to solve the optimization solution of the impedance parameters of the PI-based impedance controller in the robot constant force grinding control scenario, and improves the SAC algorithm.

[0036] The following is an introduction to the impedance control process based on the improved SAC reinforcement learning algorithm provided in the embodiment of the present application.

[0037] Figure 2 This is the second flow chart of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application. Figure 3 This is the third flow chart of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application, such as Figure 2 and Figure 3As shown in the figure, the impedance control process based on the improved SAC reinforcement learning algorithm is as follows: (1) Initialization phase ① Nonlinear tracking differentiator: Design a nonlinear tracking differentiator to perform force planning during the impact phase of the grinding robot.

[0038] ② Environmental stiffness learning: The initial estimation of environmental stiffness is performed through RELS, and the initial parameters in the impedance controller are set accordingly. The initial parameters include inertia coefficient, stiffness coefficient and damping coefficient.

[0039] ③ Reinforcement learning initialization: set the maximum number of learning sequences episodeMAX for the interaction between the polishing robot agent and the environment, and the maximum number of control steps stepMAX in each learning sequence; set the initial motion trajectory and expected contact force of the polishing robot agent; initialize the actor network, critic network and experience replay pool of the improved SAC reinforcement learning.

[0040] (2) Learning process ① Initialization state: set the initial state , including the polishing robot's position, speed, and environment stiffness estimation.

[0041] ②Control step loop For each control step in the current learning sequence: a. Environment estimation: The environment parameters are estimated through RELS, and the environment parameters include environment stiffness and environment position.

[0042] b. Trajectory adjustment: Based on the estimated environmental parameters, the preset expected contact force and the initial motion trajectory, the reference motion trajectory of the grinding robot is adjusted.

[0043] c. Robot motion and force error calculation: Control the grinding robot to move along the reference motion trajectory, obtain the actual contact force at the end of the robot, and calculate the force error between the actual contact force and the expected contact force.

[0044] d. Action selection: Based on the actor network, the terminal compensation amount is selected as the action of reinforcement learning at the current moment .

[0045] e. Impedance parameter adjustment of the impedance controller: The adjustment amount of the impedance parameter is calculated based on the force error and applied to the impedance controller.

[0046] f. Displacement command calculation: The joint displacement command of the robot is determined based on the adjusted impedance parameters, and the robot executes the joint displacement command to move.

[0047] g. Reward calculation and state transfer: Calculate the reward value of reinforcement learning at the current moment based on the reward function , determine the new state based on the critic network , collect data tuples And stored in the experience replay pool.

[0048] h. Network update: Every n control steps, data is sampled from the experience replay pool to update the actor network and the critic network.

[0049] ③Evaluation When the current learning sequence is completed, the average reward and performance of SAC are evaluated to monitor the learning progress.

[0050] (3) Algorithm termination When the maximum number of learning sequences is reached or SAC converges, the trained actor network is output for impedance control.

[0051] In SAC reinforcement learning, the actor network is responsible for generating the action probability distribution and selecting actions based on the distribution. The critic network is responsible for estimating the state or action state value to guide the update of the actor network. The convergence and stability of the algorithm need to be verified, and the performance can be optimized by adjusting hyperparameters such as network structure, learning rate, and experience replay pool size.

[0052] Furthermore, in order to improve the convergence speed of the SAC reinforcement learning algorithm, it is considered to improve the performance of the SAC algorithm.

[0053] On the one hand, the design of the reward function is crucial to the performance of the algorithm, and it should be ensured that it can reflect the desired control objectives and constraints. ,Right now:

[0054]

[0055] in, is the proportionality coefficient.

[0056] Specifically, assuming that the policy optimization problem already has an optimal policy , then the logarithm of the optimal strategy When selecting actions, the values ​​of the optimal action and the suboptimal action satisfy:

[0057] This can be used to improve the ability to distinguish good and bad actions during strategy evaluation. When performing strategy evaluation, use samples , in the reward function Add the action entropy selected by the current strategy to the , which can guide the algorithm to suppress the selection of suboptimal solutions, and adding this action entropy will not change the optimal strategy.

[0058] On the other hand, the experience replay pool is used to store and reuse past data to improve sample efficiency and reduce variance during training. Based on the SAC reinforcement learning algorithm, the embodiment of the present application adds a priority experience replay method.

[0059] The agent interacts with the environment to obtain a sequence It is stored in the experience replay pool, and the policy network and value network regularly obtain data samples from the experience replay pool to update the network training. When the reinforcement learning policy model obtains samples from the experience replay pool for training, random uniform sampling is generally used.

[0060] This experience replay method can use all historical data for learning, and each sample has the same probability of being selected, and learns without difference in probability. This ignores the importance of each experience, and the benefits obtained from different samples are not the same. In order to improve the efficiency of reinforcement learning, high-yield samples should be learned with a greater probability.

[0061] In the embodiment of the present application, when the sample is stored in the experience playback pool, the value or priority of the sample is set to increase the probability of high-yield samples being extracted for learning. At the same time, in terms of time, the most recent experience is more important, so a higher priority is set for the most recent experience. In general, in order to improve the learning efficiency of reinforcement learning samples, the probability of adopting high-yield samples can be increased by calculating the priority of training samples.

[0062] In order to improve sampling efficiency and algorithm accuracy, random priority sampling and bias are used to obtain samples from the experience replay pool for training. Specifically, in order to measure the importance of experience, the priority of experience replay is calculated based on the size of the temporal difference error (TD-Error).

[0063] Specifically, the sample Action interval (also known as TD-Error) is redefined as the absolute value of the errors of the two Q networks in SAC, that is:

[0064] in, is the target value of the Q network, Represents the current estimate of the i-th network. The experience replay priority of the sampled data in the experience replay pool Defined as .

[0065] TD-Error indicates the deviation between the current model estimate and the target value. The learning benefit is positively correlated with the size of TD-Error. The sampling probability is proportional to the size of TD-Error, satisfying:

[0066] Among them, the parameters Used to eliminate the influence of TD-Error extreme values, is a very small positive number, which is used to prevent the abnormal situation that TD-Error is 0. The above formula does not satisfy the probability distribution of random sampling and needs to be corrected by importance sampling. The correction formula is as follows:

[0067]

[0068] in, is the adjustment coefficient for the importance sampling weight, is the number of samples in the experience replay pool, is the loss function, The weight of the prioritized samples during model update can greatly reduce the impact of high-priority samples on the training process.

[0069] Furthermore, when considering reinforcement learning to update the network, uniform sampling is performed from a fixed experience replay pool. The success rate of reinforcement learning attempts is higher the later the more effective the recent experience is. The embodiment of the present application provides an adaptive experience replay sampling strategy. In the early stage of parameter update, all samples are sampled from the experience replay pool using a random strategy. The sampling space is slowly reduced for each subsequent small batch sampling, and closer data points are sampled more actively.

[0070] Specifically, suppose that k small batches need to be updated in the current update phase. At the beginning, the model is allowed to fully learn, and then the sampling of old data is slowly reduced. Considering the S-shaped shape of the Sigmoid function, the shape of the sampling function is modified. When sampling from the experience replay pool to update the algorithm, the sampling range adjustment factor is set to adjust the sampling range. Assuming that there are samples, the sampling range adjustment factor of the kth mini-batch satisfy:

[0071] in, is the learning rate of the model, is the generation time of the sampling data, The time when the earliest sample data was generated.

[0072] Furthermore, the number of valid samples considered when updating the kth mini-batch is calculated based on the sampling range adjustment factor ,satisfy:

[0073] Based on the determined number of valid samples, the latest samples are randomly selected from the experience replay pool to update the actor network and the critic network. If the determined number of valid samples is greater than the actual number of remaining samples, all remaining samples are used. For each mini-batch update, the above process is repeated until all mini-batches are updated.

[0074] The benefit of this sampling method is that the first small batch will be evenly sampled from the entire experience replay pool, which helps to gain a wide range of experience. As training progresses, it gradually tends to use the latest experience to update parameters, which helps to adapt to environmental changes more quickly; by gradually discarding the sampling of old data, the model can more efficiently use newer and more valuable data to update parameters, accelerating the learning process; referring to the sampling range adjustment factor set by the Sigmiod function makes the adjustment process of the sampling range smooth and controllable, avoiding the instability that may be caused by sudden changes in the sampling strategy; while continuously reducing the sampling range and maintaining random sampling, it can effectively avoid falling into the local optimal solution.

[0075] The robot constant force grinding control system based on improved SAC reinforcement learning provided in the present application is described below. The robot constant force grinding control system based on improved SAC reinforcement learning described below and the robot constant force grinding control method based on improved SAC reinforcement learning described above can be referenced to each other.

[0076] Figure 4 is a structural diagram of a robot constant force grinding control system based on improved SAC reinforcement learning provided in an embodiment of the present application, such as Figure 4 As shown, the device at least includes: A simulation module 401 is used to establish a contact simulation model of the polishing robot on the MuJoCo platform; The first modeling module 402 is used to perform Markov decision process modeling on the control process of the constant force grinding of the grinding robot, set the state space, action space and reward function, and establish a robot constant force grinding control model; The second modeling module 403 is used to construct a position-based impedance controller, and control the grinding robot to perform constant force grinding based on the impedance controller; the impedance controller includes unknown impedance parameters, and the impedance parameters include a proportional coefficient and an integral coefficient; The strategy determination module 404 is used to solve the impedance parameters in the impedance controller based on the improved SAC reinforcement learning, determine the optimal control strategy of the grinding robot, and realize the constant force grinding control of the grinding robot based on the optimal control strategy.

[0077] Optionally, the network structure of the improved SAC reinforcement learning includes an actor network and a critic network, and the strategy determination module 404 is specifically used for: For each control step in the current learning sequence, the environment parameters are estimated by RELS, and the environment parameters include environment stiffness and environment position; Based on the estimated environmental parameters, the preset expected contact force and the initial motion trajectory, the reference motion trajectory of the grinding robot is adjusted; Control the grinding robot to move along the reference motion trajectory, obtain the actual contact force at the end of the robot, and calculate the force error between the actual contact force and the expected contact force; Based on the actor network, select the terminal compensation as the action of reinforcement learning at the current moment ; calculating an adjustment amount of an impedance parameter based on the force error and applying the adjustment amount to an impedance controller; determining joint displacement instructions of the robot based on the adjusted impedance parameters; Calculate the current reward value of reinforcement learning based on the reward function , determine the new state based on the critic network , collect data tuples And store it in the experience replay pool; Every n control steps, data is sampled from the experience replay pool to update the actor network and the critic network; When the current learning sequence is completed, the average reward and performance of SAC are evaluated to monitor the learning progress; When the maximum number of learning sequences is reached or SAC converges, the trained actor network is output for impedance control.

[0078] Optionally, sample data from the experience replay pool to update the actor network and critic network, including: The actor network and the critic network are updated by sampling data from the experience replay pool through random priority sampling.

[0079] Optionally, update the actor network and critic network by sampling data from the experience replay pool through random priority sampling, including: Based on the generation time of the sampled data in the experience replay pool and the size of the TD-Error, the experience replay priority of the sampled data in the experience replay pool is determined.

[0080] Optionally, update the actor network and critic network by sampling data from the experience replay pool through random priority sampling, including: Determine the sampling range adjustment factor for the kth mini-batch in the experience replay pool; Determine the number of valid samples considered when updating the kth mini-batch based on the sampling range adjustment factor; Based on the determined number of valid samples, the latest samples are randomly selected from the experience replay pool to update the actor network and the critic network.

[0081] Optionally, determine the sampling range adjustment factor for the kth mini-batch in the experience replay pool, satisfying:

[0082] in, is the sampling range adjustment factor of the kth mini-batch in the experience replay pool, is the total number of samples in the experience replay pool, is the learning rate, is the generation time of the sampling data, is the time when the earliest sampling data was generated, The minimum value of the sampling range adjustment factor.

[0083] Optionally, the policy determination module 404 is specifically configured to: A nonlinear tracking differentiator is used for force planning during the impact phase of the grinding robot. The initial estimation of the environmental stiffness is performed through RELS, and the initial parameters in the impedance controller are set. The initial parameters include the inertia coefficient, stiffness coefficient and damping coefficient. Set the maximum number of learning sequences for the polishing robot agent to interact with the environment, the maximum number of control steps in each learning sequence, set the initial motion trajectory and expected contact force of the polishing robot agent, and initialize the actor network, critic network, and experience replay pool of the improved SAC reinforcement learning.

[0084] Optionally, the impedance controller satisfies:

[0085] in, is the inertia coefficient, is the damping coefficient, is the stiffness coefficient, The actual position of the robot end With expected position The position error, and They are The first and second derivatives of The actual contact force Expected contact force The force error, is the proportionality coefficient, is the integration coefficient.

[0086] Optionally, the reward function satisfies:

[0087] in, is the reward function, , , and are constants, The actual contact force Expected contact force The error, is the robot terminal speed; The entropy of the action selected by the current policy, is the temperature coefficient, is the proportionality coefficient, To reinforce the action space at the current moment, To reinforce the state space of the current moment, The current strategy for reinforcement learning.

[0088] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0089] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0090] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device. The device may include: at least one memory for storing programs and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the above embodiment.

[0091] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, the electronic device may include: a processor (Processor) 501, a communication interface (Communications Interface) 502, a memory (Memory) 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504. The processor 501 can call the software instructions in the memory 503 to execute the method described in the above embodiment.

[0092] In addition, the logic instructions in the above-mentioned memory 503 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application.

[0093] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0094] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0095] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0096] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0097] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0098] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0099] It is easy for those skilled in the art to understand that the above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A robot constant force grinding control method based on improved SAC reinforcement learning, characterized in that: include: Establish a contact simulation model of the polishing robot on the MuJoCo platform; The Markov decision process is used to model the control process of the constant force grinding of the grinding robot, and the state space, action space and reward function are set to establish the robot constant force grinding control model; Constructing a position-based impedance controller, and controlling the grinding robot to perform constant force grinding based on the impedance controller; the impedance controller includes unknown impedance parameters, and the impedance parameters include a proportional coefficient and an integral coefficient; The impedance parameters in the impedance controller are solved based on improved SAC reinforcement learning, the optimal control strategy of the grinding robot is determined, and the constant force grinding control of the grinding robot is realized based on the optimal control strategy.

2. The robot constant force grinding control method according to claim 1, characterized in that: The network structure of the improved SAC reinforcement learning includes an actor network and a critic network. The method of solving the impedance parameters in the impedance controller based on the improved SAC reinforcement learning includes: For each control step in the current learning sequence, the environment parameters are estimated by RELS, where the environment parameters include environment stiffness and environment position; Based on the estimated environmental parameters, the preset expected contact force and the initial motion trajectory, the reference motion trajectory of the grinding robot is adjusted; Controlling the polishing robot to move along the reference motion trajectory, obtaining the actual contact force of the robot end, and calculating the force error between the actual contact force and the expected contact force; Based on the actor network, select the terminal compensation as the action of reinforcement learning at the current moment ; Calculating an adjustment amount of the impedance parameter based on the force error and applying the adjustment amount to the impedance controller; determining joint displacement instructions of the robot based on the adjusted impedance parameters; Calculate the reward value of reinforcement learning at the current moment based on the reward function , determine the new state based on the critic network , collect data tuples And store it in the experience replay pool; Every n control steps, data is sampled from the experience replay pool to update the actor network and the critic network; When the current learning sequence is completed, the average reward and performance of SAC are evaluated to monitor the learning progress; When the maximum number of learning sequences is reached or SAC converges, the trained actor network is output for impedance control.

3. The robot constant force grinding control method according to claim 2, characterized in that: The updating of the actor network and the critic network by sampling data from the experience replay pool includes: The actor network and the critic network are updated by sampling data from the experience replay pool through random priority sampling.

4. The robot constant force grinding control method according to claim 3, characterized in that: The updating of the actor network and the critic network by sampling data from the experience replay pool through random priority sampling includes: Based on the generation time of the sampled data in the experience replay pool and the size of the TD-Error, the experience replay priority of the sampled data in the experience replay pool is determined.

5. The robot constant force grinding control method according to claim 3, characterized in that: The updating of the actor network and the critic network by sampling data from the experience replay pool through random priority sampling includes: Determining a sampling range adjustment factor for the kth mini-batch in the experience replay pool; Determine the number of valid samples considered when updating the k-th mini-batch based on the sampling range adjustment factor; Based on the determined number of valid samples, the latest samples are randomly selected from the experience replay pool to update the actor network and the critic network.

6. The robot constant force grinding control method according to claim 5, characterized in that: The determination of the sampling range adjustment factor of the kth small batch in the experience replay pool satisfies: in, is the sampling range adjustment factor of the kth mini-batch in the experience replay pool, is the total number of samples in the experience replay pool, is the learning rate, is the generation time of the sampling data, is the time when the earliest sampling data was generated, The minimum value of the sampling range adjustment factor.

7. The robot constant force grinding control method according to claim 2, characterized in that: The method of solving the impedance parameters in the impedance controller based on improved SAC reinforcement learning also includes: A nonlinear tracking differentiator is used for force planning during the impact phase of the grinding robot. Performing an initial estimation of the environmental stiffness through RELS, setting initial parameters in the impedance controller, wherein the initial parameters include an inertia coefficient, a stiffness coefficient, and a damping coefficient; Set the maximum number of learning sequences for the polishing robot agent to interact with the environment, the maximum number of control steps in each learning sequence, set the initial motion trajectory and expected contact force of the polishing robot agent, and initialize the actor network, critic network, and experience replay pool of the improved SAC reinforcement learning.

8. The robot constant force grinding control method according to claim 1, characterized in that: The impedance controller satisfies: in, is the inertia coefficient, is the damping coefficient, is the stiffness coefficient, The actual position of the robot end With expected position The position error, and They are The first and second derivatives of The actual contact force Expected contact force The force error, is the proportionality coefficient, is the integration coefficient.

9. The robot constant force grinding control method according to claim 1, characterized in that: The reward function satisfies: in, is the reward function, , , and are constants, The actual contact force Expected contact force The error, is the robot terminal speed; The entropy of the action selected by the current policy, is the temperature coefficient, is the proportionality coefficient, To strengthen the learning of the action space at the current moment, To reinforce the state space of the current moment, The current strategy for reinforcement learning.

10. A robot constant force grinding control system based on improved SAC reinforcement learning, characterized in that: include: The simulation module is used to build a contact simulation model of the polishing robot on the MuJoCo platform; The first modeling module is used to perform Markov decision process modeling on the control process of the constant force grinding of the grinding robot, set the state space, action space and reward function, and establish a robot constant force grinding control model; A second modeling module is used to construct a position-based impedance controller, and control the polishing robot to perform constant force polishing based on the impedance controller; the impedance controller includes unknown impedance parameters, and the impedance parameters include a proportional coefficient and an integral coefficient; A strategy determination module is used to solve the impedance parameters in the impedance controller based on improved SAC reinforcement learning, determine the optimal control strategy of the grinding robot, and realize constant force grinding control of the grinding robot based on the optimal control strategy.

Images