Robot constant force polishing control method and system based on improved SAC reinforcement learning

Abstract

The application belongs to the technical field of automation control, and specifically discloses a robot constant force polishing control method and system based on improved SAC reinforcement learning, wherein the method comprises the following steps: establishing a contact simulation model of a polishing robot on a MuJoCo platform; modeling a Markov decision process for the control process of constant force polishing of the polishing robot, setting a state space, an action space and a reward function, and establishing a robot constant force polishing control model; constructing a position-based impedance controller, and controlling the polishing robot to perform constant force polishing based on the impedance controller; the impedance controller comprises unknown impedance parameters, and the impedance parameters comprise a proportional coefficient and an integral coefficient; the impedance parameters in the impedance controller are solved based on improved SAC reinforcement learning, the optimal control strategy of the polishing robot is determined, and constant force polishing control of the polishing robot is realized based on the optimal control strategy. The application has better performance in the scene of robot constant force polishing control of continuous state and action space.

Description

Technical Field

[0001] The present application belongs to the field of automation control technology, and more specifically, relates to a robot constant force grinding control method and system based on improved SAC reinforcement learning. Background Art

[0002] The grinding force fluctuation of the grinding robot will affect the surface quality of the workpiece. It is necessary to improve the intelligence of the system through control and intelligent algorithms to achieve constant force grinding control of the robot.

[0003] The related technology provides a robot constant force control grinding method based on Proximal Policy Optimization (PPO) reinforcement learning, which uses PPO reinforcement learning to solve the unknown parameters in the impedance controller, thereby using the impedance controller to perform constant force grinding on the robot.

[0004] However, the performance of the PPO network is highly dependent on hyperparameters, especially the setting of the trimming parameters. When faced with high-dimensional state space and continuous action space tasks, the algorithm performance is average. In addition, the PPO network requires a large amount of sample data to train the model, and the sample requirements are even more significant when faced with high-dimensional state space and continuous action space tasks. Summary of the Invention

[0005] In response to the above-mentioned defects in the prior art, the present application provides a robot constant force grinding control method and system based on improved SAC reinforcement learning.

[0006] In a first aspect, the present application provides a robot constant force grinding control method based on improved SAC reinforcement learning, comprising:

[0007] Establish a contact simulation model of the polishing robot on the MuJoCo platform;

[0008] The Markov decision process is used to model the control process of the constant force grinding of the grinding robot. The state space, action space and reward function are set to establish the robot constant force grinding control model.

[0009] Construct a position-based impedance controller and control the grinding robot to perform constant-force grinding based on the impedance controller; the impedance controller includes unknown impedance parameters, including proportional coefficient and integral coefficient;

[0010] Based on the improved SAC reinforcement learning, the impedance parameters in the impedance controller are solved, the optimal control strategy of the polishing robot is determined, and the constant force polishing control of the polishing robot is realized based on the optimal control strategy.

[0011] In a second aspect, the present application also provides a robot constant force grinding control system based on improved SAC reinforcement learning, comprising:

[0012] simulate a contact simulation model of the polishing robot on a MuJoCo platform;

[0013] The first modeling module is configured to model a Markov decision process of a control process of constant force polishing of the polishing robot, set a state space, an action space and a reward function, and establish a robot constant force polishing control model.

[0014] The second modeling module is configured to construct a position-based impedance controller, and control the polishing robot to perform constant force polishing based on the impedance controller.

[0015] The strategy determination module is configured to determine an optimal control strategy of the polishing robot by solving the impedance parameters in the impedance controller based on the improved SAC reinforcement learning, and implement constant force polishing control of the polishing robot based on the optimal control strategy.

[0016] In a third aspect, the present application also provides an electronic device, comprising: at least one memory configured to store a program; and at least one processor configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0017] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and when the computer program is run on a processor, the processor is caused to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0018] In a fifth aspect, the present application also provides a computer program product, and when the computer program product is run on a processor, the processor is caused to execute the method described in the first aspect or any possible implementation manner of the first aspect.

[0019] The robot constant force polishing control method based on the improved SAC reinforcement learning provided in the present application solves the impedance parameters of the impedance controller for robot constant force polishing control through the improved SAC reinforcement learning. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in this application or related technologies, the following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is one of the flow charts of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application;

[0022] Figure 2 This is the second flow chart of the robot constant force grinding control method based on improved SAC reinforcement learning provided in an embodiment of the present application;

[0023] Figure 3 This is the third flow chart of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application;

[0024] Figure 4 Schematic diagram of the structure of a robot constant force grinding control system based on improved SAC reinforcement learning provided in an embodiment of the present application;

[0025] Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0027] Figure 1 This is one of the flow charts of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application, such as Figure 1 As shown, the method comprises at least the following steps:

[0028] S101. Establish a contact simulation model of the polishing robot on the MuJoCo platform.

[0029] Specifically, Multi-Joint dynamics and Contact (MuJoCo) is a physics engine used for interaction between robots, machine learning, and simulated environments, with powerful contact dynamics modeling and simulation capabilities.

[0030] On the MuJoCo platform, you can build a contact simulation model of a polishing robot, add force sensors to it, set the robot's gravity, damping, stiffness, friction, etc., establish the relative coordinate system of each connected part according to the correct assembly relationship, and add its physical properties.

[0031] S102. Model the control process of the constant-force grinding of the grinding robot using a Markov decision process, set the state space, action space, and reward function, and establish a robot constant-force grinding control model.

[0032] Specifically, the basic framework of robot grinding constant force control is Markov decision making, where the agent (robot terminal control unit) uses the current state Select Action Enter the next state , the goal is to maximize the cumulative reward. The Markov decision process modeling of the robot constant force polishing process is carried out, and the robot constant force polishing control model is established as follows:

[0033]

[0034] in, To describe the state space of the environment, To describe the action space of the agent’s behavior, is the state transition probability, is the reward function, is the discount factor, Reinforcement learning given random dynamics for time. and the reward function , the optimization goal is to find a strategy Make Maximize expectations.

[0035] In the modeling process of the robot constant force grinding control, the state space, action space and reward function are set. Specifically, the state space is set as:

[0036]

[0037] Set the action space to:

[0038]

[0039] Set the reward function to:

[0040]

[0041] in, 、 、 and are all constants, is the robot terminal velocity. The goal of reinforcement learning training is to minimize the error of the robot terminal contact force and update the velocity change of the robot terminal normal phase. Therefore, the error between the actual contact force and the expected contact force of the robot is used in the reward function. To reward and punish, encourage the agent to learn parameters that are more inclined to make the contact force error 0; use the robot terminal speed To reward or punish the amplitude of terminal fluctuations.

[0042] S103, constructing a position-based impedance controller, and controlling the polishing robot to perform constant-force polishing based on the impedance controller; the impedance controller includes unknown impedance parameters, and the impedance parameters include a proportional coefficient and an integral coefficient.

[0043] Specifically, the position-based impedance controller is an effective method for receiving contact force signals to track a constant desired force, especially for robots whose dynamic parameters are difficult to identify and have low openness.

[0044] For the impedance control of the robot end, it is difficult to ensure high contact force control accuracy due to the lack of force closed-loop feedback. In order to better stabilize the tracking force, a force closed-loop impedance controller based on proportional-integral (PI) is adopted. The impedance equation is:

[0045]

[0046] in, is the inertia coefficient, is the damping coefficient, is the stiffness coefficient, The actual position of the robot end With the expected position Position error, and They are The first and second derivatives of The actual contact force Expected contact force The force error, is the proportionality coefficient, is the integration coefficient.

[0047] In the PI-based impedance controller, the impedance equation includes , force closed-loop feedback is added, and PI control is used to ensure higher contact force control accuracy.

[0048] S104. Solve the impedance parameters in the impedance controller based on the improved SAC reinforcement learning, determine the optimal control strategy of the polishing robot, and implement constant force polishing control of the polishing robot based on the optimal control strategy.

[0049] Specifically, the impedance parameters of the impedance controller (proportional coefficient and the integral coefficient The choice of ) is crucial for controller stability and requires dynamic adjustment based on the force tracking error. Using reinforcement learning to self-learn the impedance parameters of the impedance controller allows for autonomous discovery of optimal behavior.

[0050] The PPO reinforcement learning algorithm, as a proximal policy optimization solution, is suitable for robot constant-force polishing control scenarios with continuous state and action spaces. However, each time the PPO reinforcement learning algorithm updates its policy, it needs to resample sufficient sample data under the current policy, resulting in a strong sample requirement. Furthermore, the PPO reinforcement learning algorithm tends to converge to a deterministic policy, making it difficult to find a global optimal solution when faced with multiple optimal policies or when the environment is noisy or uncertain. Furthermore, the performance of the PPO reinforcement learning algorithm depends on the setting of hyperparameters, especially the pruning parameter. Inappropriate hyperparameter settings can lead to performance degradation or training failure.

[0051] The SAC reinforcement learning algorithm is a reinforcement learning algorithm without an explicit random policy. It combines an actor-critic structure, soft Q-learning, and entropy regularization techniques. It is suitable for robot constant force polishing control scenarios with continuous state and action spaces. By combining off-policy updates with actor-critic learning, the SAC reinforcement learning algorithm achieves better performance on a series of continuous action benchmark tasks. Its optimal policy equation is as follows:

[0052]

[0053] in, is the temperature coefficient, which affects the weight of entropy, This is the calculation of information entropy. Compared to deterministic strategies (PPO reinforcement learning), maximum entropy retains valuable trajectories without discarding valuable actions, encouraging the agent to explore all possible optimal trajectories. The learned strategy has more possibilities. It has stronger ability to explore new strategies, preventing the model from prematurely converging to a local optimum. It also has greater stability and adaptability, allowing it to adapt and adjust more quickly to environmental uncertainties and disturbances.

[0054] Therefore, the embodiment of the present application attempts to use the Soft Actor-Critic (SAC) reinforcement learning algorithm to solve the optimization solution of the impedance parameters of the PI-based impedance controller in the robot constant force grinding control scenario, and improves the SAC algorithm.

[0055] The following is an introduction to the impedance control process based on the improved SAC reinforcement learning algorithm provided in the embodiment of the present application.

[0056] Figure 2 This is the second flow chart of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application. Figure 3 This is the third flow chart of the robot constant force grinding control method based on improved SAC reinforcement learning provided in the embodiment of the present application, as shown in FIG. Figure 2 and Figure 3 As shown in Figure 2, the impedance control process based on the improved SAC reinforcement learning algorithm is as follows:

[0057] (1) Initialization phase

[0058] ① Nonlinear tracking differentiator: Design a nonlinear tracking differentiator to perform force planning during the impact phase of the polishing robot.

[0059] ② Environmental stiffness learning: The environmental stiffness is initially estimated through RELS, and the initial parameters of the impedance controller are set accordingly. The initial parameters include the inertia coefficient, stiffness coefficient, and damping coefficient.

[0060] ③ Reinforcement learning initialization: set the maximum number of learning sequences episodes deMAX for the interaction between the polishing robot agent and the environment, and the maximum number of control steps stepMAX in each learning sequence; set the initial motion trajectory and expected contact force of the polishing robot agent; initialize the actor network, critic network, and experience replay pool of the improved SAC reinforcement learning.

[0061] (2) Learning process

[0062] ① Initialization state: set the initial state , including the position, velocity and stiffness estimation of the polishing robot.

[0063] ②Control step loop

[0064] For each control step in the current learning sequence:

[0065] a. Environment estimation: Estimation of environment parameters is performed through RELS, which includes environment stiffness and environment position.

[0066] b. Trajectory adjustment: Based on the estimated environmental parameters, the preset expected contact force and the initial motion trajectory, the reference motion trajectory of the polishing robot is adjusted.

[0067] c. Robot motion and force error calculation: Control the grinding robot to move along the reference motion trajectory, obtain the actual contact force at the end of the robot, and calculate the force error between the actual contact force and the expected contact force.

[0068] d. Action selection: Based on the actor network, the terminal compensation amount is selected as the action at the current moment of reinforcement learning .

[0069] e、Impedance parameter adjustment of impedance controller: calculate the adjustment amount of impedance parameter based on force error and apply to impedance controller.

[0070] f、Displacement instruction calculation: determine the joint displacement instruction of the robot based on the adjusted impedance parameter, and the robot executes the joint displacement instruction to move.

[0071] g、Reward calculation and state transition: calculate the reward value of the current time of reinforcement learning based on the reward function , determine the new state based on the critic network , collect data tuples and store them in the experience replay pool.

[0072] h、Network update: update the actor network and critic network every n control steps by sampling data from the experience replay pool.

[0073] ③Evaluation

[0074] When the current learning sequence is completed, evaluate the average reward and performance of SAC to monitor the learning progress.

[0075] (3) Algorithm termination

[0076] When the maximum number of learning sequences is reached or SAC converges, output the trained actor network for impedance control.

[0077] In SAC reinforcement learning, the actor network is responsible for generating action probability distribution and selecting actions according to the distribution. The critic network is responsible for estimating state or action state value to guide the update of the actor network. The convergence and stability of the algorithm need to be verified, and the performance can be optimized by adjusting the network structure, learning rate, experience replay pool size and other hyperparameters.

[0078] Further, in order to improve the convergence speed of SAC reinforcement learning algorithm, the performance of SAC algorithm is considered to be improved.

[0079] On the one hand, the design of reward function is crucial to the performance of the algorithm, which should ensure that it can reflect the desired control target and constraints. The embodiments of the present application increase an action entropy in the calculation of reward value based on the SAC reinforcement learning algorithm, that is:

[0080]

[0081]

[0082] wherein, is a proportional coefficient.

[0083] Specifically, assuming that the policy optimization problem already has an optimal policy , then the logarithm of the optimal strategy When selecting an action, the values ​​of the optimal action and the suboptimal action satisfy:

[0084]

[0085] This can be used to improve the ability to distinguish good and bad actions during strategy evaluation. When performing strategy evaluation, use samples , in the reward function Add the action entropy selected by the current strategy to the , which can guide the algorithm to suppress the selection of suboptimal solutions, and adding this action entropy will not change the optimal strategy.

[0086] On the other hand, the experience replay pool is used to store and reuse past data to improve sample efficiency and reduce variance in the training process. Based on the SAC reinforcement learning algorithm, the embodiment of the present application adds a priority experience replay method.

[0087] The agent interacts with the environment to obtain a sequence The data is stored in the experience replay pool, and the policy network and value network periodically obtain data samples from the experience replay pool to update the network training. When the reinforcement learning policy model obtains samples from the experience replay pool for training, it generally adopts a random uniform sampling method.

[0088] This experience replay method uses all historical data for learning, giving each sample the same probability of being selected, resulting in indifferent probability learning. However, this method ignores the importance of each experience, as different samples can yield different benefits. To improve the efficiency of reinforcement learning, high-yield samples should be learned with a higher probability.

[0089] In the embodiment of the present application, when samples are stored in the experience replay pool, the value or priority of the samples is set to increase the probability of high-yield samples being extracted for learning. At the same time, from a temporal perspective, the most recent experience is more important, so a higher priority is set for the most recent experience. In general, in order to improve the learning efficiency of reinforcement learning samples, the probability of adopting high-yield samples can be increased by calculating the priority of training samples.

[0090] To improve sampling efficiency and algorithm accuracy, we use random priority sampling and biasing to obtain samples from the experience replay pool for training. Specifically, to measure the importance of experience, we prioritize experience replays based on the size of the temporal difference error (TD-Error).

[0091] Specifically, the sample Action interval (also known as TD-Error) is redefined as the absolute value of the two Q network errors in SAC, that is:

[0092]

[0093] in, is the target value of the Q network, Represents the current estimate of the i-th network. The experience replay priority of the sampled data in the experience replay pool Defined as .

[0094] TD-Error indicates the deviation between the current model estimate and the target value. The learning benefit is positively correlated with the size of TD-Error. The sampling probability is proportional to the size of TD-Error, satisfying:

[0095]

[0096] Among them, the parameters Used to eliminate the influence of TD-Error extreme values, is a very small positive number, used to prevent the abnormal situation of TD-Error being 0. The above formula does not satisfy the probability distribution of random sampling and needs to be corrected by importance sampling. The correction formula is as follows:

[0097]

[0098]

[0099] in, is the adjustment coefficient of the importance sampling weight, is the number of samples in the experience replay pool, is the loss function, The weight of the prioritized samples during model update greatly reduces the impact of high-priority samples on the training process.

[0100] Furthermore, when considering reinforcement learning to update the network, uniform sampling is performed from a fixed experience replay pool. The success rate of reinforcement learning increases with later attempts, and the more recent experience is effective. This embodiment of the application provides an adaptive experience replay sampling strategy. In the initial stage of parameter update, all samples are sampled from the experience replay pool using a random strategy. For each subsequent small batch sampling, the sampling space is gradually reduced, and closer data points are sampled more actively.

[0101] Specifically, suppose that k small batches of updates are required in the current update phase. Initially, the model is allowed to fully learn, and then the sampling of old data is slowly reduced. Considering the S-shaped shape of the Sigmoid function, the shape of the sampling function is modified. When sampling from the experience replay pool to update the algorithm, the sampling range adjustment factor is set to adjust the sampling range. Assuming that there are samples, the sampling range adjustment factor of the kth mini-batch satisfy:

[0102]

[0103] in, is the learning rate of the model, is the generation time of the sampling data, The time when the earliest sample data was generated.

[0104] Furthermore, the number of valid samples considered when updating the kth mini-batch is calculated based on the sampling range adjustment factor ,satisfy:

[0105]

[0106] Based on the determined number of valid samples, the latest sample is randomly selected from the experience replay pool to update the actor network and the critic network. If the determined number of valid samples is greater than the actual number of remaining samples, all remaining samples are used. The above process is repeated for each mini-batch update until all mini-batches are updated.

[0107] The benefit of this sampling method is that the first mini-batch is sampled evenly from the entire experience replay pool, helping to gain a wide range of experience. As training progresses, the model gradually tends to use the latest experience to update parameters, helping to adapt more quickly to environmental changes. By gradually discarding the sampling of old data, the model can more efficiently use newer and more valuable data to update parameters, accelerating the learning process. The sampling range adjustment factor set by the Sigmiod function makes the sampling range adjustment process smooth and controllable, avoiding the instability that may be caused by sudden changes in the sampling strategy. Continuously narrowing the sampling range while maintaining random sampling can effectively avoid falling into local optimal solutions.

[0108] The following describes the robot constant force grinding control system based on improved SAC reinforcement learning provided in this application. The robot constant force grinding control system based on improved SAC reinforcement learning described below and the robot constant force grinding control method based on improved SAC reinforcement learning described above can be referenced to each other.

[0109] Figure 4Schematic diagram of the structure of the robot constant force grinding control system based on improved SAC reinforcement learning provided in the embodiment of the present application. Figure 4 As shown, the device at least includes:

[0110] A simulation module 401 is used to establish a contact simulation model of the polishing robot on the MuJoCo platform;

[0111] The first modeling module 402 is used to perform Markov decision process modeling on the control process of the constant force grinding of the grinding robot, set the state space, action space and reward function, and establish a robot constant force grinding control model;

[0112] A second modeling module 403 is used to construct a position-based impedance controller, and control the polishing robot to perform constant force polishing based on the impedance controller; the impedance controller includes unknown impedance parameters, and the impedance parameters include a proportional coefficient and an integral coefficient;

[0113] The strategy determination module 404 is used to solve the impedance parameters in the impedance controller based on the improved SAC reinforcement learning, determine the optimal control strategy of the polishing robot, and implement constant force polishing control of the polishing robot based on the optimal control strategy.

[0114] Optionally, the network structure of the improved SAC reinforcement learning includes an actor network and a critic network, and the strategy determination module 404 is specifically used to:

[0115] For each control step in the current learning sequence, the environment parameters are estimated by RELS, which include environment stiffness and environment position.

[0116] Adjusting the reference motion trajectory of the polishing robot based on the estimated environmental parameters, the preset expected contact force and the initial motion trajectory;

[0117] Control the grinding robot to move along the reference motion trajectory, obtain the actual contact force at the end of the robot, and calculate the force error between the actual contact force and the expected contact force;

[0118] Based on the actor network, the terminal compensation amount is selected as the action at the current moment of reinforcement learning ;

[0119] calculating an adjustment amount of an impedance parameter based on the force error and applying the adjustment amount to an impedance controller;

[0120] determining joint displacement instructions of the robot based on the adjusted impedance parameters;

[0121] Calculate the current reward value of reinforcement learning based on the reward function , determine the new state based on the critic network , collect data tuples And store it in the experience replay pool;

[0122] Every n control steps, data is sampled from the experience replay pool to update the actor network and critic network;

[0123] After completing the current learning sequence, evaluate the average reward and performance of the SAC to monitor the learning progress;

[0124] When the maximum number of learning sequences is reached or SAC converges, the trained actor network is output for impedance control.

[0125] Optionally, data is sampled from the experience replay pool to update the actor network and critic network, including:

[0126] The actor network and critic network are updated by sampling data from the experience replay pool through random priority sampling.

[0127] Optionally, update the actor network and critic network by sampling data from the experience replay pool through random priority sampling, including:

[0128] The experience replay priority of the sampled data in the experience replay pool is determined based on the generation time and the size of the TD-Error in the experience replay pool.

[0129] Optionally, update the actor network and critic network by sampling data from the experience replay pool through random priority sampling, including:

[0130] Determine the sampling range adjustment factor for the kth mini-batch in the experience replay pool;

[0131] Determine the number of valid samples considered when updating the kth mini-batch based on the sampling range adjustment factor;

[0132] Based on the determined number of valid samples, the latest samples are randomly selected from the experience replay pool to update the actor network and critic network.

[0133] Optionally, determine the sampling range adjustment factor for the kth mini-batch in the experience replay pool, satisfying:

[0134]

[0135] in, is the sampling range adjustment factor of the kth mini-batch in the experience replay pool, is the total number of samples in the experience replay pool, is the learning rate, is the generation time of the sampling data, is the time when the earliest sampling data was generated, The minimum value of the sampling range adjustment factor.

[0136] Optionally, the policy determination module 404 is specifically configured to:

[0137] A nonlinear tracking differentiator is used for force planning during the impact phase of the grinding robot.

[0138] The initial estimation of the environmental stiffness is performed through RELS, and the initial parameters of the impedance controller are set. The initial parameters include the inertia coefficient, stiffness coefficient and damping coefficient.

[0139] Set the maximum number of learning sequences for the polishing robot agent to interact with the environment, the maximum number of control steps in each learning sequence, set the initial motion trajectory and expected contact force of the polishing robot agent, and initialize the actor network, critic network, and experience replay pool of the improved SAC reinforcement learning.

[0140] Optionally, the impedance controller satisfies:

[0141]

[0142] in, is the inertia coefficient, is the damping coefficient, is the stiffness coefficient, The actual position of the robot end With the expected position Position error, and They are The first and second derivatives of The actual contact force Expected contact force The force error, is the proportionality coefficient, is the integration coefficient.

[0143] Optionally, the reward function satisfies:

[0144]

[0145] in, is the reward function, 、 、 and are all constants, The actual contact force Expected contact force The error, is the robot terminal speed; The entropy of the actions selected by the current policy, is the temperature coefficient, is the proportionality coefficient, To strengthen the learning of the action space at the current moment, To reinforce the state space of the current moment, The current strategy for reinforcement learning.

[0146] It is understandable that the detailed functional implementation of each of the above units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0147] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0148] Based on the methods described in the above embodiments, embodiments of the present application provide an electronic device. The device may include: at least one memory for storing programs and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is configured to execute the methods described in the above embodiments.

[0149] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, the electronic device may include: a processor (Processor) 501, a communication interface (Communications Interface) 502, a memory (Memory) 503, and a communication bus 504. The processor 501, the communication interface 502, and the memory 503 communicate with each other via the communication bus 504. The processor 501 may call software instructions in the memory 503 to execute the method described in the above embodiment.

[0150] In addition, the logic instructions in the aforementioned memory 503 can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application.

[0151] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0152] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0153] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0154] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC.

[0155] The above embodiments can be implemented in whole or in part through software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. A computer program product comprises one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. Available media can include magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0156] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0157] It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A robot constant force grinding control method based on improved SAC reinforcement learning, characterized in that: include: Establish a contact simulation model of the polishing robot on the MuJoCo platform; The Markov decision process is used to model the control process of the constant force grinding of the grinding robot. The state space, action space and reward function are set to establish the robot constant force grinding control model. A position-based impedance controller is constructed, and the grinding robot is controlled to perform constant-force grinding based on the impedance controller; the impedance controller includes unknown impedance parameters, which include a proportional coefficient and an integral coefficient; the impedance equation of the position-based impedance controller is: in, is the inertia coefficient, is the damping coefficient, is the stiffness coefficient, The actual position of the robot end With expected position Position error, and They are The first and second derivatives of The actual contact force Expected contact force The force error, is the proportionality coefficient, is the integral coefficient; based on the improved SAC reinforcement learning, the impedance parameters in the impedance controller are solved, the optimal control strategy of the polishing robot is determined, and the constant force polishing control of the polishing robot is realized based on the optimal control strategy; The network structure of the improved SAC reinforcement learning includes an actor network and a critic network. Solving the impedance parameters in the impedance controller based on the improved SAC reinforcement learning includes: For each control step in the current learning sequence, the environment parameters are estimated by RELS, where the environment parameters include environment stiffness and environment position; Adjusting the reference motion trajectory of the polishing robot based on the estimated environmental parameters, the preset expected contact force and the initial motion trajectory; Controlling the polishing robot to move along the reference motion trajectory, obtaining an actual contact force at the end of the robot, and calculating a force error between the actual contact force and the expected contact force; Based on the actor network, the terminal compensation amount is selected as the action at the current moment of reinforcement learning ; Calculating an adjustment amount of the impedance parameter based on the force error and applying the adjustment amount to the impedance controller; determining joint displacement commands of the robot based on the adjusted impedance parameters; Calculate the reward value of reinforcement learning at the current moment based on the reward function , determine the new state based on the critic network , collect data tuples And store it in the experience replay pool; Every n control steps, data is sampled from the experience replay pool to update the actor network and the critic network; After completing the current learning sequence, evaluate the average reward and performance of the SAC to monitor the learning progress; When the maximum number of learning sequences is reached or SAC converges, the trained actor network is output for impedance control; A nonlinear tracking differentiator is used for force planning during the impact phase of the grinding robot. Performing an initial estimation of the environmental stiffness through RELS and setting initial parameters in the impedance controller, wherein the initial parameters include an inertia coefficient, a stiffness coefficient, and a damping coefficient; Set the maximum number of learning sequences for the polishing robot agent to interact with the environment, the maximum number of control steps in each learning sequence, set the initial motion trajectory and expected contact force of the polishing robot agent, and initialize the actor network, critic network, and experience replay pool of the improved SAC reinforcement learning; The reward function satisfies: in, is the reward function, 、 、 and are all constants, The actual contact force Expected contact force The error, is the robot terminal speed; The entropy of the actions selected by the current policy, is the temperature coefficient, is the proportionality coefficient, To strengthen the learning of the action space at the current moment, To reinforce the state space of the current moment, The current strategy for reinforcement learning.

2. The robot constant force grinding control method according to claim 1, characterized in that: Sampling data from the experience replay pool to update the actor network and the critic network includes: The actor network and the critic network are updated by sampling data from the experience replay pool through random priority sampling.

3. The robot constant force grinding control method according to claim 2, characterized in that: The updating of the actor network and the critic network by sampling data from the experience replay pool through random priority sampling includes: The experience replay priority of the sampled data in the experience replay pool is determined based on the generation time of the sampled data in the experience replay pool and the size of the TD-Error.

4. The robot constant force grinding control method according to claim 2, characterized in that: The updating of the actor network and the critic network by sampling data from the experience replay pool through random priority sampling includes: Determining a sampling range adjustment factor for the kth mini-batch in the experience replay pool; Determining the number of valid samples considered when updating the k-th mini-batch based on the sampling range adjustment factor; Based on the determined number of valid samples, the latest sample is randomly selected from the experience replay pool to update the actor network and the critic network.

5. The robot constant force grinding control method according to claim 4, characterized in that: The determination of the sampling range adjustment factor of the kth mini-batch in the experience replay pool satisfies: in, is the sampling range adjustment factor of the kth mini-batch in the experience replay pool, is the total number of samples in the experience replay pool, is the learning rate, is the generation time of the sampling data, is the time when the earliest sampling data was generated, The minimum value of the sampling range adjustment factor.

6. A robot constant force grinding control system based on improved SAC reinforcement learning, characterized in that: include: The simulation module is used to build a contact simulation model of the polishing robot on the MuJoCo platform; The first modeling module is used to perform Markov decision process modeling on the control process of the constant force grinding of the grinding robot, set the state space, action space and reward function, and establish the robot constant force grinding control model; The second modeling module is used to construct a position-based impedance controller, and control the polishing robot to perform constant force polishing based on the impedance controller; the impedance controller includes unknown impedance parameters, and the impedance parameters include a proportional coefficient and an integral coefficient; the impedance equation of the position-based impedance controller is: in, is the inertia coefficient, is the damping coefficient, is the stiffness coefficient, The actual position of the robot end With expected position Position error, and They are The first and second derivatives of The actual contact force Expected contact force The force error, is the proportionality coefficient, is the integration coefficient; a strategy determination module, configured to solve the impedance parameters in the impedance controller based on improved SAC reinforcement learning, determine the optimal control strategy of the polishing robot, and implement constant-force polishing control of the polishing robot based on the optimal control strategy; The network structure of the improved SAC reinforcement learning includes an actor network and a critic network. Solving the impedance parameters in the impedance controller based on the improved SAC reinforcement learning includes: For each control step in the current learning sequence, the environment parameters are estimated by RELS, where the environment parameters include environment stiffness and environment position; Adjusting the reference motion trajectory of the polishing robot based on the estimated environmental parameters, the preset expected contact force and the initial motion trajectory; Controlling the polishing robot to move along the reference motion trajectory, obtaining an actual contact force at the end of the robot, and calculating a force error between the actual contact force and the expected contact force; Based on the actor network, the terminal compensation amount is selected as the action at the current moment of reinforcement learning ; Calculating an adjustment amount of the impedance parameter based on the force error and applying the adjustment amount to the impedance controller; determining joint displacement commands of the robot based on the adjusted impedance parameters; Calculate the reward value of reinforcement learning at the current moment based on the reward function , determine the new state based on the critic network , collect data tuples And store it in the experience replay pool; Every n control steps, data is sampled from the experience replay pool to update the actor network and the critic network; After completing the current learning sequence, evaluate the average reward and performance of the SAC to monitor the learning progress; When the maximum number of learning sequences is reached or SAC converges, the trained actor network is output for impedance control; A nonlinear tracking differentiator is used for force planning during the impact phase of the grinding robot. Performing an initial estimation of the environmental stiffness through RELS and setting initial parameters in the impedance controller, wherein the initial parameters include an inertia coefficient, a stiffness coefficient, and a damping coefficient; Set the maximum number of learning sequences for the polishing robot agent to interact with the environment, the maximum number of control steps in each learning sequence, set the initial motion trajectory and expected contact force of the polishing robot agent, and initialize the actor network, critic network, and experience replay pool of the improved SAC reinforcement learning; The reward function satisfies: in, is the reward function, 、 、 and are all constants, The actual contact force Expected contact force The error, is the robot terminal speed; The entropy of the actions selected by the current policy, is the temperature coefficient, is the proportionality coefficient, To strengthen the learning of the action space at the current moment, To reinforce the state space of the current moment, The current strategy for reinforcement learning.

Images