Robotic arm intelligent control method, system, readable storage medium and computer

Abstract

The present invention provides a method, system, readable storage medium and computer for intelligent control of a robotic arm, the method comprising: sampling an online experience buffer and an offline experience buffer of the robotic arm to construct online state parameters and offline state parameters of the robotic arm; at the current moment, controlling the robotic arm to observe itself according to the online state parameters to obtain state observation data, and obtaining a corresponding reward value according to the state observation data; performing conservative reinforcement learning on the robotic arm based on the offline experience buffer, and learning the robotic arm based on the conservative reinforcement learning result and the reward value, so as to transfer the state observation data of the robotic arm at the current moment to the state parameters at the next moment; generating state data corresponding to the robotic arm according to the state parameters at the next moment, and controlling the robotic arm to perform corresponding actions according to the state data. The present invention utilizes the offline-to-online reinforcement learning method to improve the effect and personalization level of rehabilitation training.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a robot arm intelligent control method, system, readable storage medium and computer. Background Art

[0002] Traditional rehabilitation training methods usually rely on manual guidance, which is not only inefficient, but also a waste of physical strength and energy for rehabilitation therapists. In order to solve these problems, the application of robotics in rehabilitation engineering has gradually received attention.

[0003] In recent years, the application of robotic arms in upper limb rehabilitation has made significant progress. With the assistance of robotic arms, patients can perform various rehabilitation training movements, thereby promoting the recovery of upper limb function. However, the control method of the robotic arm directly affects the effect and safety of rehabilitation training. Traditional robotic arm control methods mainly rely on pre-programmed fixed trajectories and torque control, which lacks flexibility and is difficult to adapt to the differentiated needs of individual patients. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a robot arm intelligent control method, system, readable storage medium and computer to at least solve the deficiencies in the above-mentioned technology.

[0005] The present invention provides a method for intelligently controlling a robotic arm, comprising:

[0006] Sampling the online experience buffer and the offline experience buffer of the robotic arm to construct online state parameters and offline state parameters of the robotic arm;

[0007] At the current moment, the robot arm is controlled to observe itself according to the online state parameters to obtain state observation data, and a corresponding reward value is obtained after executing a strategy action according to the state observation data;

[0008] Performing conservative reinforcement learning on the robotic arm based on the offline experience buffer, and learning the robotic arm based on the conservative reinforcement learning result and the reward value, so as to transfer the state observation data of the robotic arm at the current moment to the state parameters at the next moment;

[0009] The state data corresponding to the robot arm is generated according to the state parameters at the next moment, and the robot arm is controlled to perform a corresponding action according to the state data.

[0010] Furthermore, the step of sampling the online experience buffer and the offline experience buffer of the robot arm to construct the online state parameters and the offline state parameters of the robot arm includes:

[0011] Using a variational representation of the F-divergence algorithm, two probability density functions defined on the observation space are compared, and a density ratio of the probability density functions is estimated using a parametric model;

[0012] The online experience buffer and the offline experience buffer of the robot arm are sampled according to the density ratio to construct the online state parameters and the offline state parameters of the robot arm.

[0013] Furthermore, at the current moment, the step of controlling the robot arm to obtain state observation data obtained by self-observation according to the online state parameters, and obtaining a corresponding reward value after executing a strategy action according to the state observation data includes:

[0014] At each time step Under the condition, the robot arm is controlled to perform self-observation according to the online state parameters to obtain state observation data ;

[0015] According to the strategy of the robot arm and the state observation data Execute the corresponding action Interact with the robot's environment to generate a corresponding reward value .

[0016] Furthermore, the step of performing conservative reinforcement learning on the robotic arm based on the offline experience buffer includes:

[0017] Using the SAC algorithm and the offline experience buffer to perform conservative reinforcement learning on the robotic arm, and in the conservative reinforcement learning process, using the offline reinforcement learning algorithm and the offline experience buffer to train the robotic arm;

[0018] Through The function is a conservative estimate of the true value of the current strategy of the robot. The lower bound of the function is used to obtain the corresponding conservative reinforcement learning results.

[0019] The present invention also proposes a robot arm intelligent control system, comprising:

[0020] A data sampling module, used for sampling the online experience buffer and the offline experience buffer of the robot arm to construct the online state parameters and the offline state parameters of the robot arm;

[0021] A state observation module, used to control the robot arm to observe itself according to the online state parameters to obtain state observation data at the current moment, and to obtain a corresponding reward value after executing a strategy action according to the state observation data;

[0022] A data transfer module, used to perform conservative reinforcement learning on the robotic arm based on the offline experience buffer, and to learn the robotic arm based on the conservative reinforcement learning result and the reward value, so as to transfer the state observation data of the robotic arm at the current moment to the state parameters at the next moment;

[0023] The intelligent control module is used to generate state data corresponding to the robotic arm according to the state parameters at the next moment, and control the robotic arm to perform corresponding actions according to the state data.

[0024] Furthermore, the data sampling module includes:

[0025] A density ratio estimation unit, used for comparing two probability density functions defined in the observation space by using a variational representation of the F-divergence algorithm, and estimating a density ratio of the probability density functions by using a parameter model;

[0026] A parameter construction unit is used to sample the online experience buffer and the offline experience buffer of the robot arm according to the density ratio to construct the online state parameters and the offline state parameters of the robot arm.

[0027] Furthermore, the state observation module includes:

[0028] The state observation unit is used at each time step Under the condition, the robot arm is controlled to perform self-observation according to the online state parameters to obtain state observation data ;

[0029] A data interaction unit is used to control the robot arm according to the strategy of the robot arm. and the state observation data Execute the corresponding action Interact with the robot's environment to generate a corresponding reward value .

[0030] Furthermore, the data transfer module includes:

[0031] A reinforcement learning unit, used to perform conservative reinforcement learning on the robotic arm using the SAC algorithm and the offline experience buffer, and in the conservative reinforcement learning process, train the robotic arm using the offline reinforcement learning algorithm and the offline experience buffer;

[0032] The data processing unit is used to The function is a conservative estimate of the true value of the current strategy of the robot. The lower bound of the function is used to obtain the corresponding conservative reinforcement learning results.

[0033] The present invention also proposes a readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned intelligent control method of the robotic arm is implemented.

[0034] The present invention also proposes a computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned intelligent control method of the robotic arm when executing the computer program.

[0035] The intelligent control method, system, readable storage medium and computer of the robotic arm in the present invention utilize offline to online reinforcement learning to improve the effect and personalization level of rehabilitation training. The offline reinforcement learning method is trained by a large amount of historical data to obtain a relatively stable and reliable control strategy; the online reinforcement learning method can continuously adjust and optimize the control strategy during the actual training process, so as to better adapt to the real-time changing environment and patient needs; and in the data sampling process, a method based on the neural network approximate density ratio is used to determine the correlation between online data and offline data, so as to select those samples that are most relevant to the current strategy for training. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the intelligent control method of a robotic arm in the first embodiment of the present invention;

[0037] Figure 2 for Figure 1 Detailed flow chart of step S101;

[0038] Figure 3 for Figure 1 Detailed flow chart of step S102;

[0039] Figure 4 for Figure 1 Detailed flow chart of step S103;

[0040] Figure 5 is a framework diagram of the robot arm intelligent control method in the first embodiment of the present invention;

[0041] Figure 6 is a structural block diagram of a robot arm intelligent control system in a second embodiment of the present invention;

[0042] Figure 7 FIG. 4 is a structural block diagram of a computer in a third embodiment of the present invention.

[0043] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0044] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0046] Embodiment 1

[0047] See also Figure 1 , which shows a robot arm intelligent control method in a first embodiment of the present invention, and the method specifically includes steps S101 to S105:

[0048] S101, sampling an online experience buffer and an offline experience buffer of the robot arm to construct online state parameters and offline state parameters of the robot arm;

[0049] For further information, see Figure 2 , the step S101 specifically includes steps S1011~S1012:

[0050] S1011, using a variational representation of an F-divergence algorithm, comparing two probability density functions defined in an observation space, and estimating a density ratio of the probability density functions using a parameter model;

[0051] S1012: Sample the online experience buffer and the offline experience buffer of the robot arm according to the density ratio to construct online state parameters and offline state parameters of the robot arm.

[0052] In the specific implementation, from the offline experience buffer and online experience buffer Sampling in , sampling probability and density ratio In this way, the high density ratio Locate a sample whose correlation is close to the online strategy. Is generated The stationary distribution of any behavior strategy, Is generated This corresponds to the mixture of online policies observed during fine-tuning. However, the estimated and The probability of is difficult because these probabilities are stationary distributions of complex strategy mixtures. A method of likelihood-free density ratio estimation is used to estimate this density ratio by training a parameterized neural network. The training process is as follows:

[0053] Using the variational representation of the F-divergence algorithm, let and is a probability measure defined on some observable space, that is, and is the distribution of the probability of occurrence of different events in space X, where right Absolutely continuous, and defines the function represents the difference between the two distributions and is used in the subsequent construction of the Jensen-Shannon divergence, where represents the density ratio; Jensen-Shannon (JS) divergence is defined as , to measure the two probability distributions and The JS divergence is symmetric and bounded; X represents the calculation of the divergence on the observable space X; The lower bound of To estimate the density ratio , that is, in space X, for a given point x, the distribution and The density ratio of Relative to Probability:

[0054]

[0055] in, Indicates that the distribution Next, sample x and calculate the expected value, parameterized by a neural network whose output is forced to be non-negative via an activation function, represents convex conjugation. By considering the density and The probability distribution of and ,estimate .

[0056] S102, at the current moment, controlling the robot arm to observe itself according to the online state parameters to obtain state observation data, and obtaining a corresponding reward value after executing a strategy action according to the state observation data;

[0057] For further information, see Figure 3, the step S102 specifically includes steps S1021~S1022:

[0058] S1021, at each time step Under the condition, the robot arm is controlled to perform self-observation according to the online state parameters to obtain state observation data ;

[0059] S1022, according to the strategy of the robot arm and the state observation data Execute the corresponding action Interact with the robot's environment to generate a corresponding reward value .

[0060] In the specific implementation, at each time step Under the control, the robot arm performs self-observation according to the online state parameters to obtain state observation data , according to the strategy of the robot and state observation data Execute the corresponding action Interact with the robot's environment to generate corresponding reward values .

[0061] In this embodiment, the state observation data Including joint angles, end effector position, speed, force feedback and other information. Then, according to the task goal, such as moving the end effector to a specific position, the preset reward function is used to calculate the immediate reward value. In addition, the reward value takes into account the long-term reward, and calculates the impact of the current action on the future reward by introducing a discount factor, thereby guiding the robot arm to gradually optimize its action to achieve better task execution results.

[0062] S103, performing conservative reinforcement learning on the robotic arm based on the offline experience buffer, and learning the robotic arm based on the conservative reinforcement learning result and the reward value, so as to transfer the state observation data of the robotic arm at the current moment to the state parameters at the next moment;

[0063] For further information, see Figure 4 , the step S103 specifically includes steps S1031~S1032:

[0064] S1031, performing conservative reinforcement learning on the robotic arm using the SAC algorithm and the offline experience buffer, and in the conservative reinforcement learning process, training the robotic arm using the offline reinforcement learning algorithm and the offline experience buffer;

[0065] S1032, through The function is a conservative estimate of the true value of the current strategy of the robot. The lower bound of the function is used to obtain the corresponding conservative reinforcement learning results.

[0066] In the specific implementation, the Soft Actor-Critic (SAC) algorithm is used for learning in the off-policy reinforcement learning algorithm. function , Represents the parameters of the function, random strategy is a Gaussian model, parameterized as To ensure the effectiveness of the strategy, we minimize the objective and To update the interaction between the Actor-Critic framework; ;

[0068] ;

[0069] in, and They represent the Critic (evaluator) loss function and the Actor (strategy) loss function in the SAC algorithm respectively. represents the experience buffer, represents the delay parameter, represents the temperature parameter, Indicates the current status and actions The corresponding state-action value function is parameterized as . Indicates the current status and actions The instant rewards you receive, Represents the discount factor, which is used to calculate the discounted value of future rewards and measure the impact of long-term returns. Represents sampling a set of triplets from the experience buffer B The expected calculation is Indicated in strategy Next, according to the status Sampling to get action And perform expectation calculation, Representation strategy In Status Next select action The logarithmic probability of , corresponding to the entropy regularization term. Indicates sampling state from experience buffer B , and according to the strategy Sampling Action Expected calculations performed.

[0070] Furthermore, in the offline reinforcement learning part, the Conservative Q-Learning (CQL) method is used to train the intelligent agent (robotic arm). In this stage, the model is trained using a static data set to ensure that the intelligent agent can still learn effectively without real-time environment interaction. The core idea of ​​the CQL method is to The function makes a conservative estimate, thus providing a realistic The lower bound of the function. Specifically, by conservatively estimating the samples in the static data set, the policy error caused by overfitting is reduced, and the expected value is used to evaluate the performance of the current policy. The advantage of this is that robust policy training can be performed on offline data, avoiding generalization problems caused by overfitting the data set.

[0071] Specifically, the strategy evaluation in CQL is minimized as follows:

[0072]

[0073] Among them, the experience behavior strategy , the molecule represents the state Next select action The number of samples, the denominator represents the state The total number of occurrences. is the trade-off factor, represents the Bellman operator, which reflects the strategy In the next state The expected return under this condition is combined with the immediate reward and the expected future reward to generate an updated The first term in the above formula is the conventional Bellman backup term, and the second term is the regularization term, which reduces the value, while increasing the number of actions that have been seen value. And this trained pessimism Functions are good for fine-tuning. To calculate all possible actions Next The weighted sum of the values ​​and the logarithm are taken to reflect the current strategy for the state An optimistic estimate of all possible actions. Indicates behavior strategy based on experience Action Sampling and calculating The expected value is used to correct the state Estimates of realistically feasible strategies.

[0074] In this embodiment, the reward function is set up including factors such as the robot position, speed, acceleration, and motor offset angle:

[0075] ;

[0076] in: Represents the distance between the end effector of the robot arm and the target position; Indicates the speed of the end effector of the robot arm; represents the acceleration of the end effector of the robot arm; represents the penalty term for the motor offset angle; , , , Respectively represent the weights of position, velocity, acceleration, and offset angle, which are used to balance the impact of each on the reward.

[0077] S104, generating state data corresponding to the robotic arm according to the state parameters at the next moment, and controlling the robotic arm to perform corresponding actions according to the state data.

[0078] In the specific implementation, the state parameters at the next moment obtained above are used to generate the state data corresponding to the robot arm, and the robot arm is controlled to perform the corresponding action according to the state data, wherein the goal of the robot arm is to maximize the expected benefit In Strategy Under this condition, the unnormalized steady-state state-action distribution is defined as ,in represents the Markov chain at time step Out of state And perform actions distribution.

[0079] In this embodiment, in order to more effectively alleviate the problem of distribution shift, multiple conservatively estimated trained Function, the framework considers an ensemble consisting of N CQL agents (robotic arms) that are pre-trained using the CQL update rule. At this point, the policy is modeled as a Gaussian distribution whose mean and variance are determined by the mean and variance of the Gaussian mixture policy. During fine-tuning, each parameter is updated according to the update rule. By using conservative Function, the agent (robotic arm) takes conservative estimates of actions that it has not seen in the early stages of the fine-tuning process. This is because the states in the early stages are similar to those in the offline dataset. The function can generalize to this state, and such operation can protect the initial policy from serious guidance errors. Function, can obtain more conservative estimates for unseen data areas. The specific framework diagram is as follows Figure 5 shown.

[0080] In summary, the intelligent control method of the robotic arm in the above-mentioned embodiment of the present invention utilizes offline to online reinforcement learning to improve the effect and personalization level of rehabilitation training. The offline reinforcement learning method is trained by a large amount of historical data to obtain a relatively stable and reliable control strategy; the online reinforcement learning method can continuously adjust and optimize the control strategy during the actual training process, so as to better adapt to the real-time changing environment and patient needs; and in the data sampling process, a method based on the neural network approximate density ratio is used to determine the correlation between the online data and the offline data, so as to select those samples that are most relevant to the current strategy for training.

[0081] Embodiment 2

[0082] Another aspect of the present invention is to provide a robot arm intelligent control system, please refer to Figure 6 , which shows a robot arm intelligent control system in a second embodiment of the present invention, the system comprises:

[0083] A data sampling module 11 is used to sample the online experience buffer and the offline experience buffer of the robot arm to construct the online state parameters and the offline state parameters of the robot arm;

[0084] Furthermore, the data sampling module 11 includes:

[0085] A density ratio estimation unit, used for comparing two probability density functions defined in the observation space by using a variational representation of the F-divergence algorithm, and estimating a density ratio of the probability density functions by using a parameter model;

[0086] A parameter construction unit is used to sample the online experience buffer and the offline experience buffer of the robot arm according to the density ratio to construct the online state parameters and the offline state parameters of the robot arm.

[0087] The state observation module 12 is used to control the robot arm to observe itself according to the online state parameters to obtain state observation data at the current moment, and to obtain a corresponding reward value after executing a strategy action according to the state observation data;

[0088] Furthermore, the state observation module 12 includes:

[0089] The state observation unit is used at each time step Under the condition, the robot arm is controlled to perform self-observation according to the online state parameters to obtain state observation data ;

[0090] A data interaction unit is used to control the robot arm according to the strategy of the robot arm. and the state observation data Execute the corresponding action Interact with the robot's environment to generate a corresponding reward value .

[0091] A data transfer module 13, used to perform conservative reinforcement learning on the robotic arm based on the offline experience buffer, and to learn the robotic arm based on the conservative reinforcement learning result and the reward value, so as to transfer the state observation data of the robotic arm at the current moment to the state parameters at the next moment;

[0092] Furthermore, the data transfer module 13 includes:

[0093] A reinforcement learning unit, used to perform conservative reinforcement learning on the robotic arm using the SAC algorithm and the offline experience buffer, and in the conservative reinforcement learning process, train the robotic arm using the offline reinforcement learning algorithm and the offline experience buffer;

[0094] The data processing unit is used to The function is a conservative estimate of the true value of the current strategy of the robot. The lower bound of the function is used to obtain the corresponding conservative reinforcement learning results.

[0095] The intelligent control module 14 is used to generate state data corresponding to the robotic arm according to the state parameters at the next moment, and control the robotic arm to perform corresponding actions according to the state data.

[0096] The functions or operation steps implemented when the above modules and units are executed are generally the same as those in the above method embodiments, and will not be repeated here.

[0097] The implementation principle and technical effects of the robotic arm intelligent control system provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0098] Embodiment 3

[0099] The present invention also provides a computer, see Figure 7 , shown is a computer in the third embodiment of the present invention, including a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20. When the processor 20 executes the computer program 30, the above-mentioned intelligent control method of the robotic arm is implemented.

[0100] The memory 10 includes at least one type of readable storage medium, which includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10 may be an internal storage unit of a computer, such as a hard disk of the computer. In other embodiments, the memory 10 may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Further, the memory 10 may also include both an internal storage unit of the computer and an external storage device. The memory 10 may be used not only to store application software and various types of data installed in the computer, but also to temporarily store data that has been output or is to be output.

[0101] Among them, in some embodiments, the processor 20 can be an electronic control unit (Electronic Control Unit, abbreviated as ECU, also known as a vehicle computer), a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run the program code stored in the memory 10 or process data, such as executing access restriction programs, etc.

[0102] It should be pointed out that Figure 7 The structure shown does not constitute a limitation on the computer. In other embodiments, the computer may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0103] An embodiment of the present invention further provides a readable storage medium having a computer program stored thereon, which implements the above-mentioned intelligent control method for a robotic arm when executed by a processor.

[0104] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0105] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0106] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0107] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for intelligent control of a robotic arm, characterized in that: include: The online experience buffer and the offline experience buffer of the robot arm are sampled to construct the online state parameters and the offline state parameters of the robot arm, wherein the steps of sampling the online experience buffer and the offline experience buffer of the robot arm to construct the online state parameters and the offline state parameters of the robot arm include: Using a variational representation of the F-divergence algorithm, two probability density functions defined on the observation space are compared, and a density ratio of the probability density functions is estimated using a parametric model; Sampling the online experience buffer and the offline experience buffer of the robot arm according to the density ratio to construct online state parameters and offline state parameters of the robot arm; At the current moment, the robot arm is controlled to observe itself according to the online state parameters to obtain state observation data, and a corresponding reward value is obtained after executing a strategy action according to the state observation data; Performing conservative reinforcement learning on the robotic arm based on the offline experience buffer, and learning the robotic arm based on the conservative reinforcement learning result and the reward value, so as to transfer the state observation data of the robotic arm at the current moment to the state parameters at the next moment; The state data corresponding to the robot arm is generated according to the state parameters at the next moment, and the robot arm is controlled to perform a corresponding action according to the state data.

2. The intelligent control method for a robotic arm according to claim 1, characterized in that: At the current moment, the steps of controlling the robot arm to obtain state observation data obtained by self-observation according to the online state parameters, and obtaining a corresponding reward value after executing a strategy action according to the state observation data include: At each time step Under the condition, the robot arm is controlled to perform self-observation according to the online state parameters to obtain state observation data ; According to the strategy of the robot arm and the state observation data Execute the corresponding action Interact with the robot's environment to generate a corresponding reward value .

3. The intelligent control method for a robotic arm according to claim 1, characterized in that: The step of performing conservative reinforcement learning on the robotic arm based on the offline experience buffer comprises: Using the SAC algorithm and the offline experience buffer to perform conservative reinforcement learning on the robotic arm, and in the conservative reinforcement learning process, using the offline reinforcement learning algorithm and the offline experience buffer to train the robotic arm; Through The function is a conservative estimate of the true value of the current strategy of the robot. The lower bound of the function is used to obtain the corresponding conservative reinforcement learning results.

4. A robotic arm intelligent control system, characterized in that: include: A data sampling module is used to sample the online experience buffer and the offline experience buffer of the robot arm to construct the online state parameters and the offline state parameters of the robot arm, wherein the data sampling module includes: A density ratio estimation unit, used for comparing two probability density functions defined in the observation space by using a variational representation of the F-divergence algorithm, and estimating a density ratio of the probability density functions by using a parameter model; a parameter construction unit, configured to sample an online experience buffer and an offline experience buffer of the robot arm according to the density ratio, so as to construct online state parameters and offline state parameters of the robot arm; A state observation module, used to control the robot arm to observe itself according to the online state parameters to obtain state observation data at the current moment, and to obtain a corresponding reward value after executing a strategy action according to the state observation data; A data transfer module, used to perform conservative reinforcement learning on the robotic arm based on the offline experience buffer, and to learn the robotic arm based on the conservative reinforcement learning result and the reward value, so as to transfer the state observation data of the robotic arm at the current moment to the state parameters at the next moment; The intelligent control module is used to generate state data corresponding to the robotic arm according to the state parameters at the next moment, and control the robotic arm to perform corresponding actions according to the state data.

5. The intelligent control system for a robotic arm according to claim 4, characterized in that: The state observation module comprises: The state observation unit is used at each time step Under the condition, the robot arm is controlled to perform self-observation according to the online state parameters to obtain state observation data ; A data interaction unit is used to control the robot arm according to the strategy of the robot arm. and the state observation data Execute the corresponding action Interact with the robot's environment to generate a corresponding reward value .

6. The intelligent control system for a robotic arm according to claim 4, characterized in that: The data transfer module comprises: A reinforcement learning unit, used to perform conservative reinforcement learning on the robotic arm using the SAC algorithm and the offline experience buffer, and in the conservative reinforcement learning process, train the robotic arm using the offline reinforcement learning algorithm and the offline experience buffer; The data processing unit is used to The function is a conservative estimate of the true value of the current strategy of the robot. The lower bound of the function is used to obtain the corresponding conservative reinforcement learning results.

7. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the intelligent control method for a robotic arm as described in any one of claims 1 to 3 is implemented.

8. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the robot arm intelligent control method as described in any one of claims 1 to 3 is implemented.

Images