Mechanical arm control method and device, electronic equipment and storage medium

Abstract

The invention discloses a mechanical arm control method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in the process that a mechanical arm control model controls a mechanical arm, reward value remodeling processing is carried out in combination with a mode based on visual perception excitation, a remodeled reward value is determined, the reward sparsity can be reduced, and the reward remodeling efficiency is improved. Various kinds of information generated by the mechanical arm control model in the mechanical arm control process are used for generating sample data, then importance degree setting is carried out on the sample data based on the remodeling reward value, sampling is carried out according to the importance degree of the sample data, and compared with uniform sampling in the prior art, the sampling efficiency is improved; the technical problems of how to improve the sampling efficiency of the algorithm in the mechanical arm control process and solving the technical problems of sparse rewards of the algorithm can be solved, and the technical effects of improving the sampling efficiency of the algorithm and solving the sparse rewards of the algorithm are achieved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a robot arm control method and device, electronic equipment and storage medium. Background Art

[0002] With the development of science and technology, robotics research has always been an important field of scientific and technological development, and is widely used in home services, manufacturing, national defense science and technology, aerospace, medical assistance and other fields. With the development of artificial intelligence technology, combining artificial intelligence with robotics technology and studying robot control systems with autonomous decision-making and learning capabilities have become important branches and frontier focuses of robotics research. Among them, the robotic arm, as an important actuator of the robot, is widely used in key areas of production and life. The research and development of the robotic arm control system is of great significance to the development of robots. The robotic arm control system is based on the robot's own perception, decision-making, planning and control capabilities, and the operation goal is to achieve the target state from the initial state.

[0003] Traditional robotic arm control methods use reinforcement learning to enable the robot to continuously interact with the environment and learn through trial and error, so as to independently obtain optimization strategies and adapt to the dynamically changing external environment. However, traditional robotic arm control methods do not take into account the problems of low algorithm sampling efficiency and sparse rewards when controlling the robotic arm through deep learning. Therefore, how to improve the sampling efficiency of the algorithm and solve the sparse rewards of the algorithm during the robotic arm control process are currently urgent issues to be solved. Summary of the invention

[0004] The present application provides a robot arm control method and device, an electronic device and a storage medium to at least solve the problem of how to improve the sampling efficiency of the algorithm in the robot arm control process in the related art, and solve the problem of sparse rewards of the algorithm.

[0005] The present application provides a robot arm control method, comprising: Obtaining the state information, action information, state identification, and remodeling reward value of the robotic arm generated in the process of controlling the robotic arm based on the robotic arm control model, wherein the robotic arm control model is a model built based on a deep reinforcement learning algorithm, and the remodeling reward value is a reward value obtained by remodeling the reward value in combination with a method based on visual perception incentives; Generate sample data according to state information, action information, reshaping reward value and state identifier, and store the sample data in a sample pool; Based on the reshaping reward value, the importance of the sample data in the sample pool is set to obtain the importance of the sample data, and based on the importance of the sample data, the sample data in the sample pool is sampled to obtain the sample data; Performing model parameter optimization processing on the robot arm control model according to the sampling data to obtain an optimized robot arm control model; The robot arm is controlled according to the optimized robot arm control model.

[0006] The present application also provides a robot arm control device, comprising: An acquisition unit is used to acquire state information, action information, state identification, and remodeling reward value of the robot arm generated in the process of controlling the robot arm based on the robot arm control model, wherein the robot arm control model is a model built based on a deep reinforcement learning algorithm, and the remodeling reward value is a reward value obtained by remodeling the reward value in combination with a method based on visual perception incentives; A generating unit, used for generating sample data according to the state information, the action information, the reshaping reward value and the state identifier, and storing the sample data in a sample pool; A setting unit, used to perform importance setting processing on sample data in a sample pool based on the reshaping reward value to obtain the importance corresponding to the sample data; A sampling unit, used for performing data sampling processing on the sample data in the sample pool based on the importance of the sample data to obtain sample data; An optimization unit, used for performing model parameter optimization processing on the robot arm control model according to the sampling data to obtain an optimized robot arm control model; The control unit is used to control the robot arm according to the optimized robot arm control model.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned robot arm control methods when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned robot arm control methods are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned robot arm control methods when the computer program is executed by a processor.

[0010] Through the present application, due to a robotic arm control method and device, electronic device and storage medium, by reshaping the reward value in the process of the robotic arm control model controlling the robotic arm, combining the visual perception incentive method to determine the reshaped reward value, the reward sparsity can be reduced, and the various information generated by the robotic arm control model in the robotic arm control process is used to generate sample data, and then the importance of the sample data is set based on the reshaped reward value, and sampling is performed according to the importance of the sample data. Compared with the uniform sampling in the prior art, the sampling efficiency is improved, and the technical problem of how to improve the sampling efficiency of the algorithm in the process of robotic arm control and solve the reward sparsity of the algorithm can be solved, so as to achieve the technical effect of improving the sampling efficiency of the algorithm and solving the reward sparsity of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A schematic diagram of a flow chart of a robot arm control method provided in an embodiment of the present application; Figure 2 A schematic diagram of a process of obtaining information provided in an embodiment of the present application; Figure 3 A schematic diagram of the principle of a method for setting importance provided in an embodiment of the present application; Figure 4 A schematic diagram of a visual perception incentive strategy provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a robotic arm control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0015] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0016] Figure 1 A flow chart of a robot arm control method provided for an embodiment of the present application is provided, and the method is described in detail in conjunction with the execution flow of the robot arm control method.

[0017] like Figure 1 As shown, the robot arm control method includes: Step 101, obtaining the state information, action information, state identification, and reshaping reward value of the robot arm generated in the process of controlling the robot arm based on the robot arm control model, wherein the robot arm control model is a model built based on a deep reinforcement learning algorithm, and the reshaping reward value is a reward value obtained by reshaping the reward value in combination with a method based on visual perception incentives.

[0018] In an embodiment of the present application, the robotic arm control model is an intelligent decision-making model built based on a deep reinforcement learning algorithm, the core of which adopts but is not limited to: an improved deep deterministic policy gradient (DDPG) algorithm, which coordinates four neural networks (Actor evaluation network, Actor target network, Critic evaluation network, Critic target network) through an Actor-Critic framework to handle the control problem of the robotic arm in a continuous action space.

[0019] The state information is a multi-dimensional data set that describes the real-time motion state of the robotic arm during the control process, which includes first state information and second state information. The first state information includes the state of the robotic arm before performing an action, and the second state information includes the state of the robotic arm after performing an action. The first state information and the second state information include but are not limited to: the angle characteristics of each joint of the robotic arm, the position information of the end effector, and the spatial relationship with the target position.

[0020] Action information refers to the control instructions output by the robot control model, including the actions that the robot needs to perform, expressed as the angular velocity or angle adjustment of each joint in the next time step. For example, for a robot with N degrees of freedom, the action information can be expressed as a vector ,in For the The angle change of each joint.

[0021] The status flag is a key signal used to mark the progress of the robot arm control task. It is usually in the form of a Boolean variable (such as done) to indicate whether the task is terminated or completed. For example, when the robot arm end effector reaches the target position within the allowable error range, the status flag is marked as the termination state.

[0022] Among them, the reward value obtained by reshaping the reward value in combination with a method based on visual perception incentives means that, in the process of the robot arm control model controlling the robot arm, a first reward value is first obtained by decomposing the sparse final reward value into continuous intermediate reward signals, and then in the process of the robot arm control model controlling the robot arm, a second reward value is calculated in a method based on visual perception incentives, and then the first reward value and the second reward value are combined to obtain a reshaped reward value. Furthermore, regarding the method based on visual perception incentives, for example: internal and external incentive control methods based on visual perception, etc.

[0023] Reshape reward value is a dynamic reward calculation mechanism designed to solve the problem of sparse rewards in traditional reinforcement learning. Through reward value reshaping, the sparse final reward value is decomposed into continuous intermediate reward signals. Specifically, the calculation of the reshape reward value is based on the real-time distance change between the end effector of the robot and the target position. For example, an action information consists of 5 actions. The traditional method is to generate a reward value after the five actions are completed. The reshape reward value is based on the preset number of actions (for example, 1 action). At this time, the robot generates a first reward value for each action. Then, the first reward value is combined with the second reward value obtained based on visual perception incentives to obtain the reshape reward value. The problem of the reward value coefficient can be solved through continuous intermediate reward signals.

[0024] Step 102, generating sample data according to the state information, action information, reshaping reward value and state identifier, and storing the sample data in a sample pool.

[0025] In the embodiment of the present application, the sample data is a set of multi-dimensional information generated during the interaction between the robot arm and the environment, and each sample data includes but is not limited to: the current state information in the state information, i.e., the first state information (s), the executed action information ( ), the obtained reshaping reward value (r), the next state information, i.e., the second state information (s′), and the state identification (done). Specifically, after each step of the robot is executed, the robot control model records the state information such as the joint angle and end position at the current moment, i.e., the second state information, and combines the action instructions output by the robot control model (such as: the amount of joint angle adjustment), collects the reward value fed back by the environment after executing the action (including: the superposition result of the reshaping reward value and the external reward), and detects whether the task is terminated (state identification). Various information are combined in chronological order into complete sample data {s, ,r,s′,done}, characterizes the transfer process of the robot from the current state to the next state.

[0026] The sample pool is a database for storing sample data. Its essence is an experience replay set (which can be represented by D). During the deep reinforcement learning training process, the robot arm continuously interacts with the environment to generate new sample data, which is continuously stored in the sample pool. The sample pool adopts a first-in-first-out or dynamic capacity management strategy to ensure that the stored sample data covers the interaction experience at different training stages. For example: when the sample pool reaches the preset capacity, the new sample data will replace the oldest sample data to maintain the timeliness and diversity of the data. Through the accumulation of the sample pool, the robot arm control model can learn from a large amount of historical experience, rather than relying solely on current interaction data, thereby improving the stability of training and the generalization ability of the strategy.

[0027] The sample data generation and storage process includes but is not limited to the following steps: In each optimization of the robot control model, firstly, the action information ( ), superimposed with exploration noise (such as Gaussian noise), and then sent to the robot for execution. After execution, the environment returns the second state information (s′), the calculated reshaping reward value (r), and the state flag (done). Subsequently, the robot control model returns the five-tuple {s, , r, s′, done} is stored in the sample pool as a complete sample data. This process is repeated in the continuous action selection and state transfer of the robot arm until the task is terminated or the preset number of training steps is reached.

[0028] Step 103, based on the remodeling reward value, importance setting processing is performed on the sample data in the sample pool to obtain the importance corresponding to the sample data, and based on the importance of the sample data, data sampling processing is performed on the sample data in the sample pool to obtain sample data.

[0029] In the embodiment of the present application, the importance setting process is to assign a dynamic weight (i.e., importance) to each sample data in the sample pool, with the goal of distinguishing the value contribution of different sample data to the current training stage. The importance is calculated based on the reshaping reward value (r) of the sample and the average reshaping reward value ( ) and combined with the dynamic adjustment strategy in the training phase.

[0030] Data sampling is the process of selecting training data from the sample pool based on the importance of the sample data. When performing data sampling, the importance needs to be converted into sampling probability first, so that high-importance samples are more likely to be selected for model updating.

[0031] The sampled data is a subset selected from the sample pool based on the importance of the sample data, which reflects the emphasis on high-value experience in the current training stage. Compared with uniform sampling, importance sampling based on sample data enables the robot control model to more efficiently utilize successful experience (such as action sequences approaching the target) and critical failure cases (such as collision or deviation from the trajectory), thereby accelerating policy iteration.

[0032] Step 104 , optimizing the model parameters of the robot control model according to the sampling data to obtain an optimized robot control model.

[0033] In the embodiment of the present application, the model parameter optimization process is the step of learning from the sampled data by the robot control model, and the action strategy output by the robot control model is maximized by iteratively adjusting the weight parameters of the neural network inside the robot control model. The optimization process includes but is not limited to: two parallel stages, namely, the Q-value function approximation of the Critic network and the policy gradient improvement of the Actor network. Specifically, the sampled data {s, , r, s′, done} are input into the Critic and Actor networks to calculate the error between the predicted Q value (i.e., the second expected value) and the actual target Q value (i.e., the first expected value), as well as the policy gradient direction, and then update the network parameters through the back propagation algorithm.

[0034] The optimized robot arm control model is a stable strategy network obtained through multiple rounds of parameter iteration and update. It can generate accurate action instructions in real time according to the environmental status. For example, in the grasping task, the joint angle adjustment output by the model can make the end effector approach the target along the optimal trajectory while avoiding joint overrun or collision. Compared with the robot arm control model, the optimized robot arm control model not only has higher action decision-making efficiency, but also can adapt to complex scenarios such as dynamic changes in target position and environmental noise interference.

[0035] Step 105 , controlling the robotic arm according to the optimized robotic arm control model.

[0036] In the embodiment of the present application, when the optimized robotic arm control model performs robotic arm control, the optimization process of the model will still be executed, specifically including but not limited to: obtaining new state information, new action information, new state identification, and new reshaping reward value of the robotic arm generated in the process of robotic arm control based on the optimized robotic arm control model; If the new state is marked as a terminal state, the process of optimizing the model parameters of the optimized robot control model ends; if the state is marked as a non-terminal state, the process of optimizing the model parameters of the optimized robot control model continues. That is, new sample data is generated according to the new state information, new action information, new reshaping reward value and new state mark, and the new sample data is stored in the sample pool; Based on the new reshaping reward value, the importance of the new sample data is set in the sample pool to obtain the importance corresponding to the new sample data; based on the importance of the new sample data, data sampling is performed on the new sample data in the sample pool to obtain new sampling data; according to the new sampling data, the model parameters of the optimized robotic arm control model are optimized to obtain the robotic arm control model after secondary optimization; the robotic arm is controlled according to the robotic arm control model after secondary optimization.

[0037] Repeat the above steps and continuously optimize the robot control model during the robot control process, thereby improving the adaptability and intelligence level of the robot in unstructured environments.

[0038] A robot arm control method and device, electronic device and storage medium of the present application, in the process of the robot arm control model controlling the robot arm, performs reward value reshaping processing based on visual perception incentives to determine the reshaped reward value, which can reduce the reward sparsity, and generate sample data from various information generated by the robot arm control model in the robot arm control process, and then sets the importance of the sample data based on the reshaped reward value, and performs sampling according to the importance of the sample data, which improves the sampling efficiency compared with the uniform sampling in the prior art, and can solve the technical problems of how to improve the sampling efficiency of the algorithm in the robot arm control process and solve the reward sparsity of the algorithm, so as to achieve the technical effect of improving the sampling efficiency of the algorithm and solving the reward sparsity of the algorithm.

[0039] In an achievable embodiment of the present application, regarding the acquisition of the state information, action information, state identification, and reshaping reward value of the robot arm, the present application provides a flowchart of information acquisition, such as Figure 2 As shown, including: Step 201, determine the target position information of the target position corresponding to the robot arm and the first state information of the robot arm, wherein the target position is the position expected to be reached after the robot arm is controlled, and the first state information at least includes the current state of the robot arm before performing an action.

[0040] In the embodiments of the present application, the target position information is the final position that the robot arm needs to reach during the control process, usually in three-dimensional coordinates (such as: ) or polar coordinates (e.g. horizontal radius ,angle and Depth ). For example, in a grasping task, the target position information may include but is not limited to the spatial coordinates of the center point of the manipulated object. The target position information is input into the robot control model as the ultimate goal of the control task to guide strategy generation and deviation evaluation.

[0041] The first state information is a description of the state of the robot arm before it performs an action, including but not limited to: the real-time angles of each joint of the robot arm ( ), the current position of the end effector of the robot arm, the joint speed and the initial relative distance to the target position. Specifically, if the first state information is represented by a feature vector, for example:

[0042]

[0043] Where N is the number of degrees of freedom of the robot. The first state information provides the control model with the initial environment perception basis for action decision-making. It is the characteristic vector form of the first state information, that is, the state vector.

[0044] Step 202 : generating motion information of the robot arm through a robot arm control model according to the target position information, wherein the motion information at least includes the motion performed by the robot arm.

[0045] In the embodiment of the present application, the action information is a set of instructions generated by the robot control model, which represents the motion parameters (such as angle increment) of each joint of the robot at the next time. or angular velocity). For example, for a six-DOF robotic arm, the motion information can be expressed as , where each After normalization, it is adapted to the physical range of motion of the robot. The action information is calculated by the Actor network in the robot control model based on the first state information and the target position. Its essence is a mapping strategy from state space to action space.

[0046] Step 203, the robot arm control model controls the robot arm to perform an action according to the action information, and obtains second state information of the robot arm, wherein the second state information at least includes the current state of the robot arm after performing the action, and the state information includes the first state information and the second state information.

[0047] In the embodiment of the present application, the second state information is the state of the robot arm after executing the action corresponding to the action information, including the joint angle ( ), the new position of the end effector and the real-time relative distance to the target position. The second state information is collected in real time by the robot arm sensors (such as encoders, visual systems) and fed back to the robot arm control model to evaluate the action effect and update the subsequent strategy. The continuity of state information (transition from the first state to the second state) reflects the dynamic process of the interaction between the robot arm and the environment.

[0048] Step 204 , determining a position deviation between the robot arm and the target position according to the second state information and the target position information, and determining a state identifier according to the position deviation, wherein the state identifier is an identifier of the second state information.

[0049] In the embodiment of the present application, the position deviation is the current position of the end effector of the robot arm, that is, the current position of the robot arm in the second state information ( ) and the target position ( ) can be expressed in the following ways (e.g., Euclidean distance): The position deviation is used to quantify the degree of proximity between the state of the robot after the action is executed and the task target, and is the basis for reward value calculation and state identification determination. For example: when the position deviation is less than the preset threshold , it is determined that the robot has successfully reached the target position, and the state is marked as the terminal state.

[0050] The state flag is a task progress marker based on the position deviation, usually a Boolean variable (such as done) to indicate whether the task is terminated. The introduction of the state flag enables the control model to distinguish the training phases and reset the environment at the end of the task to start a new round of control cycle.

[0051] Step 205, when the robot arm control model controls the robot arm to perform an action according to the action information, a reward value is generated every time a preset number of actions are performed until all actions in the action information are executed, and a first reward value is obtained. The first reward value is combined with a second reward value obtained by calculating the reward value based on visual perception incentives to obtain a reshaping reward value.

[0052] In an embodiment of the present application, the reshaping reward value is an incentive signal dynamically generated by the robot control model during the execution of the action. The reshaping reward value is related to the first reward value obtained based on the position deviation and the second reward value obtained based on the visual perception incentive. The calculation of the first reward value is closely related to the change of the position deviation. After each preset number of actions (such as every 5 steps, every 1 step), the robot control model generates an intermediate reward based on the difference between the latest position deviation and the historical deviation. For example: if the robot gradually approaches the target in continuous actions, it will receive a positive reward; if it deviates from the target or stagnates, it will receive a negative penalty.

[0053] Specifically, the calculation of the first reward value can be expressed by, but not limited to, formula (1): Formula (1) in, is the first reward value, For the preset number of times, During the robot arm's action The position of the robot arm at all times, When the robot arm performs an action, Before the moment of action The position of the robot arm during the action, that is Position by Location Passed After one action, for The position deviation between the position and the target position, for The position deviation from the target position.

[0054] Further, reshape the reward value ,in, To reshape the reward value, is the second reward value.

[0055] Through the acquisition of information, the full-process closed-loop optimization of the robot arm control task is achieved. First, the fusion input of the target position information and the multi-dimensional state information enables the robot arm control model to accurately perceive the relative spatial relationship between the robot arm and the target position, providing a reliable decision-making basis for action generation. Secondly, the dynamic calculation of the position deviation and the real-time determination of the state identification enhance the system's ability to autonomously evaluate the progress of the task, ensuring rapid response to abnormal situations (such as target offset or hardware failure) in complex environments. Furthermore, the periodic reshaping of the reward value generation mechanism decomposes the sparse final target reward into progressive intermediate incentives, significantly improving the learning efficiency and strategy stability of the model in long sequence actions. In addition, through the continuous update of the first state and the second state, the robot arm control model can adapt to the dynamic changes of the robot arm (such as load changes, joint wear) online and maintain high-precision control performance.

[0056] In an achievable embodiment of the present application, determining the position deviation and determining the state identifier based on the position deviation can also be implemented in the following manner but is not limited to: determining the current position coordinates corresponding to the robotic arm based on the joint angle information of the robotic arm included in the second state information; performing coordinate comparison processing on the current position coordinates and the target position coordinates of the target position included in the target position information to obtain the position deviation between the robotic arm and the target position; when the position deviation is less than or equal to a preset deviation threshold, determining the state identifier of the second state information as a termination state; when the position deviation is greater than the preset deviation threshold, determining the state identifier of the second state information as a non-termination state.

[0057] In the embodiment of the present application, the joint angle information is the real-time angle data of each joint of the robot arm after the action is performed, which is usually collected by the joint encoder or potentiometer. The current position coordinate is the real-time position of the end effector of the robot arm in the three-dimensional space, which is calculated by forward kinematics. For example: the current position of the robot arm in the second state information ( ), the target position coordinates are the three-dimensional spatial position that the end effector is expected to reach in the robot control task, that is, the target position ( ).

[0058] Coordinate comparison is the process of calculating the spatial distance between the current position coordinates and the target position coordinates. The position deviation can be expressed as express.

[0059] Preset deviation threshold ( ) is the maximum error radius allowed between the end position of the robot arm and the target position. It is a custom threshold and is usually set according to the task accuracy requirements. For example: in a precision assembly task, Can be set to millimeter level (such as: 1mm), and in material handling tasks, It can be relaxed to centimeter level (e.g. 5cm). The selection of threshold needs to balance control accuracy and execution efficiency. Too small threshold may lead to over-adjustment, while too large threshold may reduce the quality of task completion.

[0060] The final state is when the position deviation ≤ The second state information is assigned when the robot arm reaches the target position successfully and the current control task is completed. At this time, the robot arm control model stops generating subsequent action instructions and triggers a task end signal (such as resetting the robot arm or starting the next task process). Accurate determination of the termination state avoids meaningless redundant actions of the robot arm after reaching the target, thus improving control efficiency.

[0061] The non-terminal state is when the position deviation > The second state information is assigned when the robot arm is in the state of being ...

[0062] Through joint angle analysis and dynamic position deviation evaluation, efficient closed-loop management of the robot arm control task is achieved. The real-time coordinate calculation based on forward kinematics can accurately reflect the spatial position of the end effector and provide a reliable data basis for deviation analysis. The intelligent comparison of position deviation and preset deviation threshold makes the task termination judgment both strict and flexible, which not only avoids positioning deviation caused by premature termination, but also prevents resource waste caused by excessive iteration.

[0063] In an achievable embodiment of the present application, the generation of sample data can also be implemented in the following manner but is not limited to: performing five-tuple data generation processing based on the first state information and the second state information, action information, reshaping reward value and state identifier in the state information to obtain sample data.

[0064] In the embodiment of the present application, the five-tuple data generation process is a step of encoding the multi-source information generated during the interaction between the robot arm and the environment into sample data. The five-tuple consists of the first state information (s), the action information ( ), reshaping reward value (r), second state information (s′) and state flag (done).

[0065] Through the structured generation and storage of quintuple data, the training efficiency and strategy generalization ability of the robotic arm control model are significantly improved, enabling the robotic arm control model to learn the long-term value of actions from historical experience and avoid excessive reliance on real-time interactive data. The standardized format of quintuple data facilitates collaboration with mechanisms such as priority sampling and experience playback.

[0066] In an achievable embodiment of the present application, the importance setting process can be implemented in the following manner but is not limited to: setting an initial importance for the sample data in the sample pool, and averaging the reshaping reward values ​​of all the sample data in the sample pool to obtain an average reward value; performing parameter calculation processing on the sample data in the sample pool whose reshaping reward value is greater than or equal to the average reward value through a first preset algorithm to obtain a first importance parameter, and performing parameter calculation processing on the sample data whose reshaping reward value is less than the average reward value through a second preset algorithm to obtain a second importance parameter; obtaining the number of optimization times for the robot arm control model to perform model parameter optimization processing, and calculating the importance corresponding to the sample data based on the number of optimization times, the first importance parameter and the second importance parameter.

[0067] In the embodiment of the present application, regarding the setting of the importance of sample data in the sample pool, the present application provides a principle schematic diagram of a method for setting the importance, such as Figure 3 As shown, the intelligent agent is a robotic arm control model, the experience replay pool is a sample pool, initializing sample importance is to set the initial importance, updating sample importance and obtaining sample data is a process of calculating the importance corresponding to the sample data based on the number of optimizations, the first importance parameter and the second importance parameter.

[0068] The first preset algorithm and the second preset algorithm are pre-defined algorithms. The calculation of the first importance parameter and the second importance parameter can be determined by, but not limited to, formula (2): Formula (2) in, is the first important parameter, is the second most important parameter, is the reshape reward value greater than or equal to the average reward value, Reshape rewards that are less than the average reward, is the average reward value.

[0069] The number of optimizations is the number of iterations in which the robot control model has completed parameter updates. It is used to dynamically adjust the weight coefficient in the importance calculation. The process and principle of calculating the importance corresponding to the sample data according to the number of optimizations include but are not limited to the following methods: In the early stage of training, when the number of optimizations is small, in order to prevent successful samples from being overwhelmed, the importance of samples with reward values ​​greater than the average reward value is increased. In the middle and late stages of training, when the number of optimizations is large, in order to prevent falling into the local optimal strategy and increase sample diversity, the importance of samples with reward values ​​less than the average reward value is increased.

[0070] The learning efficiency and strategy robustness of the robot control model are significantly improved through the dynamic importance calculation mechanism. The hierarchical evaluation based on the average reward value can intelligently distinguish high-value samples from potential value samples, providing a quantitative basis for priority sampling. The weight coefficient adjustment driven by the number of optimizations enables the model to adaptively balance exploration and utilization at different training stages, quickly converge to a feasible strategy in the early stage, and refine and optimize the strategy details in the later stage. The uniform setting of the initial importance provides a fair exploration basis in the early stage of training, avoiding the solidification of early strategies due to priority deviations.

[0071] In an achievable embodiment of the present application, the importance corresponding to the sample data is calculated based on the number of optimizations, the first importance parameter and the second importance parameter, which can be implemented in but not limited to the following manner: when the number of optimizations is less than a preset number threshold, the importance of the sample data whose reshaping reward value is greater than or equal to the average reward value is calculated according to the first preset parameter and the first importance parameter; the importance of the sample data whose reshaping reward value is less than the average reward value is calculated according to the second preset parameter and the second importance parameter; when the number of optimizations is greater than or equal to the preset number threshold, the importance of the sample data whose reshaping reward value is greater than or equal to the average reward value is calculated according to the third preset parameter and the first importance parameter; the importance of the sample data whose reshaping reward value is less than the average reward value is calculated according to the fourth preset parameter and the second importance parameter, wherein the fourth preset parameter is greater than the second preset parameter.

[0072] In the embodiments of the present application, the preset number threshold is a custom-set threshold, which is a parameter for dividing the training phase, and is usually set according to task complexity and experience. For example: in a scenario where the total number of training times is 5000 times, the preset number threshold can be set to 2000 times, indicating that the first 2000 times are the initial training time, and the subsequent times are the middle and late training time. The setting of the threshold affects the calculation strategy of the sample importance, which needs to be verified through experiments to adapt to specific task requirements.

[0073] The first preset parameter ( ) and the second preset parameter ( ) is used to calculate sample importance. At this stage, the model focuses on quickly accumulating high-value experience, so it gives higher weights to high-reward samples. For example: set =0.8, =0.2, Specifically, the importance of sample data whose reshaping reward value is greater than or equal to the average reward value can be calculated by using, but not limited to, formula (3): Formula (3) The importance of sample data whose reshaping reward value is less than the average reward value can be calculated by using, but not limited to, formula (4): Formula (4) in, For importance.

[0074] Furthermore, regarding the application of the number of optimizations, for example, in the first 2000 rounds of training (i.e., the number of optimizations is less than 2000, 2000 is the preset number threshold), and Reshape the reward value to 0.8 and 0.2 respectively. The greater the importance The larger the value, the greater the remodeling reward value. The more important the sample data is, the more important it is to increase Value to 0.4, increase the reshape reward value less than The importance of sample data.

[0075] Through staged adjustment of importance parameters, both the training efficiency and strategy robustness of the robotic arm control model were improved.

[0076] In an achievable embodiment of the present application, data sampling processing of sample data can be implemented in the following manner but is not limited to: calculating the sampling probability corresponding to each sample data in the sample pool according to the importance; performing data sampling processing on the sample data in the sample pool in descending order of sampling probability to obtain sample data.

[0077] In the embodiment of the present application, the calculation of the sampling probability may be performed by, but not limited to, formula (5): Formula (5) in, is the sampling probability, is the number of sampled data.

[0078] Through dynamic probability allocation and intelligent sampling mechanism, the training efficiency and strategy generalization ability of the robotic arm control model are significantly improved.

[0079] In an achievable embodiment of the present application, after obtaining the state information, action information and state identification, the following method can also be used: determine whether the state identification is a termination state; if the state identification is a termination state, end the process of optimizing the model parameters of the robotic arm control model; if the state identification is a non-termination state, continue to optimize the model parameters of the robotic arm control model.

[0080] In an embodiment of the present application, in each round of training iteration, after the robotic arm control model executes an action and obtains environmental feedback, the state flag is first parsed. If the state flag is a terminal state (done=1), the current training cycle is terminated immediately, the optimized model parameters are saved, and the robotic arm and the environment are reset to the initial state to start a new round of training; if the state flag is a non-terminal state (done=0), data continues to be sampled from the sample pool, the parameters of the Critic and Actor networks are updated, and the next control step is entered.

[0081] Through intelligent judgment of the termination state and dynamic control of the training process, the training efficiency and resource utilization of the robotic arm control model are significantly improved.

[0082] In an achievable embodiment of the present application, the model parameter optimization processing of the robotic arm control model can be implemented in the following manner but is not limited to: determine the first evaluation parameter corresponding to the first evaluation network, the first target parameter corresponding to the first target network, the second evaluation parameter corresponding to the second evaluation network, and the second target parameter corresponding to the second target network in the robotic arm control model; the first target network performs action generation processing according to the second state information in the sampling data to obtain predicted action information, and the second evaluation network performs parameter generation processing according to the first state information and action information in the sampling data to obtain a first expected value; the second target network performs parameter generation processing according to the second state information and the predicted action information to obtain a second expected value, and performs parameter generation processing according to the second expected value and the reshaping reward value in the sampling data to obtain a target value; construct a first loss function according to the target value and the first expected value, construct a second loss function according to the first expected value, and optimize the model parameters of the robotic arm control model according to the first loss function and the second loss function to obtain an optimized robotic arm control model.

[0083] In the embodiment of the present application, the first evaluation network and the first target network are two instances of the Actor network, which respectively assume the roles of strategy generation and target value calculation. Among them, the first evaluation network (Actor evaluation network) generates a real-time action strategy according to the current state, and its parameters (the first evaluation parameter is denoted as θ) are updated through the policy gradient; the first target network (Actor target network) is a delayed mirror of the evaluation network, and its parameters (the first target parameter, denoted as θ′) are synchronized with θ through a soft update mechanism, which is used to generate stable target actions to reduce value estimation deviations.

[0084] The second evaluation network and the second target network are two instances of the Critic network, responsible for action value evaluation and target value calculation respectively. The second evaluation network (Critic evaluation network) predicts the Q value based on the current state and action, and its parameters (the second evaluation parameters, denoted as ) is updated by the mean square error loss; the second target network (Critic target network) is the delayed mirror of the evaluation network, and the parameters (the second target parameters, denoted as ) through soft update and Synchronization, used to calculate the target Q value to stabilize the training process.

[0085] The action generation process is performed by the first target network. In the parameter optimization phase, the first target network receives the second state information (s′) in the sampled data and generates the predicted action information ( ). This predicted action is used to evaluate the expected action value of the next state, avoiding the estimation fluctuation caused by directly using the evaluation network parameters.

[0086] The first expected value (Q) is calculated by the second evaluation network, whose input is the first state information (s) and action information ( ), the output is the predicted Q value (first expected value) of the current state-action pair, that is: Q( ), the first expected value indicates that in state Next action The expected long-term cumulative reward is the core evaluation indicator of the Critic network.

[0087] The second expected value ( ) is calculated by the second target network, whose input is the second state information (s′) and the predicted action generated by the first target network ( ), the output is the predicted Q value (second expected value) of the next state-action pair, that is: , the second expected value indicates that in the state Next action The expected long-term cumulative reward is used to construct the target Q value.

[0088] Target value ( ) is the benchmark value for critic network optimization, which is calculated by fusing the reshaped reward value (r) with the discounted second expected value. The target value can be calculated by, but not limited to, formula (6): Formula (6) in, is the discount factor (0≤ ≤1), which is used to balance immediate rewards and long-term benefits. If the state is marked as a terminal state (done=1), the target value only retains the current reward ( = ), Indicates the sampling data The reshaping reward value of data, Indicates the sampling data The second state information of the data, Indicates the sampling data The predicted action information of each data.

[0089] The first loss function (Critic loss) is the difference between the predicted Q value (first expected value (Q)) and the target value ( ), used to optimize the parameters of the second evaluation network , the first loss function can be calculated by formula (7): Formula (7) in, is the number of sampled data, Indicates the sampling data The first state information of the data, Indicates the sampling data The loss is minimized by the gradient descent algorithm, so that the Q value prediction of the Critic network gradually approaches the real action value distribution.

[0090] The second loss function (Actor loss) is the negative mean of the Q value evaluated by the Critic network, which is used to optimize the first evaluation network parameter θ. The second loss function can be calculated using formula (8): Formula (8) in, is the second loss function. By maximizing the Q value, the strategy generated by the Actor network is guided to the direction of high-value actions, thereby improving the task completion efficiency of the robot arm.

[0091] Through the dual network architecture and collaborative optimization mechanism, the policy stability and convergence efficiency of the robot arm control model are significantly improved. The introduction of the target network effectively alleviates the value estimation bias problem. For example, the delayed synchronization mechanism suppresses the drastic fluctuation of the Q value, making the training process smoother. The mean square error optimization of the Critic network and the policy gradient improvement of the Actor network form a closed-loop feedback, so that the action strategy gradually approaches the optimal solution under the guidance of value evaluation. The soft update mechanism avoids the risk of policy mutation caused by traditional hard updates through progressive parameter synchronization, ensuring the consistency and safety of the robot arm's actions in the real environment. In addition, the synergy of the dual loss functions enables the model to simultaneously optimize action value estimation and policy generation capabilities.

[0092] In an achievable embodiment of the present application, the model parameter optimization processing of the robot arm control model according to the first loss function and the second loss function can also be implemented in but not limited to the following manner: performing parameter optimization processing on the second evaluation parameter according to the first loss function to obtain the optimized second evaluation parameter, performing parameter optimization processing on the first evaluation parameter according to the second loss function to obtain the optimized first evaluation parameter; performing function transfer processing on the first target parameter according to the optimized first evaluation parameter to obtain the optimized first target parameter, performing function transfer processing on the second target parameter according to the optimized second evaluation parameter to obtain the optimized second target parameter; based on the optimized first evaluation parameter, the optimized first target parameter, the optimized second evaluation parameter and the optimized second target parameter, an optimized robot arm control model is obtained.

[0093] In an embodiment of the present application, the error gradient is back-propagated through a gradient descent algorithm (such as an Adam optimizer) according to the first loss function to adjust The value of makes the Q value prediction of the Critic network gradually approach the real action value distribution. The optimized second evaluation parameter ( 1) It can more accurately evaluate the long-term benefits of different actions. For example, in a robotic grasping task, it can accurately distinguish the difference in cumulative rewards between efficient paths and inefficient actions.

[0094] According to the second loss function, the policy gradient is back-propagated through the gradient ascent algorithm to adjust the value of θ so that the action strategy generated by the Actor network can obtain a higher Q value evaluation. The first evaluation parameter after optimization ( ) can generate better joint control instructions, for example: in the robot arm obstacle avoidance task, output smooth and safe trajectory adjustment.

[0095] Function transfer processing is a soft update mechanism for the target network parameters, which gradually synchronizes the optimization results of the evaluation network to the target network through weighted averaging. Specifically, it includes but is not limited to: The first target parameter ( ) Update: The first evaluation parameter after optimization 1 and the current first target parameter By weight factor (eg: 0.001) Fusion: , The first evaluation parameter is optimized so that the parameters of the Actor target network slowly approach the evaluation network to maintain the stability of strategy generation.

[0096] The second target parameter ( ) Update: The optimized second evaluation parameter 1 and the current second target parameter By the same factor Fusion: , The second evaluation parameter after optimization makes the Q-value prediction of the Critic target network change with delay, thus suppressing the fluctuation of value estimation.

[0097] Through step-by-step parameter optimization and soft synchronization mechanism, the training stability and strategy performance of the robotic arm control model are significantly improved.

[0098] In an achievable embodiment of the present application, when the first reward value is combined with the second reward value obtained by calculating the reward value based on visual perception incentive, the following method may also be used but is not limited to: performing information dimensionality reduction processing on the image information obtained by the robot arm control model during the robot arm control process to obtain a first image vector, and converting the first state information into a state vector and performing data fusion processing with the first image vector to obtain a first target vector; inputting the first target vector into a preset prediction model for prediction processing to obtain a second target vector, and performing prediction error calculation based on the second target vector and the first true vector converted from the second state information to obtain a first prediction error; inputting the second target vector into a preset prediction model for prediction processing to obtain a third target vector, and performing prediction error calculation based on the second true vector converted from the third state information to obtain a second prediction error, where the third state information is the next state information of the second state information; until a preset number of prediction errors are obtained, performing average value calculation processing based on the preset number of prediction errors to obtain an average prediction error, and combining the average prediction error with the external reward value of the robot arm control model during the robot arm control process to obtain a second reward value; combining the second reward value with the first reward value to obtain a reshaping reward value.

[0099] In the embodiments of the present application, in the actual application of the robot arm, unlike the simulation environment, simple joint information input into the algorithm cannot guarantee the effect. In actual scenarios, the state information input of the robot arm is mostly based on visual images, which is also the main medium for the interaction between the robot arm and the environment. However, taking the original high-dimensional perception information as input and outputting the corresponding control information will cause problems such as information redundancy and increased training complexity for the deep reinforcement learning control algorithm.

[0100] When deep reinforcement learning is applied in a real robot arm operation environment, in addition to the need for efficient environmental state representation, there is also the problem of reward mechanism. In the real environment, it is impossible to manually set intermediate rewards like in the simulation environment, or the reward setting is too complicated, resulting in excessive human intervention, which reduces the intelligence level of the system. Therefore, when the deep reinforcement learning method is directly applied to the real robot arm control environment, it often encounters the problem of missing external rewards or sparse rewards.

[0101] In order to facilitate understanding of the implementation process of the embodiment of the present application, the embodiment of the present application provides a schematic diagram of a visual perception incentive strategy, such as Figure 4 As shown, the state prediction network is a preset prediction model built into the intelligent agent (robotic arm control model), the preset prediction model is a custom-set convolutional neural network model, and the control module is a module in the intelligent agent (robotic arm control model) for controlling the robotic arm.

[0102] A convolutional neural network is used to compress the original image information into a low-dimensional, efficient space, represented as the first image vector At the same time, the state vector corresponding to the fusion state information To better capture the status information of objects and robotic arms during the operation and control of the robotic arms. The neural network structure is used as the state prediction model. And the action vector corresponding to the action information executed As input, predict the next state (the second target vector ), the real next state (the first real vector )and Calculate the first prediction error .

[0103] For the most recent continuous For each prediction model, the average prediction error needs to be calculated to characterize the quality of the prediction action. The average prediction error can be expressed by formula (9): Formula (9) in, is the average prediction error, is the preset quantity, is the number of prediction errors among the preset number of prediction error.

[0104] The reward obtained can be used as an incentive signal for the agent (robotic arm control model), prompting the agent to choose actions that can maximize the learning process, thereby learning the operation task more efficiently. In order to introduce this self-motivated reward into the robotic arm control model, it can be combined with external rewards and expressed by formula (10): Formula (10) in, is the external reward value, is the second reward value.

[0105] Through visual information fusion and prediction error-driven reward design, the performance of the robotic arm control model in complex scenes is significantly improved.

[0106] In summary, this application can achieve the following technical effects: A robot arm control method and device, electronic device and storage medium of the present application, in the process of the robot arm control model controlling the robot arm, performs reward value reshaping processing based on visual perception incentives to determine the reshaped reward value, which can reduce the reward sparsity, and generate sample data from various information generated by the robot arm control model in the robot arm control process, and then sets the importance of the sample data based on the reshaped reward value, and performs sampling according to the importance of the sample data, which improves the sampling efficiency compared with the uniform sampling in the prior art, and can solve the technical problems of how to improve the sampling efficiency of the algorithm in the robot arm control process and solve the reward sparsity of the algorithm, so as to achieve the technical effect of improving the sampling efficiency of the algorithm and solving the reward sparsity of the algorithm.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0108] The embodiment of the present application also provides a robot arm control device, Figure 5 A schematic diagram of the structure of a robotic arm control device provided for the application, such as Figure 5 As shown, including: An acquisition unit 51 is used to acquire state information, action information, state identification, and remodeling reward value of the robot arm generated in the process of controlling the robot arm based on the robot arm control model, wherein the robot arm control model is a model constructed based on a deep reinforcement learning algorithm, and the remodeling reward value is a reward value obtained by remodeling the reward value in combination with a method based on visual perception incentives; A generating unit 52, for generating sample data according to the state information, the action information, the reshaping reward value and the state identifier, and storing the sample data in a sample pool; A setting unit 53, configured to perform importance setting processing on sample data in the sample pool based on the reshaping reward value to obtain the importance corresponding to the sample data; The sampling unit 54 is used to perform data sampling processing on the sample data in the sample pool based on the importance of the sample data to obtain the sample data; The optimization unit 55 is used to optimize the model parameters of the robot control model according to the sampling data to obtain an optimized robot control model; The control unit 56 is used to control the robot arm according to the optimized robot arm control model.

[0109] In one embodiment of the present application, the acquisition unit 51 is further configured to: Determine target position information of a target position corresponding to the robot arm and first state information of the robot arm, wherein the target position is a position expected to be reached after the robot arm is controlled, and the first state information at least includes a current state of the robot arm before performing an action; According to the target position information, generating the motion information of the robot arm through the robot arm control model, wherein the motion information at least includes the motion performed by the robot arm; The robot arm control model controls the robot arm to perform an action according to the action information, and obtains second state information of the robot arm, wherein the second state information at least includes a current state of the robot arm after performing the action, and the state information includes the first state information and the second state information; Determine a position deviation between the robot arm and the target position according to the second state information and the target position information, and determine a state identifier according to the position deviation, wherein the state identifier is an identifier of the second state information; In the process of the robotic arm control model controlling the robotic arm to perform actions according to the action information, a reward value is generated every time a preset number of actions are performed until all actions in the action information are executed, and a first reward value is obtained. The first reward value is combined with a second reward value obtained by calculating the reward value based on visual perception incentives to obtain a reshaping reward value.

[0110] In one embodiment of the present application, the acquisition unit 51 is further configured to: Determine the current position coordinates corresponding to the robotic arm according to the joint angle information of the robotic arm included in the second state information; Performing coordinate comparison processing based on the current position coordinates and the target position coordinates of the target position included in the target position information to obtain a position deviation between the robot arm and the target position; When the position deviation is less than or equal to a preset deviation threshold, determining the state identifier of the second state information as a termination state; When the position deviation is greater than a preset deviation threshold, the state identifier of the second state information is determined as a non-terminal state.

[0111] In one embodiment of the present application, the generating unit 52 is further configured to perform five-tuple data generation processing according to the first state information and the second state information, the action information, the reshaping reward value and the state identifier in the state information to obtain sample data.

[0112] In one embodiment of the present application, the setting unit 53 is further configured to: Set the initial importance of the sample data in the sample pool, and average the reshape reward values ​​of all the sample data in the sample pool to obtain the average reward value; In the sample pool, sample data with a reshape reward value greater than or equal to the average reward value are subjected to parameter calculation processing by a first preset algorithm to obtain a first importance parameter, and sample data with a reshape reward value less than the average reward value are subjected to parameter calculation processing by a second preset algorithm to obtain a second importance parameter; The optimization times of the robot arm control model for model parameter optimization processing are obtained, and the importance corresponding to the sample data is calculated based on the optimization times, the first importance parameter and the second importance parameter.

[0113] In one embodiment of the present application, the setting unit 53 is further configured to: When the number of optimizations is less than a preset number threshold, the importance of sample data whose remodeling reward value is greater than or equal to the average reward value is calculated according to the first preset parameter and the first importance parameter; Calculate the importance of sample data whose remodeling reward value is less than the average reward value according to the second preset parameter and the second importance parameter; When the number of optimizations is greater than or equal to a preset number threshold, the importance of sample data whose remodeling reward value is greater than or equal to the average reward value is calculated according to the third preset parameter and the first importance parameter; The importance of the sample data whose reshaping reward value is less than the average reward value is calculated according to the fourth preset parameter and the second importance parameter, wherein the fourth preset parameter is greater than the second preset parameter.

[0114] In one embodiment of the present application, the sampling unit 54 is further used for: Calculate the sampling probability corresponding to each sample data in the sample pool according to the importance; According to the order of sampling probability from large to small, the sample data is sampled and processed in the sample pool to obtain the sampled data.

[0115] In one embodiment of the present application, the optimization unit 55 is further configured to: Determine whether the status indicator is a terminal state; If the state is marked as a termination state, the process of optimizing the model parameters of the robot control model is terminated; If the state is marked as a non-terminal state, the model parameters of the robot control model will continue to be optimized.

[0116] In one embodiment of the present application, the optimization unit 55 is further configured to: Determine a first evaluation parameter corresponding to the first evaluation network, a first target parameter corresponding to the first target network, a second evaluation parameter corresponding to the second evaluation network, and a second target parameter corresponding to the second target network in the robot arm control model; The first target network performs action generation processing according to the second state information in the sampled data to obtain predicted action information, and the second evaluation network performs parameter generation processing according to the first state information and action information in the sampled data to obtain a first expected value; The second target network performs parameter generation processing according to the second state information and the predicted action information to obtain a second expected value, and performs parameter generation processing according to the second expected value and the reshaping reward value in the sampled data to obtain a target value; A first loss function is constructed according to the target value and the first expected value, a second loss function is constructed according to the first expected value, and model parameters of the robotic arm control model are optimized according to the first loss function and the second loss function to obtain an optimized robotic arm control model.

[0117] In one embodiment of the present application, the optimization unit 55 is further configured to: Performing parameter optimization processing on the second evaluation parameter according to the first loss function to obtain an optimized second evaluation parameter, and performing parameter optimization processing on the first evaluation parameter according to the second loss function to obtain an optimized first evaluation parameter; Performing function transfer processing on the first target parameter according to the optimized first evaluation parameter to obtain the optimized first target parameter, and performing function transfer processing on the second target parameter according to the optimized second evaluation parameter to obtain the optimized second target parameter; Based on the optimized first evaluation parameter, the optimized first target parameter, the optimized second evaluation parameter and the optimized second target parameter, an optimized robotic arm control model is obtained.

[0118] In one embodiment of the present application, the acquisition unit 51 is further configured to: Performing information dimension reduction processing on image information acquired by the robot arm control model during the robot arm control process to obtain a first image vector, and converting the first state information into a state vector and performing data fusion processing with the first image vector to obtain a first target vector; Inputting the first target vector into a preset prediction model for prediction processing to obtain a second target vector, and performing prediction error calculation based on the second target vector and the first true vector converted from the second state information to obtain a first prediction error; The second target vector is input into a preset prediction model for prediction processing to obtain a third target vector, and a prediction error is calculated according to the second real vector converted from the third state information to obtain a second prediction error, where the third state information is the next state information of the second state information; until a preset number of prediction errors are obtained, average value calculation processing is performed according to the preset number of prediction errors to obtain an average prediction error, and the average prediction error is combined with an external reward value of the robot arm control model during the robot arm control process to obtain a second reward value; The second reward value is combined with the first reward value to obtain a reshaping reward value.

[0119] The description of the features in the embodiment corresponding to the robot arm control device can refer to the relevant description of the embodiment corresponding to the robot arm control method, which will not be repeated here.

[0120] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned robot arm control method embodiments.

[0121] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned robot arm control method embodiments when running.

[0122] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0123] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned robot arm control method embodiments are implemented.

[0124] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned robot arm control method embodiments are implemented.

[0125] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0126] The above is a detailed introduction to a robot arm control method and device, electronic device and storage medium provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A robot arm control method, characterized in that: include: Obtaining state information, action information, state identification, and remodeling reward value of the robotic arm generated in the process of controlling the robotic arm based on the robotic arm control model, wherein the robotic arm control model is a model constructed based on a deep reinforcement learning algorithm, and the remodeling reward value is a reward value obtained by remodeling the reward value in combination with a method based on visual perception incentives; Generate sample data according to the state information, the action information, the reshaping reward value and the state identifier, and store the sample data in a sample pool; Based on the remodeling reward value, the sample data in the sample pool is processed for importance setting to obtain the importance corresponding to the sample data, and based on the importance of the sample data, data sampling is processed on the sample data in the sample pool to obtain sample data; Performing model parameter optimization processing on the robotic arm control model according to the sampling data to obtain an optimized robotic arm control model; The robotic arm is controlled according to the optimized robotic arm control model.

2. The robot arm control method according to claim 1, characterized in that: The obtaining of the state information, action information, state identification, and reshaping reward value of the robot arm generated during the robot arm control process based on the robot arm control model includes: Determine target position information of a target position corresponding to the robotic arm and first state information of the robotic arm, wherein the target position is a position expected to be reached after the robotic arm is controlled, and the first state information at least includes a current state of the robotic arm before performing an action; According to the target position information, generating the action information of the robotic arm through the robotic arm control model, wherein the action information at least includes the action performed by the robotic arm; The robot arm control model controls the robot arm to perform an action according to the action information to obtain second state information of the robot arm, wherein the second state information at least includes a current state of the robot arm after performing the action, and the state information includes the first state information and the second state information; Determine a position deviation between the robot arm and the target position according to the second state information and the target position information, and determine the state identifier according to the position deviation, wherein the state identifier is an identifier of the second state information; In the process of the robotic arm control model controlling the robotic arm to perform an action according to the action information, a reward value is generated every time a preset number of actions are performed until all actions in the action information are executed, thereby obtaining a first reward value, and the first reward value is combined with a second reward value obtained by calculating the reward value based on visual perception incentives to obtain the reshaping reward value.

3. The robot arm control method according to claim 2, characterized in that: Determining the position deviation between the robot arm and the target position according to the second state information and the target position information, and determining the state identifier according to the position deviation includes: Determine the current position coordinates corresponding to the robotic arm according to the joint angle information of the robotic arm included in the second state information; Performing coordinate comparison processing according to the current position coordinates and the target position coordinates of the target position included in the target position information to obtain a position deviation between the mechanical arm and the target position; When the position deviation is less than or equal to a preset deviation threshold, determining the state identifier of the second state information as a termination state; When the position deviation is greater than a preset deviation threshold, the state identifier of the second state information is determined as a non-terminal state.

4. The robot arm control method according to claim 2, characterized in that: Generating sample data according to the state information, the action information, the remodeling reward value and the state identifier includes: A five-tuple data generation process is performed according to the first state information and the second state information in the state information, the action information, the reshaping reward value, and the state identifier to obtain the sample data.

5. The robot arm control method according to claim 1, characterized in that: The importance setting process of the sample data in the sample pool based on the remodeling reward value to obtain the importance corresponding to the sample data includes: Setting an initial importance for the sample data in the sample pool, and performing average processing on the reshaping reward values ​​of all the sample data in the sample pool to obtain an average reward value; In the sample pool, the sample data whose remodeling reward value is greater than or equal to the average reward value is subjected to parameter calculation processing by a first preset algorithm to obtain a first importance parameter, and the sample data whose remodeling reward value is less than the average reward value is subjected to parameter calculation processing by a second preset algorithm to obtain a second importance parameter; The number of optimizations of the robot arm control model for model parameter optimization processing is obtained, and the importance corresponding to the sample data is calculated based on the number of optimizations, the first importance parameter, and the second importance parameter.

6. The robot arm control method according to claim 5, characterized in that: The calculating the importance corresponding to the sample data based on the number of optimizations, the first importance parameter, and the second importance parameter comprises: When the number of optimizations is less than a preset number threshold, calculating the importance of the sample data whose remodeling reward value is greater than or equal to the average reward value according to the first preset parameter and the first importance parameter; Calculating the importance of the sample data whose remodeling reward value is less than the average reward value according to the second preset parameter and the second importance parameter; When the number of optimizations is greater than or equal to the preset number threshold, calculating the importance of the sample data whose remodeling reward value is greater than or equal to the average reward value according to a third preset parameter and the first importance parameter; The importance of the sample data whose reshaping reward value is less than the average reward value is calculated according to a fourth preset parameter and the second importance parameter, wherein the fourth preset parameter is greater than the second preset parameter.

7. The robot arm control method according to claim 1, characterized in that: The data sampling process is performed on the sample data in the sample pool based on the importance of the sample data to obtain the sample data, which includes: Calculate the sampling probability corresponding to each sample data in the sample pool according to the importance; According to the order of the sampling probabilities from large to small, data sampling processing is performed on the sample data in the sample pool to obtain the sample data.

8. The robot arm control method according to claim 1, characterized in that: After obtaining the state information, action information, state identification, and reshaping reward value of the robotic arm generated during the robotic arm control process based on the robotic arm control model, the method further includes: Determine whether the status indicator is a terminated state; If the state is marked as a termination state, the process of performing model parameter optimization processing on the robot arm control model is terminated; If the state is identified as a non-terminal state, the model parameter optimization process of the robot arm control model continues.

9. The robot arm control method according to claim 2, characterized in that: The performing model parameter optimization processing on the robot arm control model according to the sampling data to obtain the optimized robot arm control model comprises: Determine a first evaluation parameter corresponding to the first evaluation network, a first target parameter corresponding to the first target network, a second evaluation parameter corresponding to the second evaluation network, and a second target parameter corresponding to the second target network in the manipulator control model; The first target network performs action generation processing according to the second state information in the sampled data to obtain predicted action information, and the second evaluation network performs parameter generation processing according to the first state information and the action information in the sampled data to obtain a first expected value; The second target network performs parameter generation processing according to the second state information and the predicted action information to obtain a second expected value, and performs parameter generation processing according to the second expected value and the reshaping reward value in the sampled data to obtain a target value; A first loss function is constructed according to the target value and the first expected value, a second loss function is constructed according to the first expected value, and model parameters of the robotic arm control model are optimized according to the first loss function and the second loss function to obtain the optimized robotic arm control model.

10. The robot arm control method according to claim 9, characterized in that: The performing model parameter optimization processing on the robotic arm control model according to the first loss function and the second loss function to obtain the optimized robotic arm control model comprises: Performing parameter optimization processing on the second evaluation parameter according to the first loss function to obtain an optimized second evaluation parameter, and performing parameter optimization processing on the first evaluation parameter according to the second loss function to obtain an optimized first evaluation parameter; Performing function transfer processing on the first target parameter according to the optimized first evaluation parameter to obtain the optimized first target parameter, and performing function transfer processing on the second target parameter according to the optimized second evaluation parameter to obtain the optimized second target parameter; The optimized robotic arm control model is obtained based on the optimized first evaluation parameter, the optimized first target parameter, the optimized second evaluation parameter and the optimized second target parameter.

11. The robot arm control method according to claim 2, characterized in that: The combining of the first reward value and the second reward value obtained by calculating the reward value based on visual perception incentive to obtain the reshaping reward value includes: Performing information dimension reduction processing on image information acquired by the robot control model during the robot control process to obtain a first image vector, and converting the first state information into a state vector and performing data fusion processing with the first image vector to obtain a first target vector; Inputting the first target vector into a preset prediction model for prediction processing to obtain a second target vector, and performing prediction error calculation based on the second target vector and the first true vector converted from the second state information to obtain a first prediction error; The second target vector is input into the preset prediction model for prediction processing to obtain a third target vector, and a prediction error is calculated according to the second real vector converted from the third state information to obtain a second prediction error, wherein the third state information is the next state information of the second state information; until a preset number of prediction errors are obtained, average value calculation processing is performed according to the preset number of prediction errors to obtain an average prediction error, and the average prediction error is combined with an external reward value of the robot arm control model during the robot arm control process to obtain the second reward value; The second reward value is combined with the first reward value to obtain the reshaping reward value.

12. A robot arm control device, characterized in that: include: An acquisition unit, used to acquire state information, action information, state identification, and remodeling reward value of the robotic arm generated in the process of controlling the robotic arm based on the robotic arm control model, wherein the robotic arm control model is a model constructed based on a deep reinforcement learning algorithm, and the remodeling reward value is a reward value obtained by remodeling the reward value in combination with a method based on visual perception incentives; A generating unit, configured to generate sample data according to the state information, the action information, the reshaping reward value and the state identifier, and store the sample data in a sample pool; A setting unit, configured to perform importance setting processing on the sample data in the sample pool based on the remodeling reward value to obtain the importance corresponding to the sample data; A sampling unit, configured to perform data sampling processing on the sample data in the sample pool based on the importance of the sample data to obtain sample data; An optimization unit, used for performing model parameter optimization processing on the robotic arm control model according to the sampling data to obtain an optimized robotic arm control model; A control unit is used to control the robotic arm according to the optimized robotic arm control model.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the robot arm control method as claimed in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the robot arm control method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the robot arm control method according to any one of claims 1 to 11 are implemented.

Images