Mechanical arm visual servo control method and equipment based on target condition reinforcement learning

Abstract

The invention discloses a mechanical arm visual servo control method and equipment based on target condition reinforcement learning, and belongs to the technical field of mechanical arm control. A mechanical arm environment is built, a camera is additionally arranged at the tail end of a mechanical arm, a computer is used for designing a safety reinforcement learning model, then online learning is carried out according to a depth deterministic strategy gradient algorithm and a post-experience playback algorithm, and the mechanical arm environment is matched. The position of the feature point in the camera picture serves as input to control the mechanical arm to move and lock the two-dimensional code attached to the surface of the object, and finally the visual servo task is completed. The method is simple in step, low in environment requirement and rapid in response of the mechanical arm, and compared with a traditional control method based on pseudo-inverse, the method has remarkable effect improvement in the aspects of the number of iterations and robustness.

Description

Technical Field

[0001] The present invention relates to the field of visual servo control, and specifically to a robotic arm visual servo control method and device based on target-conditioned reinforcement learning. Background Art

[0002] Visual servo control plays a crucial role in the field of robotic arms, aiming to precisely control the movement of the robotic arm through visual feedback so that it can complete complex tasks such as grasping, assembly, and welding. Most traditional visual servo control methods are based on fixed control strategies and predefined rules.

[0003] In the prior art, when the robotic arm faces complex and changeable task scenarios and environments, it often lacks flexibility and adaptability. Since it is unable to dynamically adjust the control strategy according to real-time visual information and task objectives, when the position and posture of the target object change significantly, or there are interference factors in the environment, the control effect will be greatly reduced, and even the task cannot be completed. At the same time, traditional methods require a large amount of manual design and debugging, with high development costs and low efficiency.

[0004] The prior art with the publication number CN113146616A discloses a four-degree-of-freedom robotic arm visual servo control method. First, the overall arm mode of the robotic arm is set to the visual servo mode. In each control cycle of the visual servo mode, it is judged whether the visual measurement pose data is valid. If it is invalid for multiple consecutive cycles, the robotic arm stops moving, the overall arm mode is converted to the servo standby mode, and the joint control mode is converted to the position servo mode. In each control cycle of the visual servo mode, if the visual measurement pose data is valid and the joint control mode is in the speed control mode, the planned end four-dimensional velocity VW_POR is calculated and output; then the planned joint angular velocity and planned joint angular position are obtained through inverse kinematics and output as control commands to control the angular velocity and angular position of the joint in the next control cycle. This method requires obtaining a large amount of parameter information and has a large amount of calculation, and the reaction time of the robotic arm is long. Summary of the Invention

[0005] Object of the Invention: Aiming at the above-mentioned prior art, a robotic arm visual servo control method and device based on target-conditioned reinforcement learning are proposed; in a complex and changeable scenario, this method uses target-conditioned reinforcement learning to dynamically generate a control strategy according to real-time visual information and task objectives, improving the flexibility and adaptability of visual servo control, reducing the cost of manual design and debugging, and being able to show good performance in an environment with interference factors.

[0006] Technical Solution: To achieve the above technical objectives, for a robotic arm visual servo control method based on target-conditioned reinforcement learning of the present invention, a camera is set at the end of the robotic arm and a robotic arm environment is built. The steps are as follows:

[0007] S1. Set the pixel points of four known coordinates in the image captured by the robotic arm camera as fixed target points;

[0008] S2. Set a QR code on the object as the recognition target for the robotic arm visual servo task, and the pose of the recognition target is fixed after initialization;

[0009] S3. The robotic arm obtains four moving feature points by capturing the QR code with the camera, and uses the safety reinforcement learning model to set the pixel coordinates of the moving feature points and the fixed target points as the achieved target and the desired target in the target-conditioned reinforcement learning respectively;

[0010] S4. The safety reinforcement learning model includes the Deep Deterministic Policy Gradient algorithm DDPG, the Lagrange constraint, the Hindsight Experience Replay algorithm HER, and the safety critic network safety-critic. Among them, the DDPG algorithm includes the actor network and the critic network;

[0011] S5. After training the safety reinforcement learning model with the historical interaction data of the robotic arm environment, directly input the current state and the desired target into the actor network in DDPG. The safety reinforcement learning model can directly output the joint target velocity action of the robotic arm through the information of the recognition target captured by the camera, control the robotic arm to lock the target with the QR code, and control the movement of the robotic arm to move the moving feature points to the pixel coordinates of the four fixed target points in the captured image, completing the visual servo control task.

[0012] Furthermore, before the robotic arm camera captures the fixed target points, it is necessary to first determine the camera image orientation, and the method is as follows:

[0013] The image orientation of the camera is the vector starting from the center of the captured image and pointing upward in the camera coordinate system. The camera image orientation is always fixed in the camera coordinate system, denoted as ;

[0014] The camera image orientation will change with the movement of the robotic arm in the world coordinate system. By calculating the representation of the camera image orientation in the world coordinate system to ensure the correct refresh of the camera image; Obtain the relative pose information of each joint of the robotic arm. The calculation process of the vector is as follows:

[0015] ;

[0016] In the above formula represents the rotation matrix of the coordinate system rotating around the axis by angle, represents the rotation matrix of the coordinate system rotating around the axis by The rotation matrix of an angle indicating that the coordinate system rotates around axis by the rotation matrix of the angle;

[0017] ;

[0018] The above formula is the rotation matrix between the joint coordinate systems of each robotic arm, where is the joint coordinate system relative to the joint coordinate system's rotation matrix, is the joint 's rotation angle, respectively represent the world coordinate system, the end effector coordinate system of the robotic arm, and the camera coordinate system;

[0019] The representation of the camera image direction in the world coordinate system is as follows:

[0020] ;

[0021] In the formula, is the rotation matrix of the camera coordinate system relative to the end effector coordinate system of the robotic arm, represents the rotation matrix of the end effector coordinate system of the robotic arm relative to the robotic arm joint 8 coordinate system, represents the representation of the camera image direction in the camera coordinate system.

[0022] Furthermore, the safety reinforcement learning model performs HER operations and normalization operations on the robotic arm environment historical interaction data: actions, states, rewards, achieved goals, and desired goals to complete preprocessing, reconstructs the reward data to alleviate the reward sparsity problem of this robotic arm environment; constructs a safety-critic network using a cost function, inputs the robotic arm environment historical interaction cost data stored in the replay buffer into the safety-critic network, and inputs the robotic arm environment historical interaction reward data in the replay buffer into the critic network critic in the DDPG network; uses the regularization term and Lagrange constraint term in the actor network actor in the DDPG network to restrict the robotic arm movement actions to ensure the safety of the learning process, and realizes the Lagrange constraint by introducing the output value of the safety-critic network into the actor network in the DDPG network;

[0023] Among them, the actor network, critic network, and safety-critic network of DDPG all adopt a fully connected structure, including an input layer, a hidden layer, and an output layer; the actor network takes the state of the robotic arm environment and the desired goal As input, an action vector is output. The action vector forms a regularization term loss through the L2 norm regularization. The safety-critic network takes the state and action of the robotic arm environment as input and outputs a cost value, which forms a cost value term loss. The critic network takes the state, action, and expected goal of the robotic arm environment as input and outputs an action value, which forms an action value term loss with the constant -1. Finally, the Adam optimizer updates the actor network parameters according to the gradients calculated from the actor network loss 1, actor network loss 2, and actor network loss 3, so as to control the robotic arm environment to complete the visual servo task as soon as possible on the premise of ensuring that the QR code does not exceed the camera screen.

[0024] Furthermore, the method for detecting the moving feature points of the QR code is as follows:

[0025] Suppose the state space of the robotic arm environment contains four moving feature points. The pixel coordinates of the four moving feature points are obtained by detecting through the open-source moving feature point detection function cv2.aruco.detectMarkers() of opencv;

[0026] The pixel coordinates of the four moving feature points of the QR code are represented as an 8-dimensional target space: , where is the pixel coordinate of the th feature point. This target space represents the moving range space of the four detected moving feature points of the QR code in the image. The achieved goal and the expected goal at time t in the target conditional reinforcement learning are both in this target space. is designed in the robotic arm environment as the pixel coordinates of the four moving feature points of the QR code at the current moment . is specified as the pixel coordinates of four fixed target points that match the moving feature points in the camera screen during environment initialization.

[0027] Furthermore, the robotic arm environment uses a seven-axis robotic arm. The state and action of the robotic arm environment are as follows:

[0028] The state of the robotic arm environment is represented by a state space, including: the position and velocity of the end effector of the robotic arm, the gripper width, the seven joint angles and velocities, and the achieved goal range space. The robotic arm state in this range space is used as the input of the actor network, critic network, and safety-critic network to complete the training of the safety reinforcement learning model. This robotic arm state space is a 29-dimensional vector, represented as:

[0029] ;

[0030] Among them, represents the information at the end of the seven-axis robotic arm: end position, end velocity, and end gripper width, a total of 7 dimensions; represents the joint angles of the seven-axis robotic arm, a total of 7 dimensions; represents the joint rotational speeds of the seven-axis robotic arm, a total of 7 dimensions; represents the pixel coordinates of the four moving feature points of the QR code, a total of 8 dimensions;

[0031] The actions in the robotic arm environment are identified in the action space, that is, the range space of the movement speeds of the seven joints and one gripper of the robotic arm. In the action space of the robotic arm environment, the actions are used as the inputs of the actor network, critic network, and safety-critic network to complete the training of the safety reinforcement learning model. The action space of this robotic arm environment is an 8-dimensional vector, expressed as:

[0032] ;

[0033] which includes the target rotational speeds of the joints of the robotic arm , and the target speed of the end gripper movement .

[0034] Furthermore, the safety reinforcement learning model generates rewards for the historical interaction data in the robotic arm environment through a reward function. The reward function belongs to a sparse reward function:

[0035] ;

[0036] In the above formula , where is the achieved target at time , is the expected target at time , that is, the Euclidean distance between the two, is the hyperparameter distance threshold. Only when the distance between the feature points in the image and the target point is less than the threshold , will a reward of 0 be obtained, otherwise a reward of -1 will be obtained; this kind of reward is sparse, and the HER algorithm is used to preprocess the data to alleviate the sparse reward and increase the sample efficiency;

[0037] The safety-critic network inputs the state and actions of the robotic arm to evaluate the danger level of the robotic arm's actions. Among them, the cost function:

[0038] ;

[0039] In the formula, is A flag variable indicating whether the environment terminates at a certain moment. When the environment terminates, that is, when the QR code on the object surface goes beyond the camera screen, the robotic arm is given an environmental cost of 0 to punish it for the action that causes the environment to terminate at that moment. Otherwise, a cost of -1 is given.

[0040] Furthermore, the specific processes of HER operation and normalization operation are as follows:

[0041] (1) HER operation:

[0042] 1) Select a trajectory data generated by the interaction between the robotic arm and the environment from the replay buffer. The trajectory data is the interaction data containing multiple moments recorded from t = 0 until the end of the interaction with the environment;

[0043] 2) Randomly select moments and in this trajectory data, and obtain the corresponding interaction data at these moments as and respectively, where represents the robotic arm state, represents the action, represents the reward function, represents the desired goal, represents the achieved goal;

[0044] 3) Replace in the data at moment with in the data at moment, that is, ;

[0045] 4) Recalculate the reward at moment according to the reward function ;

[0046] 5) The data after completing the HER operation is ;

[0047] (2) Normalization operation:

[0048] The robotic arm state contains various types of information such as the three-dimensional position, speed, gripper width, and pixel coordinates of the robotic arm. Use the following formula to perform normalization operations on the state and the desired goal , and splice the robotic arm state with the desired goal :

[0049] , ;

[0050] In the formula, is the state of the robotic arm, is the desired target, are the mean and standard deviation of the robotic arm state, are the mean and standard deviation of the desired target, and the concatenation operation is in front, behind.

[0051] Furthermore, the training process of the safety reinforcement learning model is as follows:

[0052] The safety-critic network uses cost information to construct an assessment of the danger level of actions. The target policy after introducing cost is as follows:

[0053] ;

[0054] where are the time discount factor of the reward and the cost threshold respectively, are the policy state, action, reward function, cost, and the distribution of state and action generated by the policy respectively; the meaning of this target policy is that under the condition of meeting the safety constraints, that is, the environmental steps where the QR code remains in the camera screen exceed the corresponding threshold , the expectation of the cumulative reward of the robotic arm is maximized; the loss function of the safety-critic network is as follows:

[0055] ;

[0056] where the data is sampled from the replay buffer , and is composed of the state, action, cost, desired target of the robotic arm at the current moment and the state at the next moment, is the time discount factor of the cost, and the policy is the deterministic policy of the DDPG network, which takes the state and the desired target as inputs and outputs a deterministic action; and are the target network parameters and network parameters of the safety-critic network, which are updated by the way of soft update; the output of the safety-critic network is the cost value , whose input is the state and action, representing the assessment of the danger of the action, and completes the assessment of the danger of the robotic arm action causing the QR code to exceed the screen; the cost value output by the safety-critic network is added to the update of the actor network of the DDPG network to ensure that the update of the actor network considers the danger of the action.

[0057] Furthermore, based on the update methods of the critic network and the actor network in the DDPG network, a regularization term and a safety-critic network are added to the actor network. The specific content is as follows:

[0058] (1)Update of the critic network:

[0059] ;

[0060] where is the reward at the current moment, and are the parameters of the critic network and the critic target network respectively. The output of the critic network is the action value , whose input is the state, action, and expected target, representing the estimation of the action value;

[0061] (2)Update of the actor network. In the original DDPG, the update function of the actor network only contains the action value term . The action regularization term and the cost value term are modified and added to ensure that the update of the actor network takes into account the amplitude and danger of the action. The specific representation is as follows:

[0062] ;

[0063] where is the parameter of the actor network, is the deterministic action generated by the actor network with the state and the expected target as the input. is the L2 norm regularization term for the action generated by the actor network. are the weight parameters of the action regularization term and the cost value term respectively. The loss function of the actor network includes three parts: the action value term output by the critic network, the action regularization term, and the cost value term output by the safety-critic network. The action value term ensures that the update of the actor network parameters moves in the direction of increasing the action value. The action regularization term ensures that the amplitude of the action output by the actor network is reduced. The cost value term ensures that the update of the actor network parameters moves in the direction of reducing the cost value.

[0064] A computer device includes a processor and a memory. The processor is electrically connected to the memory. The memory is used to store instructions and data. The processor is used to execute the method for realizing robotic arm visual servo control based on goal-conditioned reinforcement learning.

[0065] Beneficial effects: When the present invention is applied to complex and variable task scenarios and environments, it can effectively improve the flexibility and adaptability of the robotic arm. When the position and posture of the target object change significantly and there are a large number of environmental interference factors, target-conditioned reinforcement learning is utilized to dynamically generate a control strategy based on real-time visual information and task objectives, improving the flexibility and adaptability of visual servo control, and still being able to maintain the control effect of the robotic arm. The present invention can effectively reduce the development cost and the cost of manual design and debugging, and has wide practicability. Description of the Drawings

[0066] Figure 1 FIG. is a schematic flow chart of controlling the movement of the robotic arm by the robotic arm visual servo control method based on target-conditioned reinforcement learning of the present invention;

[0067] Figure 2 FIG. is a model training flow chart of the robotic arm visual servo control method based on target-conditioned reinforcement learning in an embodiment of the present invention;

[0068] Figure 3 FIG. is a model structure diagram when the actor network in the DDPG network used in the present invention is updated;

[0069] Figure 4 FIG. is a movement diagram of four moving feature points collected by a camera when the robotic arm moves in an embodiment of the present invention;

[0070] Figure 5 FIG. is a curve diagram of the joint angles of the robotic arm when the robotic arm moves in an embodiment of the present invention;

[0071] Figure 6 FIG. is a schematic diagram of the structure of the robotic arm in an embodiment of the present invention;

[0072] Figure 7 FIG. is a movement diagram of four moving feature points collected by a camera when the robotic arm is under environmental interference in an embodiment of the present invention;

[0073] Figure 8 FIG. is a curve diagram of the joint angles of the robotic arm when the robotic arm is under environmental interference in an embodiment of the present invention. Detailed Embodiments

[0074] The following further explains the present invention with reference to the accompanying drawings.

[0075] As Figure 1As shown in the figure, the present invention discloses a robotic arm visual servo control method based on goal-conditioned reinforcement learning: designing the traditional visual servo task into a goal-conditioned reinforcement learning task, that is, training the robotic arm to move the feature points in the camera (the camera is fixed at the end of the robotic arm) to the fixed target points, where the pixel coordinates of the feature points and the fixed target points are respectively set as the achieved goal and the desired goal, and using the Deep Deterministic Policy Gradient (DDPG) algorithm and the Hindsight Experience Replay (HER) algorithm for online learning. To avoid the problem of low learning efficiency caused by the feature points exceeding the camera's field of view during the training process, a regularization term and a Lagrangian constraint term are introduced to limit the actions to achieve the safety of the training process. At the same time, a visual servo control simulation platform is built with the robotic arm as the research object to verify the algorithm, as shown in the appendix Figure 6 As shown in the figure, the algorithm uses the position of the feature points in the camera image as the input to control the movement of the robotic arm and lock the two-dimensional code attached to the object surface to complete the visual servo task. The results show that compared with the traditional pseudo-inverse-based control method, the method of the present invention has a significant improvement in the number of iterations and robustness.

[0076] S1. Set any four pixel points with known coordinates in the image captured by the robotic arm camera as the fixed target points;

[0077] S2. Set a two-dimensional code on the object as the recognition target of the robotic arm visual servo task, and the pose of the recognition target is fixed after initialization;

[0078] S3. The robotic arm obtains four moving feature points by capturing the two-dimensional code with the camera, and controls the movement of the robotic arm to make the moving feature points move to the pixel coordinates of the four fixed target points in the captured image, thereby controlling the robotic arm to achieve the visual servo task;

[0079] The method for detecting the moving feature points of the two-dimensional code is as follows: assume that the state space of the two-dimensional code of the robotic arm environment contains four moving feature points, and the pixel coordinates of the four moving feature points are obtained by detecting with the open-source moving feature point detection function cv2.aruco.detectMarkers() of opencv; during training, it is necessary to limit the actions of the robotic arm to ensure that the four moving feature points of the two-dimensional code in the camera image are always clear. If the two-dimensional code is incomplete or not fully displayed, the moving feature point detection function will automatically report an error and the current training process will be forced to terminate;

[0080] Represent the pixel coordinates of the four moving feature points of the two-dimensional code as an 8-dimensional target space goal: , where is the The pixel coordinates of the feature points. The target space represents the movement range space of the four moving feature points of the detected QR code in the screen. The achieved goal at time t in the target-conditioned reinforcement learning and the desired goal are both in this target space. In the robotic arm environment, it is designed as the current moment The pixel coordinates of the four moving feature points of the QR code are specified as the pixel coordinates of the four fixed target points that match the moving feature points in the camera screen during environment initialization.

[0081] The robotic arm environment uses a seven-axis robotic arm. The state of the robotic arm environment is represented by a state space, including: the position and velocity of the end effector of the robotic arm, the width of the gripper, the seven joint angles and velocities, and the range space of the achieved goal. The robotic arm state within this range space is used as the input for the actor network, critic network, and safety-critic network to complete the training of the safety reinforcement learning model. This robotic arm state space is a 29-dimensional vector, expressed as:

[0082] ;

[0083] Among them, represents the information of the end effector of the seven-axis robotic arm: the end position, end velocity, and end gripper width, a total of 7 dimensions; represents the joint angles of the seven-axis robotic arm, a total of 7 dimensions; represents the joint rotation speeds of the seven-axis robotic arm, a total of 7 dimensions; represents the pixel coordinates of the four moving feature points of the QR code, a total of 8 dimensions;

[0084] The actions in the robotic arm environment are identified by an action space, that is, the range space of the movement speeds of the seven joints and one gripper of the robotic arm. In the action space of the robotic arm environment, the actions are used as the input for the actor network, critic network, and safety-critic network to complete the training of the safety reinforcement learning model. The action space of this robotic arm environment is an 8-dimensional vector, expressed as:

[0085] ;

[0086] which includes the target rotation speeds of the joints of the robotic arm , and the target speed of the end gripper movement ;

[0087] S4. Set the pixel coordinates of the moving feature points and the fixed target points as the achieved goal ag and the desired goal dg in the target-conditioned reinforcement learning respectively;

[0088] The rewards in the historical interaction data of the robotic arm environment for the safety reinforcement learning model are generated through a reward function, and the reward function belongs to a sparse reward function.

[0089] ;

[0090] In the above formula , where is the achieved goal at time is the expected goal at time That is, the Euclidean norm distance between the two. is the hyperparameter distance threshold. Only when the distance between the feature points in the picture and the target point is less than the threshold , a reward of 0 will be obtained; otherwise, a reward of -1 will be obtained. This kind of reward is sparse, and the HER algorithm is used to preprocess the data to alleviate the sparse reward and increase the sample efficiency.

[0091] The safety-critic network inputs the state and action of the robotic arm and evaluates the danger level of the robotic arm's action. The cost function is:

[0092] ;

[0093] In the formula, is a flag variable indicating whether the environment terminates at time . When the environment terminates, that is, the QR code on the object surface exceeds the camera screen, a cost of 0 is given to the robotic arm environment to punish its action that causes the environment to terminate at that moment; otherwise, a cost of -1 is given.

[0094] S5. Build a safety reinforcement learning model. The safety reinforcement learning model includes a Deep Deterministic Policy Gradient algorithm (DDPG) network, a Lagrange constraint, a Hindsight Experience Replay (HER) algorithm, and a safety-critic network. The DDPG algorithm includes an actor network and a critic network.

[0095] The safety reinforcement learning model performs HER operation and normalization operation on the historical interaction data of the robotic arm environment: actions, states, rewards, achieved goals, and desired goals to complete preprocessing, and reconstructs the reward data to alleviate the reward sparsity problem of the robotic arm environment; inputs the historical interaction cost data of the robotic arm environment stored in the buffer into the safety-critic network, and inputs the historical interaction reward data in the buffer into the critic network in DDPG; uses the regularization term and Lagrange constraint term in the actor network in DDPG to restrict the movement actions of the robotic arm to ensure the safety of the learning process, and realizes the Lagrange constraint by introducing the output value of the safety-critic network into the actor network in the DDPG network;

[0096] Among them, the actor network, critic network, and safety-critic network in the DDPG network all adopt a fully connected structure, including an input layer, a hidden layer, and an output layer. The hidden layers of the actor network, critic network, and safety-critic network all adopt a structure of 256*4. The actor network takes the state (29 dimensions), and the desired goal (8 dimensions) as inputs, and outputs an 8-dimensional action vector. The action vector forms the first item of the actor network loss through L2 regularization and the regularization term weight . The safety-critic network takes the state (29 dimensions), and the action (8 dimensions) as inputs, and outputs the cost value (1 dimension). The cost value and the corresponding cost weight form the second item of the actor network loss . The critic network takes the state (29 dimensions), the action (8 dimensions), and the desired goal (8 dimensions) as inputs, outputs the action value, and forms the third item of the actor network loss with the constant -1; finally, uses the Adam optimizer to update the actor network parameters according to the gradient calculated by the loss, and realizes controlling the robotic arm to complete the visual servo task as soon as possible on the premise of ensuring that the QR code does not exceed the screen, as Figure 3 shown;

[0097] The training process of the safety reinforcement learning model is as follows: The safety-critic network constructs an evaluation of the danger degree of actions using cost information . The target policy after introducing the cost is as follows:

[0098] ;

[0099] wherein are respectively the time discount factor of the reward and the cost threshold, are respectively the policy, state, action, reward function, cost, and the distribution of state and action generated by the policy; the meaning of this target policy is that under the condition of meeting the safety constraint, that is, the number of environmental steps when the QR code remains in the camera screen exceeds the corresponding threshold , the expected maximum cumulative reward of the robotic arm; the loss function of the safety-critic network is as follows:

[0100] ;

[0101] where the data is sampled from the replay buffer , and is composed of the state, action, cost, expected target of the robotic arm at the current moment, and the state at the next moment, is the time discount factor of the cost, and the policy is the deterministic policy of the DDPG network, taking the state and the expected target as inputs and outputting a deterministic action; and are the target network parameters and network parameters of the safety-critic network, and are updated in a soft update manner; the output of the safety-critic network is the cost value , whose input is the state and action, representing the evaluation of the danger of the action, and completing the evaluation of the danger that the action of the robotic arm causes the QR code to exceed the screen; the cost value output by the safety-critic network is added to the update of the actor network of the DDPG network to ensure that the update of the actor network considers the danger of the action;

[0102] Based on the update methods of the critic network and the actor network in the DDPG network, a regularization term and a safety-critic network are added to the actor network, and the specific content is as follows:

[0103] (1) Update of the critic network:

[0104] ;

[0105] wherein is the reward at the current moment, and are respectively the parameters of the critic network and the critic target network, and the output of the critic network is the action value , whose input is the state, action, and expected target, representing the estimation of the action value;

[0106] (2) Update of the actor network. In the original DDPG, the update function of the actor network only contains the action value term , modify and add the action regularization term and the cost value term to ensure that the update of the actor network takes into account the amplitude and risk of the action. After modification, it is as follows:

[0107] ;

[0108] where are the actor network parameters, is the deterministic action generated by the actor network with the state and the desired target as inputs. is the L2 norm regularization term for the action generated by the actor network. are the weight parameters of the action regularization term and the cost value term respectively. The actor network loss function includes three parts: the action value term output by the critic network, the action regularization term, and the cost value term output by the safety - critic network. The action value term ensures that the update of the actor network parameters moves in the direction of increasing the action value. The action regularization term ensures that the amplitude of the action output by the actor network is reduced. The cost value term ensures that the update of the actor network parameters moves in the direction of reducing the cost value;

[0109] S6. After the safety reinforcement learning model is trained, directly input the current state and the desired target into the actor network. The safety reinforcement learning model directly outputs the target speed action of the robotic arm joints to control the robotic arm to lock the target with the QR code and complete the visual servo control task.

[0110] A computer - readable storage medium stores a computer program, and the computer program is adapted to be loaded and executed by a processor to perform the robotic arm visual servo control method based on target - conditional reinforcement learning.

[0111] A computer device includes a processor and a memory. The processor is electrically connected to the memory. The memory is used to store instructions and data, and the processor is used to execute the robotic arm visual servo control method based on target - conditional reinforcement learning.

[0112] As Figure 2 shown, the training process of the robotic arm visual servo control method based on target - conditional reinforcement learning in this embodiment includes the following steps:

[0113] Step 1: Construct a visual servo robotic arm environment, and the robotic arm environment includes a robotic arm;

[0114] Step 2: Data collection or import of saved data. In this data collection phase, the robotic arm interacts with the environment for 1000 episodes. The specific steps for each episode are as follows:

[0115] Step 2.1: Determine whether the environment has terminated or ended. If so, exit this episode; otherwise, continue.

[0116] Step 2.2: Randomly generate an action and complete one step of interaction with the environment.

[0117] Step 2.3: Store the data (action, state, reward, cost, and whether it has terminated) in the replay buffer.

[0118] Step 2.4: Return to Step 2.1.

[0119] Step 3: Training of the safety reinforcement learning model of this method; during the training phase, the robotic arm interacts with the environment for 20000 episodes. The specific steps are as follows:

[0120] Step 3.1: Determine whether the number of iterations in the training phase exceeds the maximum number of iterations. If so, the training ends and the trained model parameters are obtained; otherwise, continue.

[0121] Step 3.2: The robotic arm completes 1 episode of interaction with the environment, i.e., Steps 2.1 - 2.4, but in Step 2.2, the action is generated by the robotic arm according to the current actor network in the DDPG network.

[0122] Step 3.3: Randomly sample actions, states, rewards, and costs from the stored data and perform data preprocessing. The preprocessing includes HER operations and normalization operations.

[0123] Step 3.4: Update the safety - critic network.

[0124] Step 3.5: Update the critic network.

[0125] Step 3.6: Update the actor network.

[0126] Step 3.7: If the number of iterations is a multiple of 100, evaluate the actor network. If the number of iterations is a multiple of 1000, save all network parameters; otherwise, continue.

[0127] Step 3.8: Return to Step 3.1.

[0128] In step 2, it is determined whether to enter the data acquisition phase at the start of training based on whether to use the cache pool variable use_buffer. The purpose of this operation is to avoid the time-consuming operation of data acquisition during multiple trainings, and to locally save the data collected during a certain training for use in subsequent trainings.

[0129] In step 2.2, certain range restrictions are imposed on the randomly generated actions to ensure that the length of the collected trajectory is not too short (the environment does not end prematurely). The restriction operations are as follows:

[0130] ;

[0131] where is a uniform distribution with a range of , and is the randomly generated action.

[0132] After performing the HER operation on the sampled data in step 3.3, the mean and variance of the state and the desired target need to be updated. The update formulas are as follows:

[0133] ;

[0134] where are the expectation of the random variable, the variance of the random variable, the th data , a very small amount (taking ), and are the updated data mean and variance.

[0135] As shown in Figure 4 , Figure 5 and Figure 6 , the effect diagram of using the trained safe reinforcement learning model to control the robotic arm to complete tasks in an offline state. Figure 4 In Figure 5 , the moving feature points of the blue square are the original positions of the two-dimensional moving feature points in the picture, the red dots are the real-time positions of the four moving feature points of the QR code during the movement, and the green dots are the fixed target pixel points in the picture. Figure 3 In Figure 6 , the position of the end of the robotic arm converges to stability within 10 steps, and the number of simulation steps also corresponds to the number of red dots in Figure 6 - 8

[0136] Figure 7

[0136] Figure 7The blue box in represents the position of the QR code in the image after disturbing the position and pose of the QR code, corresponding to Figure 8 the moment when the simulation step in is equal to 50 (the moment when the disturbance is added). Subsequently, the safety reinforcement learning model controls the robotic arm to complete the visual servo task again, moving the four corner points of the QR code in the image to the target position. Figure 7 The red dots in are the real-time detection points of the movement of the QR code corner points in the image, and the number of them is less than 5 (for a single corner point), corresponding to Figure 8 the number of simulation steps less than 5 used by the robotic arm joint angles to lock the target again after the disturbance in , reflecting the anti-disturbance ability of the proposed scheme to control the robotic arm to complete the visual servo task.

Claims

1. A robot arm visual servo control method based on target condition reinforcement learning, wherein a camera is set at the end of the robot arm and a robot arm environment is built, characterized in that: S1. Set four pixel points with known coordinates in the image captured by the robot camera as fixed target points; S2, setting a QR code on the object as the recognition target of the robot arm visual servo task, and the position of the recognition target is fixed after initialization; S3, the robot arm obtains four moving feature points by shooting the QR code with a camera, and uses the secure reinforcement learning model to set the pixel coordinates of the moving feature points and the fixed target points as the achieved target and the expected target in the target condition reinforcement learning respectively; S4. The secure reinforcement learning model includes a deep deterministic policy gradient algorithm, Lagrangian constraints, a post-experience replay algorithm, and a secure critic network, wherein the deep deterministic policy gradient algorithm includes an actor network and a critic network; S5. After the safety reinforcement learning model is trained using the historical interaction data of the robotic arm environment, the current state and expected goal are directly input into the actor network in the deep deterministic policy gradient algorithm. The safety reinforcement learning model can directly output the target speed action of the robotic arm joint through the information of the identified target captured by the camera, control the robotic arm to lock the target with the QR code, and control the movement of the robotic arm to move the mobile feature points to the pixel coordinates of the four fixed target points in the captured image, thereby completing the visual servo control task.

2. The method for controlling a robot arm visual servo based on target condition reinforcement learning according to claim 1, characterized in that: Before the robot arm camera shoots a fixed target point, you need to determine the direction of the camera screen as follows: The camera's screen direction is the vector in the camera coordinate system that starts from the center of the shooting screen and points to the top of the screen. The camera's screen direction is always fixed in the camera coordinate system and is set to ; The direction of the camera image in the world coordinate system will change with the movement of the robot arm. By calculating the representation of the camera image direction in the world coordinate system To ensure the correct refresh of the camera image; obtain the relative position information of each joint of the robot arm, vector The calculation process is as follows: ； In the above formula Represents the coordinate system around Axis rotation The rotation matrix of the angle, Represents the coordinate system around Axis rotation The rotation matrix of the angle, Represents the coordinate system around Axis rotation The rotation matrix of the angle; ； The above formula is the rotation matrix between the coordinate systems of each robot joint, where For joints Coordinate system relative to joint The rotation matrix of the coordinate system, For joints The rotation angle of They represent the world coordinate system, the robot end gripper coordinate system and the camera coordinate system respectively; Representation of the camera screen direction in the world coordinate system as follows: ； In the formula, is the rotation matrix of the camera coordinate system relative to the gripper coordinate system at the end of the robot arm, Represents the rotation matrix of the robot arm end gripper coordinate system relative to the robot arm joint 8 coordinate system, Represents the camera image direction in the camera coordinate system.

3. The method for manipulator visual servo control based on target condition reinforcement learning according to claim 1, characterized in that: The safety reinforcement learning model performs post-experience replay algorithm operations and standardization operations on the historical interaction data of the robot arm environment: actions, states, rewards, achieved goals, and expected goals to complete preprocessing, and reconstructs the reward data to alleviate the reward sparsity problem of the robot arm environment; uses the cost function to build a safety critic network, inputs the robot arm environment historical interaction cost data stored in the replay buffer to the safety critic network, and inputs the robot arm environment historical interaction reward data in the replay buffer to the critic network in the deep deterministic policy gradient algorithm; uses the regularization term and Lagrangian constraint term in the actor network in the deep deterministic policy gradient algorithm to restrict the movement of the robot arm to ensure the safety of the learning process, and implements the Lagrangian constraint by introducing the output value of the safety critic network into the actor network in the deep deterministic policy gradient algorithm; Among them, the actor network, critic network and safety critic network of the deep deterministic policy gradient algorithm all adopt a fully connected structure, including an input layer, a hidden layer and an output layer; the actor network takes the state of the robot environment and the expected goal as input, and outputs an action vector. The action vector constitutes a regularization term loss through the two-norm regularization. The safety critic network takes the robot environment state and action as input, outputs the cost value, and the cost value constitutes the cost value term loss. The critic network takes the robot environment state, action and expected goal as input, outputs the action value, and constitutes the action value term loss with the constant -1; finally, the actor network parameters are updated according to the gradients calculated by the regularization term loss, the cost value term loss and the action value term loss using the adaptive moment estimation optimizer, so as to control the robot environment to complete the visual servoing task as soon as possible while ensuring that the QR code does not exceed the camera screen.

4. The method for manipulator visual servo control based on target condition reinforcement learning according to claim 3 is characterized in that: The method for detecting the moving feature points of a QR code is as follows: Assume that the state space of the robot environment contains four moving feature points. The pixel coordinates of the four moving feature points are obtained by detecting the moving feature point detection function cv2.aruco.detectMarkers() of opencv. The pixel coordinates of the four moving feature points of the QR code are represented as an 8-dimensional target space goal: ,in For the The pixel coordinates of the feature points. The target space represents the moving range of the four moving feature points of the detected QR code in the picture. The target space represents the reached target at time t in the target condition reinforcement learning. and expected goals are all in the target space. Designed to be the current moment in the robot environment The pixel coordinates of the four moving feature points of the QR code, The pixel coordinates of four fixed target points in the camera frame that are assigned to the locations of the moving feature points when the environment is initialized.

5. The method for manipulator visual servo control based on target condition reinforcement learning according to claim 4 is characterized in that: The robot environment uses a seven-axis robot. The states and actions of the robot environment are as follows: The state of the robot environment is represented by the state space, including: the end position and speed of the robot, the width of the gripper, the seven joint angles and speeds, and the range space of the target. The robot state in this range space is used as the input of the actor network, the critic network, and the safety critic network to complete the training of the safety reinforcement learning model. The robot state space is a 29-dimensional vector, represented as: ； in, Represents the end information of the seven-axis robot arm: end position, end speed, end gripper width, a total of 7 dimensions; Represents the joint angles of the seven-axis robot arm, a total of 7 dimensions; Indicates the joint speed of the seven-axis robot arm, a total of 7 dimensions; Represents the pixel coordinates of the four moving feature points of the QR code, a total of 8 dimensions; The action of the robot environment is identified in the action space, that is, the range space of the movement speed of the seven joints and one gripper of the robot. In the action space of the robot environment, the action is used as the input of the actor network, the critic network, and the safety critic network to complete the training of the safety reinforcement learning model. The action space of the robot environment is an 8-dimensional vector, which is expressed as: ； This includes the target speed of each joint of the robot arm , end gripper movement target speed .

6. The method for manipulator visual servo control based on target condition reinforcement learning according to claim 1, characterized in that: The safety reinforcement learning model generates rewards for the robot arm's environment historical interaction data through a reward function, which is a sparse reward function: ； In the above formula ,in for The goal of the moment has been reached, for The desired goal at the moment, That is, the two-norm distance between the two, is the hyperparameter distance threshold. , you will get a reward of 0, otherwise you will get a reward of -1; This kind of reward is sparse, and the HER algorithm is used to preprocess the data to alleviate the sparse reward and increase sample efficiency; The safety critic network inputs the state and action of the robot arm and evaluates the dangerousness of the robot arm's action, where the cost function is: ； In the formula, for A flag variable indicating whether the environment is terminated at the moment. When the environment is terminated, that is, the QR code on the surface of the object exceeds the camera screen, the robot arm is given an environmental cost of 0 to punish it for taking an action that terminates the environment at that moment. Otherwise, a cost of -1 is given.

7. The method for manipulator visual servo control based on target condition reinforcement learning according to claim 6, characterized in that: The specific process of post-experience replay algorithm operation and standardization operation is as follows: (1) Post-experience replay algorithm operation: 1) Select a trajectory data generated by the interaction between the robot arm and the environment from the playback buffer area. The trajectory data is the interaction data containing multiple moments from t=0 to the end of the interaction with the environment; 2) Randomly select a time in the trajectory data and At the moment, the interaction data at the corresponding moment are and ,in Indicates the status of the robot arm. Indicates action, represents the reward function, Indicates desired goal, Indicates that the goal has been achieved; 3) In the time data Depend on In the time data Instead, that is ; 4) Based on the reward function Recalculate Rewards of the moment ; 5) The data after the post-experience playback algorithm is completed is ; (2) Standardized operations: The robot state contains different types of information such as the robot's three-dimensional position, speed, gripper width, and pixel coordinates. The following formula is used to standardize the state and the desired target and splice the robot state with the desired target: ， ； In the formula, is the robot arm status, For the desired goal, is the mean and standard deviation of the robot state, is the mean and standard deviation of the expected target. The concatenation operation is the state first and the expected target second.

8. The method for manipulator visual servo control based on target condition reinforcement learning according to claim 6, characterized in that: The training process of the security reinforcement learning model is as follows: Security Critic Network Exploitation Cost Information Construct an assessment of the degree of danger of the action, and introduce the target strategy after the cost is as follows: ； in are the time discount factor and cost threshold of the reward, respectively. They are strategy, state, action, reward function, cost, and the distribution of state and action generated by the strategy; the meaning of the target strategy is to meet the safety constraints, that is, the environment step of the QR code remaining in the camera screen exceeds the corresponding threshold , the expected cumulative reward of the robotic arm is maximized; the loss function of the security critic network is as follows: ； The data Sampling in playback buffer , which is composed of the robot's current state, action, cost, expected goal, and state at the next moment. is the time discount factor for the cost, strategy It is a deterministic policy for the deep deterministic policy gradient algorithm, which takes the state and the desired goal as input and outputs a deterministic action; and are the target network parameters and network parameters of the security critic network, Updated by soft update; the output of the security critic network is the cost value , whose input is state and action, representing the evaluation of the dangerousness of the action, and completing the dangerousness evaluation of the robot arm action causing the QR code to go out of the screen; the cost value output by the safety critic network is added to the actor network update of the deep deterministic policy gradient algorithm to ensure that the update of the actor network takes into account the dangerousness of the action.

9. The method for manipulator visual servo control based on target condition reinforcement learning according to claim 8, characterized in that: Based on the critic network and actor network update method in the deep deterministic policy gradient algorithm, regularization terms and security critic networks are added to the actor network. The specific contents are as follows: (1) Reviewer Network Update: ； in is the reward at the current moment, and are the parameters of the critic network and the critic target network respectively. The output of the critic network is the action value. , whose input is state, action and desired goal, represents the estimation of action value; (2) Update of the actor network. The actor network update function in the original deep deterministic policy gradient algorithm only contains the action value term , modify and add action regularization items , cost value items , to ensure that the update of the actor network takes into account the magnitude and dangerousness of the action, as follows: ； in are the actor network parameters, For actor network with status and expected goals Deterministic actions generated for input, is the two-norm regularization term for generating actions for the actor network, are the weight parameters of the action regularization term and the cost value term respectively. The actor network loss function consists of three parts: the action value term output by the critic network, the action regularization term, and the cost value term output by the security critic network. The action value term ensures that the update of the actor network parameters is in the direction of increasing the action value. The action regularization term ensures that the amplitude of the action output by the actor network is reduced. The cost value term ensures that the update of the actor network parameters is in the direction of reducing the cost value.

10. A computer device, characterized in that: It includes a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the robot arm visual servo control method based on target condition reinforcement learning as described in any one of claims 1-9.

Images