Manipulator grasping alignment method, system and storage medium

Abstract

The present invention belongs to the field of robot control technology, and specifically relates to a robot grasping alignment method, system and storage medium. The robot grasping alignment method comprises acquiring and extracting the outline of a target object based on a picture of the target object; calculating the position state information and shape of the target object according to the outline of the target object; determining the robot grasping gesture based on the shape of the target object; using the position state information of the target object as the state space of a preset reinforcement learning algorithm, and outputting the robot wrist joint movement angle; and realizing the grasping alignment of the target object by the robot based on the robot grasping gesture and the robot wrist joint movement angle. The present invention solves the problem of low intelligence and flexibility of existing robots.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control, and in particular relates to a robot grasping alignment method, system and storage medium. Background Art

[0002] As an important part of industrial automation and intelligent manufacturing, robots play a key role in production, manufacturing, medical care, services and other fields. With the advancement of science and technology and the continuous improvement of application needs, higher requirements are placed on the intelligence and flexibility of robots.

[0003] Traditional manipulators usually rely on preset programs and fixed action sequences to complete tasks, lacking the ability to make autonomous decisions and adapt to the environment. This approach can meet basic needs in scenarios with simple structures and stable environments, but often performs poorly in complex and changing dynamic environments. For example, when the position or posture of the target object changes, it is difficult for traditional manipulators to make corresponding adjustments, which can easily cause problems such as grasping failure or collision.

[0004] In addition, the operation of traditional manipulators often requires manual programming and debugging, which is time-consuming and labor-intensive, and it is difficult to quickly adapt to new task requirements. In some highly dynamic scenarios, such as collaborative robots, service robots and other application scenarios, the limitations of traditional manipulators are more obvious, and it is difficult to meet the needs of flexible interaction and real-time decision-making. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a robot gripping alignment method, system and storage medium, which solve the problem of how to accurately grip and align a target object.

[0006] To achieve the above purpose, the present invention adopts the following technical solution.

[0007] The first aspect of the present invention provides a robot gripping alignment method, comprising: an upper-layer robot gripping gesture selection step and a lower-layer robot wrist joint gripping alignment step; wherein:

[0008] The upper-layer manipulator grasping gesture selection step includes: acquiring and extracting the outline of the target object based on the image of the target object; calculating the position state information and shape of the target object according to the outline of the target object; and determining the manipulator grasping gesture based on the shape of the target object;

[0009] The lower-layer robot wrist joint grasping alignment step includes: using the position state information of the target object as the state space of the preset reinforcement learning algorithm, and outputting the robot wrist joint action angle;

[0010] The robot grasps the target object and aligns it based on the robot's grasping gesture and the movement angle of the robot's wrist joint.

[0011] As an implementation manner of the present invention, the step of acquiring and extracting a contour of the target object based on a picture of the target object includes:

[0012] The camera on the robot arm obtains the image of the target object in real time;

[0013] Preprocessing the image of the target object to obtain a preprocessed image of the target object;

[0014] Performing edge detection on the preprocessed image of the target object to obtain an image of the target object after edge detection;

[0015] Performing dilation processing on the image of the target object after edge detection to obtain the image of the target object after dilation processing;

[0016] The contour of the target object is extracted from the image after the expansion process to obtain the contour of the target object.

[0017] As an implementation manner of the present invention, the step of calculating the position state information and shape of the target object according to the contour of the target object includes:

[0018] Obtain the geometric midpoint coordinates of the minimum circumscribed rectangle of the target object's outline and the ratio of the lengths of the two symmetry axes, as well as the roundness of the target object's outline;

[0019] The position state information of the target object is obtained by performing relative coordinate calculation on the geometric midpoint coordinates of the minimum circumscribed rectangle and the optical axis point coordinates in the camera coordinate system; and

[0020] When the roundness of the target object's contour is greater than the roundness judgment threshold, the shape of the target object is judged to be a sphere; otherwise, it is a cuboid or a cube, wherein: when the ratio of the lengths of the two symmetry axes of the minimum circumscribed rectangle of the target object is greater than the axis length ratio judgment threshold, the shape of the target object is judged to be a cuboid, otherwise, it is a cube.

[0021] As an implementation mode of the present invention, the method of using the position state information of the target object as the state space of the preset reinforcement state information learning algorithm to output the joint motion angle of the robot wrist includes:

[0022] Get the position status information of the target object, where the position status information of the target object is It indicates that, t For the moment, is the environmental state, t The position state information of the target object obtained by the upper-level manipulator grasping gesture selection step at the moment As environmental status ;

[0023] A double-delayed deep deterministic gradient strategy reinforcement learning algorithm is used as a preset reinforcement learning algorithm, wherein the double-delayed deep deterministic gradient strategy reinforcement learning algorithm is represented by a TD3 reinforcement learning algorithm, and the state space of the TD3 reinforcement learning algorithm is all possible position state information of the target object;

[0024] The position status information of the target object Input into the TD3 reinforcement learning algorithm and output a set of angle values , this group of angle values ​​is the joint action angle of the robot wrist.

[0025] As an embodiment of the present invention, the reward function of the TD3 reinforcement learning algorithm is set to give different reward values ​​according to the degree of alignment of the joints of the robotic wrist after the action is completed. , where the misalignment value is defined as: , misalignment value d The smaller the value, the higher the degree of alignment of the current robot. d The larger the value, the lower the current alignment.

[0026] As an embodiment of the present invention, the position state information of the target object is Input into the TD3 reinforcement learning algorithm and output a set of angle values ,include:

[0027] exist t time:

[0028] The agent receives the output of the upper-level robot grip gesture selection step t The position status information of the target object at the moment As environmental status ;

[0029] The environmental status Input to the policy network of the agent Make a decision and output the two wrist joint action angles through the policy network As an action .

[0030] As an embodiment of the present invention, after the robot grasps and aligns the target object based on the robot grasping gesture and the robot wrist joint action angle, the method includes:

[0031] The motor of the wrist joint performs the action After the execution is completed, the time becomes t +1 moment;

[0032] exist t +1 time:

[0033] The agent receives the output of the upper-level robot grip gesture selection step t +1 The position status information of the target object at time As environmental status ;

[0034] The environmental status Substitute the preset reward function to get the action Reward value ;

[0035] The four-tuple sample data Stored in the sample data pool.

[0036] As an embodiment of the present invention, the lower layer TD3 reinforcement learning algorithm includes six networks: Two value networks, Two target value networks and one strategy network and a target policy network ;

[0037] The training process of the TD3 reinforcement learning algorithm includes:

[0038] Extract a batch of four-tuple sample data from the sample data pool ;

[0039] , where each element of the vector is independently derived from a truncated normal distribution Extract from the neural network, the subscript now indicates the current parameters of the neural network;

[0040] Two target value networks are used for prediction:

[0041] ;

[0042] ;

[0043] Calculate TD target:

[0044] ;

[0045] Two value networks are used for prediction:

[0046] ;

[0047] ;

[0048] Calculate the TD error:

[0049] ;

[0050] ;

[0051] Update the two value networks above:

[0052] ;

[0053] ;

[0054] Every k Update the policy network and target network parameters once:

[0055] Use the policy network to make predictions: , and then update the policy network:

[0056] ;

[0057] Update the target network parameters:

[0058] ;

[0059] ;

[0060] ;

[0061] in, are the parameters of the value network, are the parameters of the two target value networks respectively, are the policy network parameters and target policy network parameters respectively, The action value predictions made by the two value networks are: The predictions made by the two target value networks respectively, For TD target, are the TD errors of the two value networks, are the learning rates for updating the model parameters of the value network and the policy network, respectively. The updated parameters of the target network, is the discount rate, α is the learning rate, is the gradient of ω, For θ The gradient of For a gradient.

[0062] The second aspect of the present invention provides a manipulator grasping alignment system, comprising: an upper-layer manipulator grasping gesture selection unit, a lower-layer manipulator wrist joint grasping alignment unit and an execution unit; wherein:

[0063] The upper layer manipulator gripping gesture selection unit is at least used for: acquiring and extracting the outline of the target object based on the image of the target object; calculating the position state information and shape of the target object according to the outline of the target object; and determining the manipulator gripping gesture based on the shape of the target object;

[0064] The lower layer robot wrist joint grasping alignment unit is at least used to: use the position state information of the target object as the state space of the preset reinforcement learning algorithm, and output the robot wrist joint action angle;

[0065] The execution unit is at least used for: realizing the grasping alignment of the manipulator on the target object based on the grasping gesture of the manipulator and the action angle of the joint of the manipulator wrist.

[0066] A third aspect of the present invention provides a readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the steps of the method described in the first aspect of the present invention.

[0067] In summary, compared with the prior art, the present invention includes at least one of the following beneficial technical effects:

[0068] 1. The present invention can perform efficient grasping operations by equipping the manipulator with a monocular camera and deploying the strategy method to control the manipulator;

[0069] 2. The present invention can directly process parameters such as the position state information of the target object in real time, output precise joint angle control instructions, continuously optimize the grasping strategy through continuous environmental interaction and learning process, improve the grasping accuracy and adaptability of the manipulator, and realize autonomous learning and decision-making capabilities, which greatly enhances the application potential of the manipulator in complex and changeable environments and provides strong support for the development of intelligent manufacturing and robotics technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0071] Figure 1 It is a method block diagram of a manipulator grasping alignment method according to an embodiment of the present invention.

[0072] Figure 2 It is a method flow chart of a manipulator grasping alignment method according to an embodiment of the present invention.

[0073] Figure 3 It is a block diagram of a target object contour recognition method in an upper-layer algorithm of an embodiment of the present invention.

[0074] Figure 4 It is a block diagram for calculating the position state information of a target object in an upper layer algorithm of an embodiment of the present invention.

[0075] Figure 5 It is a flow chart of a method for target object shape recognition and grasping gesture selection in the upper-layer algorithm of an embodiment of the present invention.

[0076] Figure 6 It is a flow chart of a wrist joint control method for robot grasping alignment in the lower-level algorithm of an embodiment of the present invention.

[0077] Figure 7 This is a graph showing how round rewards change with the number of rounds during an agent training process according to an embodiment of the present invention.

[0078] Figure 8 This is a graph showing how round rewards change with the number of rounds during an agent evaluation process according to an embodiment of the present invention.

[0079] Fig. 9 4 is a block diagram of a manipulator grasping and alignment system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application. In addition, it should be understood that the specific implementation methods described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0081] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments of the present application. In addition, in the following embodiments, the description of each embodiment has its own emphasis, and for parts not described in detail in one embodiment, reference can be made to the relevant description of other embodiments.

[0082] OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library that provides a large number of image and video processing functions, including rich processing functions, such as:

[0083] cvtColor(): used to convert the color space of an image, such as from BGR (blue, green, red) to a grayscale image, or to convert to other color spaces;

[0084] GaussBlur(): Used to perform Gaussian blur on an image to reduce image noise and details by smoothing the image;

[0085] Canny operator: used to detect edges in images;

[0086] dilate(): Used to dilate an image to increase the area of ​​the white (or foreground color) area in the image, which helps to enhance the edges and features in the image;

[0087] findContours(): used to detect and extract contours in an image. It can find all connected white areas in an image and return their contours, which can be used to identify and locate objects in an image.

[0088] minAreaRect(): Used to calculate the minimum area rectangle enclosing a given set of points or contours.

[0089] like Figure 1 and Figure 2 As shown, the first aspect of the present invention provides a manipulator grasping alignment method, which comprises two sub-methods, namely an upper-level manipulator grasping gesture selection step and a lower-level manipulator wrist joint grasping alignment step.

[0090] The method corresponding to the upper-level manipulator grasping gesture selection step is the upper-level algorithm, which specifically includes the following steps:

[0091] Step 100: Acquire and extract the contour of the target object based on the image of the target object;

[0092] Step 200: Calculating the position state information and shape of the target object according to the contour of the target object;

[0093] Step 300: Calculate the shape of the target object according to the contour of the target object to determine or select the gripping gesture of the manipulator.

[0094] The method corresponding to the grasping alignment step of the lower-level wrist joint is the lower-level algorithm, which includes the following steps:

[0095] Step 400: Using the position state information of the target object as the state space of a preset reinforcement learning algorithm to output the joint motion angle of the robotic wrist.

[0096] The gripping alignment of the manipulator to the target object is achieved through the manipulator gripping gesture determined by the upper-layer manipulator gripping gesture selection step and the manipulator wrist joint motion angle output by the lower-layer manipulator wrist joint gripping alignment method.

[0097] The present invention utilizes the image information of the target object, and after the upper-level manipulator grasping gesture selection step and the calculation processing of the lower-level manipulator wrist joint grasping alignment method, it can directly output the wrist joint angle of the intelligent manipulator during grasping alignment, thereby realizing the grasping alignment control of the manipulator.

[0098] like Figure 3 As shown, in one embodiment of the present invention, the step of acquiring and extracting the contour of the target object based on the image of the target object includes:

[0099] Step S101: collecting a video stream containing a target object in real time through a camera on the manipulator, and obtaining a picture of the target object based on the video stream containing the target object, wherein the camera may specifically be a monocular camera disposed at the base of the manipulator;

[0100] Step S102: using the cvtColor() function to perform binarization processing on the image of the target object to obtain the image of the target object after binarization processing;

[0101] Step S103: using the GaussBlur() function to perform Gaussian blur processing on the image of the target object after the binarization processing, to obtain the image of the target object after the Gaussian blur processing; the binarization processing and Gaussian blur processing can be understood as pre-processing steps;

[0102] Step S104: using the edge detection Canny operator to perform edge detection on the image of the target object after Gaussian blur processing, to obtain the image of the target object after edge detection;

[0103] Step S105: using the dilate() function to dilate the image of the target object after edge detection to obtain the image of the target object after dilation;

[0104] Step S106: using findContours() function to extract the contour of the image of the target object after the dilation process, to obtain the contour of the target object.

[0105] The present invention can enable the manipulator to dynamically perceive environmental changes and adapt to changes in the position and posture of the target object through the collection of real-time video streams. Binarization processing can simplify the image, highlight the contrast between the target object and the background, and facilitate subsequent image processing and analysis. Gaussian blur processing can smooth the image, reduce noise interference, and improve the accuracy of subsequent edge detection. Edge detection can identify the contour features of the target object and provide key information for subsequent object recognition and positioning. Expansion processing can fill in the small gaps that may exist after edge detection, making the contour of the target object more continuous and complete. Contour extraction can obtain the precise shape of the target object, providing an important basis for the grasping planning and path generation of the manipulator. Steps S101-S106 of the present invention form a complete image processing flow, which can extract useful feature information from the original image, improve the recognition ability of the present invention for the target object, enable the manipulator to have stronger environmental adaptability and task flexibility, and do not require frequent manual programming and debugging, thereby improving the work efficiency and response speed of the manipulator.

[0106] like Figure 4 As shown, in one embodiment of the present invention, the step of calculating the position state information of the target object according to the contour of the target object includes:

[0107] Step S201: using the minAreaRect() function to obtain the minimum circumscribed matrix of the contour of the target object and the coordinates of the four corner points of the minimum circumscribed matrix;

[0108] Step S202: obtaining the coordinate information of the symmetry axis endpoints of the minimum circumscribed rectangle by performing midpoint coordinate calculation on the coordinate information of the four corner points of the minimum circumscribed rectangle;

[0109] Step S203: obtaining the geometric midpoint coordinates of the minimum circumscribed rectangle by calculating the midpoint coordinates of the symmetry axis endpoints of the minimum circumscribed rectangle;

[0110] Step S204: acquiring the position state information of the target object by performing relative coordinate calculation on the geometric midpoint coordinates of the minimum circumscribed rectangle and the optical axis point coordinates in the camera coordinate system.

[0111] Specifically, the coordinates of the four corner points of the minimum bounding rectangle of the target object are the coordinates of the pixel coordinate system in the target object image, and the coordinates of the geometric midpoint of the minimum bounding rectangle are the coordinates of the point in the pixel coordinate system. , the optical axis point in the camera coordinate system is the origin of the camera coordinate system, which is converted to the midpoint of the image in the pixel coordinate system. The corresponding coordinates are The position state information of the target object is the relative coordinates of the geometric center point of the minimum circumscribed rectangle of the target object and the camera optical axis point in the pixel coordinate system, that is, ,in:

[0112] ;

[0113] .

[0114] like Figure 5 As shown, in one embodiment of the present invention, the shape of the target object is calculated according to the contour of the target object, the shape of the target object is judged, and a suitable manipulator grasping gesture is selected according to the judged shape of the target object, including:

[0115] Step S301: roundness of the contour of the target object, by judging the roundness of the contour of the target object, to achieve the spherical shape judgment of the target object;

[0116] Step S302: Calculate the ratio of the lengths of the two symmetric axes of the minimum circumscribed rectangle of the target object's outline;

[0117] Step S303: determining whether the target object is a rectangular parallelepiped or a cube by determining the axis length ratio;

[0118] Step S304: Selecting a corresponding robot gripping gesture according to the shape of the target object.

[0119] Specifically, the roundness calculation method of the target object is: ,in C is the contour perimeter, S is the contour area; the logic of the roundness judgment is: when the roundness of the target object is greater than the roundness judgment threshold When , the target object is judged to be a sphere; the ratio of the lengths of the two symmetry axes is k The calculation method is: ,in are the lengths of the two symmetric axes of the contour respectively; the ratio of the two symmetric axis lengths is determined as the ratio of the axis lengths k Greater than the axis length ratio judgment threshold When , the target object is judged to be a cuboid, otherwise it is a cube. In this embodiment, the roundness judgment threshold is set to , set the axis length ratio judgment threshold to .

[0120] Specifically, the gripping gestures of the manipulator include gesture 1, gesture 2 and gesture 3. The relationship between the shape of the target object and the corresponding gripping gesture of the manipulator is as follows: gripping gesture 1 is a five-finger pinching gripping gesture, corresponding to a rectangular parallelepiped; gripping gesture 2 is a five-finger pinching gripping gesture, corresponding to a cube; gripping gesture 3 is a five-finger grasping gesture, corresponding to a sphere.

[0121] Specifically, the shape of the target object designed in the present invention can be understood as a large category of shape categories. For example, a cylinder or a cuboid that is relatively slender on a geometric scale can use gesture 1; objects with relatively close major and minor axes on a geometric scale use grasping gesture 2, and geometric objects with a higher roundness use grasping gesture 3, etc. The present invention does not impose too many restrictions, and those skilled in the art can achieve grasping of a variety of objects through the grasping method provided by the present invention.

[0122] The present invention realizes automatic recognition and classification of the shape of the target object (sphere, cuboid, cube) by calculating and judging the roundness of the contour of the target object and the ratio of the length of the symmetry axis of the minimum circumscribed rectangle, and selects the corresponding manipulator grasping gesture (five-finger pinching, five-finger pinching, five-finger grasping) according to the recognition result, so that the manipulator can adaptively adjust the grasping mode according to objects of different shapes, improves the accuracy and stability of grasping, enhances the adaptability and operational flexibility of the manipulator to objects of various shapes, realizes intelligent object grasping without human intervention, and greatly improves the efficiency and reliability of automated production and operation.

[0123] like Figure 6 As shown, in one embodiment of the present invention, the position state information of the target object is used as the state space of the preset reinforcement learning algorithm to output the joint motion angle of the robot wrist, including:

[0124] Step S401: Acquire the position status information of the target object, where the position status information of the target object is It indicates that, t For the moment, is the environmental state, t Environmental status at any moment The position state information of the target object obtained by the upper-level manipulator grasping gesture selection step ;

[0125] Step S402: Using a double-delayed deep deterministic gradient strategy reinforcement learning algorithm as a preset reinforcement learning algorithm, the double-delayed deep deterministic gradient strategy reinforcement learning algorithm is represented by a TD3 reinforcement learning algorithm, and the state space of the TD3 reinforcement learning algorithm is all possible position state information of the target object; the position state information of the target object is Input into the policy network of the TD3 reinforcement learning algorithm;

[0126] Step S403: The strategy network outputs a set of angle values , this group of angle values ​​is the joint action angle of the robot wrist.

[0127] Specifically, the lower-level dual-delayed deep deterministic gradient strategy (TD3) reinforcement learning algorithm is used to solve the control problem in the continuous action space. In the robot grasping task, the policy network of the TD3 algorithm can receive the position state information of the target object as input and output the action angle of the robot wrist joint. Through continuous interaction and learning with the environment, the TD3 algorithm can gradually optimize the policy network, enabling the robot to complete the grasping alignment task more accurately and stably.

[0128] The underlying double-delayed deep deterministic gradient policy (TD3) reinforcement learning algorithm contains six core network components: two Q networks (Value Network), Two Target Q Networks (Goal Value Network), a Policy Network and a target policy network .

[0129] In the robot grasping task, the policy network of the TD3 algorithm can receive the position state information of the target object as input and output the action angle of the robot wrist joint. Through continuous interaction and learning with the environment, the policy network can gradually optimize the grasping strategy and achieve precise grasping alignment control.

[0130] In one embodiment of the present invention, the TD3 reinforcement learning algorithm is used to determine the movement of the wrist joint by receiving the position state information of the target object. t Environmental status at any moment The target object position status information output by the upper algorithm , the state space of the algorithm is all possible position state information of the target object; the algorithm t Moment of action The angle of the robot wrist joint ; The action space of the algorithm is the maximum range of motion of the wrist joint: , .

[0131] In one embodiment of the present invention, the specific steps of establishing a Markov process for the process of the manipulator grasping and aligning the target object include:

[0132] exist t time:

[0133] The agent receives the output of the upper-level robot grip gesture selection step t The position status information of the target object at the moment As environmental status ;

[0134] The environmental status Input to the policy network of the agent Make a decision and output the two wrist joint action angles through the policy network As an action .

[0135] Specifically, the present invention uses a reinforcement learning algorithm, and the agent is an abstract concept in the reinforcement learning algorithm, that is, the subject of algorithm execution and training. When the algorithm is used for control, the policy network of the agent is used for output. The entire grasping alignment process is modeled as a Markov decision process, which means that the decision (output angle value) of each time step depends only on the current state (position state of the target object), and does not depend on the previous historical state.

[0136] In one embodiment of the present invention, the method of realizing the gripping alignment of the manipulator on the target object based on the gripping gesture of the manipulator and the action angle of the wrist joint of the manipulator includes:

[0137] The motor of the wrist joint performs the action After the execution is completed, the time becomes t +1 moment;

[0138] exist t +1 time:

[0139] The agent receives the output of the upper-level robot grip gesture selection step t +1 The position status information of the target object at time As environmental status ;

[0140] The environmental status Substitute the preset reward function to get the action Reward value ;

[0141] The four-tuple sample data Stored in the sample data pool.

[0142] In one embodiment of the present invention, the reward function of the TD3 reinforcement learning algorithm is set to give different reward values ​​according to the degree of alignment of the joints of the robotic wrist after the action is completed. , where the misalignment value is defined as: , misalignment value d The smaller the value, the higher the degree of alignment of the current robot. d The larger the value, the lower the current alignment.

[0143] Specifically, in an application scenario of the present invention, according to different misalignment degree values, the specific setting of the preset reward function is:

[0144] ;

[0145] in , are the weights of each penalty term. The position state information output by the upper-level manipulator grasping gesture selection step The misalignment value of the manipulator in the current state can be calculated. When the misalignment value is less than the reward zone threshold of 80, the agent enters the alignment reward zone and receives a reward. In the alignment reward zone, two step thresholds are set, 50 and 30 respectively. When the misalignment value enters different step thresholds, the algorithm agent will receive reward values ​​of different sizes. By dividing the alignment reward zone into steps, the agent is encouraged to decide on actions with the highest possible alignment. When the misalignment value is greater than the reward zone threshold of 80, the agent enters the misalignment penalty zone and receives a penalty. The larger the misalignment value, the greater the penalty value given, and the smaller the misalignment value, the smaller the penalty value given.

[0146] like Figure 7 and Figure 8 As shown, Figure 7 The cumulative reward of each round changes with the number of rounds during the training process of the agent. Figure 8 The cumulative reward of the trained agent changes with the number of rounds during the model evaluation process. In an application scenario of the present invention, Figure 7 The vertical axis is the agent's cumulative round reward, and the horizontal axis is the number of rounds. It can be observed that in the initial exploration stage and greedy annealing stage of the agent, the cumulative round reward value is unstable and has a large fluctuation amplitude. As the number of training steps exceeds the initial exploration stage and greedy annealing stage, the agent's cumulative round reward gradually reaches a higher level, all falling into the alignment reward area and tending to be stable. This shows that the training of the policy network has converged, and the agent can decide on wrist joint actions with better alignment effects for target objects in different positions. Figure 8 The cumulative reward of each round changes with the number of rounds in the model evaluation. The vertical axis is the cumulative reward of the agent, and the horizontal axis is the number of rounds. It can be observed that most of the cumulative reward values ​​of each round fall into the alignment reward area, indicating that the gripping alignment effect of the manipulator is very good in most cases. Only a very small number of cumulative reward values ​​of each round fall into the misalignment penalty area. For the points where the cumulative reward of each round falls into the misalignment penalty area, the subsequent actions bring the cumulative reward of each round back to the alignment reward area. This shows that when the agent encounters the manipulator losing the gripping alignment state, the agent can make action decisions in time to bring the manipulator and the target object back to the gripping alignment state. The gripping alignment process of the manipulator is quantitatively evaluated, and the overall gripping alignment rate is 96.64%, indicating that the gripping alignment effect of the manipulator is good in most cases, and the effectiveness of the underlying algorithm is verified.

[0147] In one embodiment of the present invention, the training process of the TD3 reinforcement learning algorithm includes:

[0148] Extract a batch of four-tuple sample data from the sample data pool ;

[0149] , which means that the sample data will be extracted Input to the target policy network and output the predicted action ; where the vector Each element of is independently derived from a truncated normal distribution Extract from the sample data, the subscript now represents the current parameters of the neural network; Input to the target policy network and output the predicted action ;

[0150] Two target value networks are used for prediction:

[0151] ;

[0152] ;

[0153] Calculate TD target:

[0154] ;

[0155] Two value networks are used for prediction:

[0156] ;

[0157] ;

[0158] Calculate the TD error:

[0159] ;

[0160] ;

[0161] Update the two value networks above:

[0162] ;

[0163] ;

[0164] Every k Update the policy network and target network parameters once per round;

[0165] Use the policy network to make predictions: , and then update the policy network:

[0166] ;

[0167] Update the target network parameters:

[0168] ;

[0169] ;

[0170] .

[0171] Among them, the TD (Temporal Defference) method is a model training method, namely the temporal difference method. t+ The results of time 1 and t The TD target is t The result of +1 moment is compared with t The result at the moment is more "accurate", and the TD error is t +1 moment result and t The error of the result at the moment, and the TD error is used to correct the parameters of the model. The meaning of each parameter is: is a set of sample data in the sample data pool. They are t The state of the environment, actions, and reward values ​​at each moment, for t+ The environmental status at moment 1, are the parameters of the value network, are the parameters of the two target value networks respectively, are the policy network parameters and target policy network parameters respectively, The action value predictions made by the two value networks are: The predictions made by the two target value networks respectively, For TD target, are the TD errors of the two value networks, are the learning rates for updating the model parameters of the value network and the policy network, respectively. The updated parameters of the target network, is the discount rate, α is the learning rate, is the gradient of ω, For θ The gradient of For a The gradient of t The meaning of parameters corresponding to different moments, such as Represents the corresponding value network's action value at time t TD error.

[0172] The specific algorithm used in the present invention is the hierarchical reinforcement learning algorithm, namely Hierarchical Reinforcement Learning (HRL), which is a type of reinforcement learning algorithm. The intelligent agent is an abstract concept in the reinforcement learning algorithm, that is, the subject that makes decisions in the algorithm and is the object to be trained in the algorithm. The algorithm divides the control process of the manipulator into two layers, the upper layer obtains the information of the target object through machine vision, and the lower layer is the TD3 algorithm, which controls the manipulator by receiving the target object position state information output by the upper layer. Therefore, in the present invention, the intelligent agent is specifically defined as: the intelligent agent of the lower-layer mechanical wrist joint control algorithm (i.e., the TD3 algorithm).

[0173] The agent of the TD3 algorithm has six networks, including two value networks, two target value networks, a policy network, and a target policy network. In this case, the TD3 network is the agent network, and the TD3 agent network is a component of the lower-level algorithm of the HRL hierarchical reinforcement learning algorithm.

[0174] In an application scenario of the present invention, some important hyperparameter settings for the agent network training process are shown in Table 1. By setting a smaller optimizer learning rate to reduce the rate of change of the agent network parameters; by setting a larger policy network update delay to improve the accuracy of the value network's evaluation results for the policy network, and by reasonably setting the key hyperparameters of the agent algorithm to optimize the training and evaluation results of the agent.

[0175] Table 1:

[0176] .

[0177] In one embodiment of the present invention, the method further includes randomly initializing the state, setting the initial exploration phase, and using -Greedy degree annealing method is used to improve the stability and generalization of method decisions.

[0178] Specifically, the joint state of the robot wrist should be randomly initialized and reset after the current round ends and before the next round begins. Random noise is added to the reset process of the robot wrist joint to randomly initialize the state reset. After adding random noise, the wrist joint rotation angle of the robot is different each time the state is reset, and the position state obtained from the environment is also different. The random initialization state can enhance the agent's exploration of the environment and accelerate the convergence speed of the strategy.

[0179] Specifically, before the number of training steps reaches the upper limit of the number of steps in the initial exploration phase, the agent will randomly select actions from the exploration action space to interact with the environment, so as to complete the exploration of the environment. The exploration action space is a subset of the set actual action space. When exploring the environment, the agent extracts actions from the exploration action space and uses the actions to complete the exploration of the environment. In this embodiment, the exploration action space is set as: The purpose of setting the initial exploration phase is to ensure that the sample data pool contains enough random exploration samples, so as to avoid falling into the local optimum, accelerate the learning process of the agent, and improve the generalization of the final strategy.

[0180] Specifically, each time the agent decides to take an action, the agent will The probability of randomly extracting actions from the exploration action space completes the agent's exploration of the environment interaction. At the same time, the agent will also be 1- The probability of using the agent strategy network to execute the action that the agent considers optimal is to complete the use of the agent in the interaction with the environment. This can balance the relationship between exploration and exploitation in the interaction between the agent and the environment. -In the greedy degree annealing algorithm, - The greediness starts from a larger value, which is set to 1 in this embodiment, and gradually decreases to 0 as the number of training steps increases. - Greedy annealing in the early stage, - The greediness value is large, and the agent has a high probability of randomly extracting actions from the explored action space. As the number of training steps increases, -The greediness value gradually decreases, and the agent gradually increases the probability of using its own strategy network to make decisions. -When the greediness value is annealed to 0, the agent will use its own policy network to decide actions every time. -The process of gradual annealing of the greediness value is also the process in which the interaction between the agent and the environment gradually changes from exploration to utilization. The process of -greediness value marks the gradual evolution of the agent's interaction with the environment, transitioning from the initial exploration phase to the exploitation phase. - Greediness can further increase the number of random exploration samples of the agent, and make the agent's transition between exploration and utilization smoother, thereby optimizing the algorithm performance.

[0181] Compared with traditional manipulators, the present invention can decompose complex grasping tasks into multiple simple subtasks through the upper-level manipulator grasping gesture selection step and the lower-level manipulator wrist joint grasping alignment step. For example, the high-level strategy method determines which grasping posture to grasp, while the low-level strategy method is responsible for the wrist joint movement of the manipulator during the grasping process. This hierarchical structure enables the manipulator to make real-time adjustments in a constantly changing environment, improving the accuracy of grasping and operation. At the same time, the algorithm-based self-learning ability can continuously optimize the manipulator's grasping strategy, enhance its adaptability, and thus improve work efficiency.

[0182] like Fig. 9 As shown, in the second aspect of the present invention, a manipulator grasping alignment system comprises: an upper-layer manipulator grasping gesture selection unit, a lower-layer manipulator wrist joint grasping alignment unit and an execution unit; wherein:

[0183] The upper layer manipulator gripping gesture selection unit is at least used for: acquiring and extracting the outline of the target object based on the image of the target object; calculating the position state information and shape of the target object according to the outline of the target object; and determining the manipulator gripping gesture based on the position state information and shape of the target object;

[0184] The lower layer robot wrist joint grasping alignment unit is at least used to: use the position state information of the target object as the state space of the preset reinforcement learning algorithm, and output the robot wrist joint action angle;

[0185] The execution unit is at least used for: realizing the grasping alignment of the manipulator on the target object based on the grasping gesture of the manipulator and the action angle of the joint of the manipulator wrist.

[0186] Based on the same idea as the method in the above embodiment, the system provided by the present invention can implement the method in the above embodiment. For the convenience of explanation, the structural diagram of the system embodiment only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the system, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0187] A third aspect of the present invention discloses a readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the steps of the method described in any one of the above embodiments.

[0188] Computer-readable storage media may include: any entity or device capable of carrying a computer program, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc. A computer program includes a computer program code. The computer program code may be in source code form, object code form, an executable file, or some intermediate form, etc. A computer-readable storage medium may include: any entity or device capable of carrying a computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc.

[0189] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0190] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0191] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot gripping alignment method, characterized in that: include: The upper layer robot gripping gesture selection step and the lower layer robot wrist joint gripping alignment step; wherein: The upper-layer manipulator grasping gesture selection step includes: obtaining and extracting the outline of the target object based on the image of the target object; calculating the position state information and shape of the target object according to the outline of the target object; determining the manipulator grasping gesture based on the shape of the target object; wherein the image of the target object is obtained in real time through the camera on the manipulator; the manipulator grasping gesture includes gesture 1, gesture 2 and gesture 3; selecting the corresponding manipulator grasping gesture according to the shape of the target object; the relationship between the shape of the target object and the corresponding manipulator grasping gesture is: grasping gesture 1 is a five-finger pinching grasping gesture, corresponding to a rectangular parallelepiped; grasping gesture 2 is a five-finger pinching grasping gesture, corresponding to a cube; grasping gesture 3 is a five-finger grasping gesture, corresponding to a sphere; The lower-layer robot wrist joint grasping alignment step includes: using the position state information of the target object as the state space of the preset reinforcement learning algorithm, and outputting the robot wrist joint action angle; Based on the gripping gesture of the manipulator and the angle of the joint movement of the wrist, the manipulator is aligned with the target object; wherein: The method of using the position state information of the target object as the state space of the preset reinforcement learning algorithm and outputting the joint motion angle of the robotic wrist includes: Get the position status information of the target object, where the position status information of the target object is It indicates that, t For the moment, is the environmental state, They are t The relative horizontal and vertical coordinates of the geometric center point of the minimum circumscribed rectangle of the target object and the camera optical axis point in the pixel coordinate system at the moment are t The position state information of the target object obtained by the upper-level manipulator grasping gesture selection step at the moment As environmental status ; A double-delayed deep deterministic gradient strategy reinforcement learning algorithm is used as a preset reinforcement learning algorithm, and the double-delayed deep deterministic gradient strategy reinforcement learning algorithm is represented by a TD3 reinforcement learning algorithm; The position status information of the target object Input into the TD3 reinforcement learning algorithm and output a set of angle values , this group of angle values ​​is the joint action angle of the robot wrist, where They are the wrist rotation angle and wrist turning angle of the robotic wrist joint action respectively.

2. The robot gripping alignment method according to claim 1, characterized in that: The step of acquiring and extracting a contour of the target object based on the image of the target object includes: Preprocessing the image of the target object to obtain a preprocessed image of the target object; Performing edge detection on the preprocessed image of the target object to obtain an image of the target object after edge detection; Performing dilation processing on the image of the target object after edge detection to obtain the image of the target object after dilation processing; The contour of the target object is extracted from the image after the expansion process to obtain the contour of the target object.

3. The robot gripping alignment method according to claim 1, characterized in that: The step of calculating the position state information and shape of the target object according to the contour of the target object comprises: Obtain the geometric midpoint coordinates of the minimum circumscribed rectangle of the target object's outline and the ratio of the lengths of the two symmetry axes, as well as the roundness of the target object's outline; Obtaining the position state information of the target object by performing relative coordinate calculation on the geometric midpoint coordinates of the minimum circumscribed rectangle and the optical axis point coordinates in the camera coordinate system; and / or When the roundness of the target object's contour is greater than the roundness judgment threshold, the shape of the target object is judged to be a sphere; otherwise, it is a cuboid or a cube, wherein: when the ratio of the lengths of the two symmetry axes of the minimum circumscribed rectangle of the target object is greater than the axis length ratio judgment threshold, the shape of the target object is judged to be a cuboid, otherwise, it is a cube.

4. The robot gripping alignment method according to claim 1, characterized in that: The reward function of the TD3 reinforcement learning algorithm is set to give different reward values ​​according to the degree of alignment of the robotic wrist joint after the action is completed. , where the misalignment value is defined as: , The relative horizontal and vertical coordinates of the geometric center point of the minimum circumscribed rectangle of the target object and the camera optical axis point in the pixel coordinate system, respectively, and the misalignment degree value d The smaller the value, the higher the degree of alignment of the current robot. d The larger the value, the lower the current alignment.

5. The robot gripping alignment method according to claim 1, characterized in that: The position state information of the target object Input into the TD3 reinforcement learning algorithm and output a set of angle values ,in, They are the wrist rotation angle and wrist turning angle of the robot wrist joint action, including: exist t time: The agent receives the output of the upper-level robot grip gesture selection step t The position status information of the target object at the moment As environmental status ; The environmental status Input to the policy network of the agent Make a decision and output the two wrist joint action angles through the policy network As the movement angle of the robot wrist joint .

6. The robot gripping alignment method according to any one of claims 1 to 5, characterized in that: After the robot grasps and aligns the target object based on the robot grasping gesture and the robot wrist joint action angle, the method includes: The mechanical wrist joint motor performs the mechanical wrist joint movement angle After the execution is completed, the time becomes t +1 moment; exist t +1 time: The agent receives the output of the upper-level robot grip gesture selection step t +1 The position status information of the target object at time As environmental status ; The environmental status Substitute the preset reward function to obtain the joint action angle of the robotic wrist Reward value ; The four-tuple sample data Stored in the sample data pool.

7. The robot gripping alignment method according to claim 1, characterized in that: The TD3 reinforcement learning algorithm contains six networks: Two value networks, Two target value networks and one strategy network and a target policy network ; The training process of the TD3 reinforcement learning algorithm includes: Extract a batch of four-tuple sample data from the sample data pool ; , which means that the sample data will be extracted The input is fed into the target policy network and the output is the predicted action ; where the vector Each element of is independently derived from a truncated normal distribution Extract from the neural network, the subscript now indicates the current parameters of the neural network; Two target value networks are used for prediction: ； ； Calculate TD target: ； Two value networks are used for prediction: ； ； Calculate the TD error: ； ； Update the two value networks above: ； ； Every k Update the policy network and target network parameters once: Use the policy network to make predictions: , and then update the policy network: ； Update the target network parameters: ； ； ； in, are the parameters of the value network, are the parameters of the two target value networks respectively, are the policy network parameters and target policy network parameters respectively, The action value predictions made by the two value networks are: The predictions made by the two target value networks respectively, For TD target, are the TD errors of the two value networks, are the learning rates for updating the model parameters of the value network and the policy network, respectively. The updated parameters of the target network, is the discount rate, For The gradient of For θ The gradient of For a gradient.

8. A robot gripping alignment system, characterized in that: include: The upper layer robot gripping gesture selection unit, the lower layer robot wrist joint gripping alignment unit and the execution unit; wherein: The upper-layer manipulator grasping gesture selection unit is at least used for: acquiring and extracting the outline of the target object based on the image of the target object; calculating the position state information and shape of the target object according to the outline of the target object; determining the manipulator grasping gesture based on the shape of the target object; wherein the image of the target object is acquired in real time through the camera on the manipulator; the manipulator grasping gesture includes gesture 1, gesture 2 and gesture 3; selecting the corresponding manipulator grasping gesture according to the shape of the target object; the relationship between the shape of the target object and the corresponding manipulator grasping gesture is: grasping gesture 1 is a five-finger pinching grasping gesture, corresponding to a rectangular parallelepiped; grasping gesture 2 is a five-finger pinching grasping gesture, corresponding to a cube; grasping gesture 3 is a five-finger grasping gesture, corresponding to a sphere; The lower layer robot wrist joint grasping alignment unit is at least used to: use the position state information of the target object as the state space of the preset reinforcement learning algorithm, and output the robot wrist joint action angle; The execution unit is at least used to: realize the gripping alignment of the manipulator on the target object based on the gripping gesture of the manipulator and the action angle of the joint of the manipulator wrist; wherein: The method of using the position state information of the target object as the state space of the preset reinforcement learning algorithm and outputting the joint motion angle of the robotic wrist includes: Get the position status information of the target object, where the position status information of the target object is It indicates that, t For the moment, is the environmental state, They are t The relative horizontal and vertical coordinates of the geometric center point of the minimum circumscribed rectangle of the target object and the camera optical axis point in the pixel coordinate system at the moment are t The position state information of the target object obtained by the upper-level manipulator grasping gesture selection step at the moment As environmental status ; A double-delayed deep deterministic gradient strategy reinforcement learning algorithm is used as a preset reinforcement learning algorithm, and the double-delayed deep deterministic gradient strategy reinforcement learning algorithm is represented by a TD3 reinforcement learning algorithm; The position status information of the target object Input into the TD3 reinforcement learning algorithm and output a set of angle values ,in, They are the wrist rotation angle and wrist turning angle of the robotic wrist joint action respectively.

9. A readable storage medium storing a computer program, characterized in that: The computer program is used by a processor to execute the steps of the robot grasping alignment method as described in any one of claims 1-7.

Images