Object placement method and device based on mechanical arm, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an object placement method and device based on a mechanical arm, electronic equipment and a storage medium, and belongs to the technical field of imitation learning. The method comprises the steps of obtaining target expert teaching data for executing a target task; performing grabbing action information generation on the environment state image based on a grabbing action generation network to obtain grabbing action information; performing placement action information generation based on a placement action generation network to obtain first placement action information, the placement action generation network being a network constructed based on a twin network; inputting the first placement action information and the grabbing action information into a preset local structure learning network for position optimization to obtain second placement action information; and determining target placement action information based on the first placement action information and the second placement action information, and placing the target object according to the target placement action information. According to the embodiment of the invention, the object placement accuracy can be improved when the mechanical arm executes the task.

Description

Technical Field

[0001] The present application relates to the technical field of reinforcement learning and imitation learning, and in particular to a method and device for placing an object based on a robotic arm, an electronic device, and a storage medium. Background Art

[0002] A robotic arm is an automated mechanical device that can accurately locate a point in three-dimensional (or two-dimensional) space according to the instructions it receives to perform specific tasks. In order to enable the robotic arm to be better applied to complex tasks involving multiple action sequences such as assembly and stacking, the relevant technology usually learns the motion trajectory features from expert teaching tasks, builds a strategy learning model for multiple sequence actions such as grasping, classification, and assembly through a neural network, and controls the robotic arm to place objects in complex tasks based on the trained model.

[0003] However, in the process of training the action strategy learning model with related technologies, it is quite difficult to collect a large amount of high-precision expert teaching task data. If less expert teaching task data is used for model training, the generalization degree of the strategy model will be low, resulting in low accuracy of grasping and placing of the robotic arm and low success rate in executing multi-sequence tasks. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and device for object placement based on a robotic arm, an electronic device and a storage medium, which can achieve efficient learning of the motion trajectory characteristics of multi-sequence tasks while obtaining a small number of sample expert teaching task data, thereby improving the accuracy of object placement by the robotic arm and thus improving the success rate of executing multi-sequence tasks.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for placing an object based on a robotic arm, the method comprising:

[0006] Acquire target expert teaching data for performing a target task, wherein the target expert teaching data includes an environment state image, and the environment state image includes a target object required for performing the target task;

[0007] Generating grasping action information for the environment state image based on a preset grasping action generation network to obtain grasping action information of the target object;

[0008] Generating placement action information for the environment state image and the grabbing action information based on a preset placement action generation network to obtain first placement action information, wherein the placement action generation network is a network constructed based on a twin network;

[0009] Inputting the first placement action information and the grabbing action information into a preset local structure learning network for position optimization to obtain second placement action information;

[0010] Target placement action information is determined based on the first placement action information and the second placement action information, so as to control the robot arm to place the target object according to the target placement action information.

[0011] In some embodiments, the placement action generation network includes a first feature extraction subnetwork, a second feature extraction subnetwork, and a first correlation filtering subnetwork, and the placement action generation network based on a preset placement action generates placement action information on the environment state image and the grabbing action information to obtain the first placement action information, including:

[0012] Performing feature extraction on the environment state image based on the first feature extraction subnetwork to obtain first feature information;

[0013] Extracting features from the grasping action information based on the second feature extraction subnetwork to obtain second feature information;

[0014] Performing feature matching on the first feature information and the second feature information based on the first correlation filter subnetwork to obtain a first feature matching result;

[0015] The placement action information is generated based on the first feature matching result and the grabbing action information to obtain first placement action information.

[0016] In some embodiments, the first placement action information includes first placement pixel coordinates, the grabbing action information includes grabbing a target object image, the local structure learning network includes a third feature extraction subnetwork and a second correlation filtering subnetwork, and the first placement action information and the grabbing action information are input into a preset local structure learning network for position optimization to obtain the second placement action information, including:

[0017] Perform image extraction according to the first placement pixel coordinates to determine a local placement position image;

[0018] Performing feature extraction on the grasped target object image based on the third feature extraction subnetwork to obtain grasped target object image features;

[0019] Performing feature extraction on the local placement position image based on the third feature extraction subnetwork to obtain local placement position image features;

[0020] Performing feature matching on the captured target object image feature and the local placement position image feature based on a second correlation filter subnetwork to obtain a second feature matching result;

[0021] Position optimization is performed based on the second feature matching result to obtain second placement action information.

[0022] In some embodiments, the grasping action generation network includes a fully convolutional layer, and the training method of the grasping action generation network includes:

[0023] Acquire expert teaching training data, wherein the expert teaching training data includes a teaching environment state image and a teaching grasping action posture;

[0024] Performing coordinate system conversion on the posture of the teaching grasping action to obtain pixel coordinates of the teaching grasping action;

[0025] Extracting features of the teaching environment state image based on the full convolution layer to obtain teaching environment state image features;

[0026] Predicting the grasping action according to the teaching environment state image features and the teaching grasping action pixel coordinates to obtain multiple predicted grasping actions;

[0027] Based on a preset grasping action value function, obtaining a grasping action value corresponding to executing each of the predicted grasping actions, and obtaining a grasping action value corresponding to each pixel coordinate;

[0028] Based on a preset probability distribution function and the grasping action value, the grasping success rate corresponding to each pixel coordinate is calculated to obtain a probability distribution characteristic graph related to the grasping success rate;

[0029] Based on the pixel coordinates corresponding to the maximum grasping success rate in the probability distribution characteristic map related to the grasping success rate, the teaching grasping action information in the teaching environment state image is determined.

[0030] In some embodiments, the training method of the placement action generation network includes:

[0031] Acquire the teaching grasping action information and the teaching environment state image generated by the grasping action generation network;

[0032] Perform image extraction according to the teaching grasping action information to obtain a target object image;

[0033] Extracting features of the target object image to obtain target object image features;

[0034] Extracting fusion features of the teaching environment state image to obtain fusion features of the teaching environment state image;

[0035] Predicting a placement action based on the target object image features and the teaching environment state image fusion features to obtain a plurality of predicted placement actions;

[0036] Based on a preset placement action value function, obtaining a placement action value corresponding to executing each of the predicted placement actions, and obtaining a placement action value corresponding to each pixel coordinate;

[0037] Based on the probability distribution function and the placement action value, calculating the placement success rate corresponding to each pixel coordinate, and obtaining a probability distribution feature graph related to the placement success rate;

[0038] Based on the pixel coordinates corresponding to the maximum placement success rate in the probability distribution characteristic map related to the placement success rate, the teaching placement action information in the teaching environment state image is determined, and the teaching placement action information includes the teaching placement action pixel coordinates and the teaching placement action angle.

[0039] In some embodiments, the teaching environment state image includes a current teaching environment state image and a target teaching environment state image, the target teaching environment state image represents the environment state image after each action is performed, and the fusion feature extraction of the teaching environment state image to obtain the fusion feature of the teaching environment state image includes:

[0040] Extracting features of the current teaching environment state image to obtain features of the current teaching environment state image;

[0041] Extracting features of the target teaching environment state image to obtain features of the target teaching environment state image;

[0042] The current teaching environment state image feature and the target teaching environment state image feature are feature-fused to obtain the teaching environment state image fusion feature.

[0043] In some embodiments, the expert teaching training data includes a plurality of action sequences required to perform the target task, each of the action sequences corresponds to a reward value corresponding to the action sequence taken during the process of performing the target task, and the method for obtaining the expert teaching training data includes:

[0044] Training the grab action generation network and the placement action generation network based on the expert teaching training data to obtain a target training loss;

[0045] Calculating a sampling probability of each of the action sequences in executing the target task according to the target training loss and the reward value corresponding to each action sequence;

[0046] Based on the sampling probability, action sequences are sampled in the expert teaching training data, and the grasping action generation network and the placing action generation network are trained according to the sampled action sequences.

[0047] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes an object placement device based on a robotic arm, which is applied to the method described in the first aspect.

[0048] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0049] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0050] The object placement method and device, electronic device and storage medium based on a robotic arm proposed in the present application obtain target expert teaching data for performing a target task, the target expert teaching data includes an environment state image, and the environment state image includes a target object required to perform the target task; based on a preset grasping action generation network, grasping action information is generated for the environment state image to obtain grasping action information of the target object. Based on a preset placement action generation network, placement action information is generated for the environment state image and grasping action information to obtain first placement action information. Since when the robotic arm performs a target task, the execution success rate of the current action will affect the execution of subsequent actions, and ultimately affect the execution success rate of the entire task. Therefore, the generation of the first placement action information is based on the successful determination of the target object to be grasped and the successful execution of the grasping action. The placement action generation network in the present application uses the grasping action information as input to generate the first placement action information, ensuring the accuracy of the placement action and improving the success rate of performing the target task. Among them, the placement action generation network of the present application is a network constructed based on a twin network, which can match the characteristics of the grasped target object with the characteristics of the environmental state image, and can more efficiently learn the trajectory characteristics of the expert when performing the target task based on the target expert teaching data, thereby improving the efficiency of action strategy learning. Furthermore, the first placement action information and the grasping action information are generated and input into the preset local structure learning network for further local position optimization. On the basis of the first placement action information, the placement action is locally optimized according to the preliminarily determined first placement action information and grasping action information to obtain the second placement action information, thereby improving the accuracy of the placement action. Finally, the target placement action information is determined based on the first placement action information and the second placement action information, so as to control the robotic arm to place the target object according to the target placement action information. Therefore, the present application can further improve the success rate of the robotic arm in performing multi-sequence tasks on the premise of improving the accuracy of the placement action. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic diagram of a model framework corresponding to the object placement method provided in an embodiment of the present application;

[0052] Figure 2 is a flow chart of the object placement method provided in an embodiment of the present application;

[0053] Figure 3 is a schematic diagram of an environmental status image provided by an embodiment of the present application;

[0054] Figure 4 is a schematic diagram of a grab action generation network provided in an embodiment of the present application;

[0055] Figure 5 is a schematic diagram of a placement action generation network provided in an embodiment of the present application;

[0056] Figure 6 is a schematic diagram of a locally optimized action strategy learning model provided in an embodiment of the present application;

[0057] Figure 7 It is a schematic diagram of a model framework for target state fusion guidance provided in an embodiment of the present application;

[0058] Figure 8 is a schematic diagram of a target state fusion guided grasping action generation network provided in an embodiment of the present application;

[0059] Fig. 9 It is a schematic diagram of a target state fusion guided placement action generation network provided in an embodiment of the present application;

[0060] Fig.10 is a schematic diagram of an expert teaching data enhancement module provided in an embodiment of the present application;

[0061] Fig.11 is a structural schematic diagram of an object placement device based on a robotic arm provided in an embodiment of the present application;

[0062] Fig.12 It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] First, some nouns involved in this application are analyzed:

[0067] Reinforcement Learning (RL): Reinforcement learning is an important branch of machine learning, which aims to enable intelligent agents to learn the best decision-making strategy through interaction with the environment. In reinforcement learning, there are two basic elements: state and action. Performing an action in a certain state is a strategy. The purpose of reinforcement learning is to obtain the best strategy through continuous exploration and learning.

[0068] Fully Convolutional Networks (FCN): A neural network used for semantic segmentation that can adapt to input images of different sizes. The fully convolutional neural network is mainly divided into two parts: the full convolution part and the deconvolution part. Among them, the full convolution part is some classic convolutional neural networks (such as ResNet, etc.), which are used to extract features from the input image; the deconvolution part obtains a semantic segmentation image of the same size as the original input image by upsampling. The advantage of the fully convolutional neural network is that the convolutional layer structure replaces the fully connected layer of the convolutional neural network. Therefore, the output of the neural network is converted from image classification prediction to a pixel-level heat map.

[0069] Markov Decision Process (MDP): The robotic grasping and placing task can be expressed as a Markov decision process, usually represented by a four-tuple (S, A, P, R), where S represents the current state space, A represents the action space, P represents the transition probability, and R represents the reward function.

[0070] Deep Q-Network (DQN): A deep neural network is used as a function approximator to estimate the Q-value function. The input of the deep Q-network is the state of the current environment, and the output is the Q-value estimate corresponding to all possible actions. The deep Q-network can obtain the end-to-end mapping function Q(S, A; θ) from the state space S to the action space A.

[0071] Siamese Network: A Siamese network is composed of two or more identical networks. The basic idea is to input the input data into two identical neural networks at the same time, and the two networks share the same weights and parameters. By learning the representation of the input data in the two networks, the Siamese network can calculate the similarity between the two input data. The Q-value function is a measure of the expected value of the long-term cumulative return of performing an action in a given state.

[0072] Based on this, the embodiments of the present application provide a method and device for object placement based on a robotic arm, an electronic device and a storage medium, which can achieve efficient learning of the motion trajectory characteristics of multi-sequence tasks while obtaining a small number of sample expert teaching task data, thereby improving the accuracy of the robotic arm's grasping and placement, and thus improving the success rate of executing multi-sequence tasks.

[0073] The object placement method based on a robotic arm provided in the embodiment of the present application relates to the technical field of reinforcement learning and imitation learning. The object placement method based on a robotic arm provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements an image repair method, etc., but is not limited to the above forms.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] Before introducing the method provided in the embodiment of the present application in detail, the application scenario of the object placement method based on the robot arm provided in the present application is first described in detail. In the present application, the example tasks that the robot arm can perform are classification tasks, stacking tasks, matching tasks and assembly tasks. Classification tasks include classifying multiple common objects according to factors such as color or shape, or grabbing and placing specified objects in a container (such as a plate); stacking tasks include piling scattered blocks into a pyramid according to specified colors; matching tasks include putting objects of different shapes into molds of corresponding shapes; assembly tasks include assembling parts of some products, etc. The robot arm can perform more complex action tasks to ensure that the robot arm can be applied to more scenarios.

[0076] The object placement method based on a robotic arm provided in the embodiment of the present application is essentially to build an action strategy learning model based on imitation learning, and to control the robotic arm to perform a specific task through a trained action strategy learning network. The specific task includes multiple action sequences, mainly grasping actions and placing actions. The action strategy learning network learns the actions to perform tasks based on expert teaching data, which can reduce the exploration cost of the model.

[0077] In order to illustrate the action strategy learning model provided in the embodiment of the present application, the state space and the action space are first introduced.

[0078] The state space refers to the set of all possible visual states in the robot's working environment, and the robot can make corresponding action decisions based on the current state. For a specific task, the corresponding working environment is fixed. For example, for a pyramid stacking task, the number of objects required and the shape of the pyramid are fixed.

[0079] The state space provided in the embodiment of the present application is a visual perception image of the robot arm workspace when executing a multi-sequence task. t For example, the camera collects an RGB-D image (including RGB image and depth image) at time t to represent the current state s t RGB-D images contain color information and depth information, where RGB represents the information of the three color channels of red, green, and blue, and D represents the depth information, which can more accurately locate and identify the target object. Depth images provide distance information of objects, which can help solve problems such as lighting changes and improve the accuracy and robustness of target detection.

[0080] In some embodiments, in order to reduce the impact of environmental factors such as object occlusion, depth cameras are fixedly installed at 45° diagonally above, 45° diagonally above the left side, and 45° diagonally above the right side of the experimental platform for performing multi-sequence tasks to obtain RGB-D images from different perspectives.

[0081] Furthermore, the RGB image and the depth map are converted into the form of a 3D point cloud, and then the 3D point cloud is orthogonally projected and converted into a height map looking down in the direction of gravity. Finally, the height maps from different perspectives are fused and reconstructed into a global projection form. The method for collecting environmental state images provided in the embodiment of the present application can overcome the problem of appearance distortion caused by collecting images with ordinary cameras, so that the action strategy learning model can learn the complete shape features and color features of objects. It should be noted that the teaching environment state images in the expert teaching training data for training the action strategy learning model are also obtained through the above collection method, which will not be repeated here.

[0082] The action space is the set of all possible actions that the robot can take in each environment state. The action space contains the grasping action a pick and place action a place , the placement action is premised on the successful execution of the grab action.

[0083] When collecting expert teaching training data and testing the action strategy learning model, the grasping action and placement action are represented by the posture of the end effector of the robot arm (including the grasping action posture and the placement action posture). The expert teaching training data can be randomly generated through the simulation environment corresponding to the specific task to avoid the interference of environmental noise, or the trajectory of the task can be collected through the physical robot arm.

[0084] Compared with the related art that saves the expert teaching task training data in the form of video to learn the semantic features of the continuous motion trajectory of the expert teaching training data, the embodiment of the present application uses the robot arm itself as a teaching expert to collect the grasping action and the placement action. The motion trajectory is autonomously planned by the inverse solution algorithm module of the robot arm, that is, the position and corresponding visual perception image of the end effector of the robot arm when performing the grasping and placement actions. There is no need to collect the motion trajectory from the grasping position to the placement position, and the entire teaching data collection process is more efficient.

[0085] In the training phase of the action strategy learning model, the grasping action pose and placement action pose of the robot arm in the robot arm coordinate space in the expert teaching task training data are converted into the pixel coordinates (u, v) and angle θ of the environment state image in the camera coordinate space, and the pixel coordinates (u, v) and angle θ of the environment state image are used as the model input. In the testing phase of the action strategy learning model, the environment state image of the target expert teaching data is used as the model input, and the grasping action and placement action are predicted according to the action strategy learning model to determine the optimal grasping pixel coordinates and placement pixel coordinates (u, v) and placement action angle θ, and then converted to the pose (x, y, z, θ) of the end effector of the robot arm.

[0086] The trained action strategy learning model can generate the optimal action strategy for executing the target task based on the environment state image of the workspace, and output the best grasping action and placement action. The mapping relationship between the environment state image and the action strategy is as follows:

[0087] f(o t )→a t =(T pick ,T place )∈Α

[0088] Among them, Α represents all possible action spaces predicted by the action strategy learning model, o t represents the environment state image, a t represents an action space predicted based on the environment state image at time t, T pick is the position of the end effector (e.g. vacuum gripper) of the robot arm when performing the grasping action (x pick ,y pick ,z pick ,θ pick ), T place is the position of the vacuum gripper at the end of the robot arm when performing the placement action (x place ,y place ,z place ,θ place ).

[0089] In some embodiments, see Figure 1 , Figure 1 It is a schematic diagram of a model framework corresponding to the object placement method provided in the embodiment of the present application. The action strategy learning model provided in the embodiment of the present application includes an expert teaching data enhancement module, a global grasping and placing action learning module, and a local structure learning network. The expert teaching data enhancement module is used to perform reward optimization sampling and data enhancement on the collected expert teaching training data to obtain the target expert data. The method of reward optimization sampling will be described in detail later.

[0090] Specifically, the global grasping and placing action learning module includes a grasping action generation network and a placing action generation network. The grasping action generation network is a grasping action Q-value function network built based on a fully convolutional neural network, which is used to learn the strategic features of grasping actions based on expert teaching training data, and then apply the trained grasping action generation network to a new working environment to test the performance of the network.

[0091] The global grasping and placing action learning module also includes a placing action generation network. The placing action generation network is a dual network model constructed by a dual-stream feedforward fully convolutional network. The placing action generation network can perform feature template matching on the features of the grasped target object and the features of the global environment state. The global environment state features are the image features of the teaching environment state image in the model testing phase and the image features of the environment state image in the testing phase. A similarity comparator can be established between the features of the determined grasping target object and the features of the global environment state, and the optimal placement strategy can be derived by combining the placement strategy provided in the expert teaching training data.

[0092] The detailed structures of the grab action generation network and the placement action generation network will be described later. The local structure learning network includes a third feature extraction subnetwork and a second correlation filter subnetwork, and the local structure learning network will be described in detail later.

[0093] In some embodiments, see Figure 2 , Figure 2 201 to 205:

[0094] Step 201, obtaining target expert teaching data for performing a target task;

[0095] Step 202, generating grasping action information for the environment state image based on a preset grasping action generation network to obtain grasping action information of the target object;

[0096] Step 203, generating placement action information based on the environment state image and the grabbing action information based on a preset placement action generation network to obtain first placement action information;

[0097] Step 204, inputting the first placement action information and the grabbing action information into a preset local structure learning network for position optimization to obtain second placement action information;

[0098] Step 205 , determining target placement action information based on the first placement action information and the second placement action information, so as to control the robot arm to place the target object according to the target placement action information.

[0099] Specifically, based on the action strategy learned from the expert teaching training data, the optimal action strategy for executing the target task can be derived from the environment state image, wherein the target task set by the expert teaching training data is the same as the target task set by the target expert teaching data.

[0100] In step 201 of some embodiments, the target expert teaching data is the expert teaching data required in the model testing and model application process. It should be noted that the expert teaching training data is used to train the action strategy learning model, and the target expert teaching data is used to test the action strategy learning model. The training set and the test set are both expert teaching data for performing the same target task, but the expert teaching data of the training set is different from the expert teaching data of the test set. The method for collecting the target expert teaching data has been introduced in the previous steps and will not be repeated here.

[0101] Specifically, the target expert teaching data includes an environment state image, which represents the working area when the robot arm performs the target task and a global observation image of the target object required to perform the target task. For example, if a classification task needs to be performed, a current visual image of objects of different shapes or colors on the experimental platform is obtained to display the characteristics of the target object and its placement in the task execution area.

[0102] In some embodiments, see Figure 3 , Figure 3 It is a schematic diagram of the environment status image provided by the embodiment of the present application. The environment status image 300 includes an RGB height map 301 and a depth height map 302. The smaller rectangles of different colors in the RGB height map 301 represent the target objects that need to be grasped and placed to perform the task, and the brown ones represent the execution area of ​​the target task. In the depth height map 302, the yellow solid box is the corresponding target object in the RGB height map 301, and the yellow wireframe is the execution area of ​​the target task.

[0103] Specifically, the pixel resolution of the environment state image is 240×240, and each pixel can represent a 3.125×3.125mm three-dimensional space. During the testing and application process, the action strategy learning model can generate the optimal grasping and placing actions under the current environment state based on the environment state image and the action strategy features taught by experts learned in the training phase.

[0104] In step 202 of some embodiments, the preset grasping action generation network can generate the optimal grasping action under the current environment state according to the environment state image, and obtain grasping action information. The grasping action information is represented by the grasping action pixel coordinates (u, v)∈o in the environment state image. t The end effector of the robot arm used in the embodiment of the present application for grasping is a vacuum gripper, and there is no need to consider the angle θ of the grasping action. pick .

[0105] In addition, the vacuum gripper used in this application can also avoid collision problems caused by the close arrangement of target objects and poor grasping angles when performing grasping actions. Therefore, only after the target object to be grasped is determined and the optimal placement action position is calculated through the placement action generation network, the optimal placement action angle θ needs to be considered. place .

[0106] After determining the pixel coordinates of the grasping action, the coordinate system of the pixel coordinates of the grasping action can be converted according to the hand-eye calibration method of the robot arm and the camera, that is, converted from the camera coordinate system to the robot arm coordinate system to obtain the three-dimensional coordinates of the robot arm end position (x pick ,y pick ,z pick ).

[0107] The grasping action generation network is essentially a pixel-level representation action Q-value function network, as shown in the following formula:

[0108]

[0109] Among them, Q pick ((u,v)|o t ) represents the environment state image o t In the example, the grasping action value function Q pick The pixel coordinates (u, v) corresponding to the maximum value of T pick is the 3D pose coordinate of the end effector of the robot arm (e.g. vacuum gripper) when performing the grasping action. t In , the pixel distribution of grasping actions corresponding to successful grasps can be multimodal, especially when there are multiple instances of the same target object in the work scenario where the target task is performed, or when the shape of the target object is symmetric.

[0110] Furthermore, the target object is grasped according to the determined grasping action pixel coordinates (u, v), and the characteristic information of the grasped target object is obtained according to the grasping action pixel coordinates. The characteristic information of the target object is the image of the environment state with the grasping action pixel coordinates as the center. t The target object image φ(o t [T pick ]). In the embodiment of the present application, the size of the captured target object image is 64x64, and the feature information of the captured target object will be used as an input of the placement action generation network.

[0111] In some embodiments, the training method of the grasping action generation network includes but is not limited to the following steps:

[0112] Obtain expert teaching training data, which includes teaching environment state images and teaching grasping action postures;

[0113] Perform coordinate system transformation on the posture of the taught grasping action to obtain the pixel coordinates of the taught grasping action;

[0114] Based on the full convolution layer, feature extraction is performed on the teaching environment state image to obtain the teaching environment state image features;

[0115] Predicting the grasping action according to the teaching environment state image features and the pixel coordinates of the teaching grasping action to obtain multiple predicted grasping actions;

[0116] Based on the preset grasping action value function, the grasping action value corresponding to each predicted grasping action is obtained, and the grasping action value corresponding to each pixel coordinate is obtained;

[0117] Based on the preset probability distribution function and the grasping action value, the grasping success rate corresponding to each pixel coordinate is calculated to obtain a probability distribution feature map related to the grasping success rate;

[0118] Based on the pixel coordinates corresponding to the maximum grasping success rate in the probability distribution characteristic map related to the grasping success rate, the teaching grasping action information in the teaching environment state image is determined.

[0119] In this embodiment, the grasping action generation network is actually an action Q-value function network constructed based on a fully convolutional neural network, that is, the Q-value function is estimated by a fully convolutional neural network. In the model training stage, the input of the grasping action generation network is the expert teaching training data, and the expert teaching training data is the trajectory data of the end of the robotic arm that performs the target task optimally. The training goal of the grasping action generation network is to learn the mapping relationship from the teaching environment state image features to the teaching grasping action pixel coordinates, and continuously adjust the internal weight parameters by predicting the Q value of the action to minimize the error between the predicted grasping action pixel coordinates and the actual teaching grasping action pixel coordinates.

[0120] Specifically, the expert teaching training data includes the teaching environment state image and the teaching grasping action posture. The teaching environment state image is the visual perception image of the teaching expert when performing the target task. t , also including RGB height map and depth height map.

[0121] The teaching grasping action posture is the posture (x, y, z) of the vacuum gripper at the end of the robot arm when performing the grasping action. The teaching grasping action posture is transformed into a coordinate system to obtain the corresponding teaching grasping action pixel coordinates (u, v) in the height map. Furthermore, the teaching environment state image is extracted through the full convolution layer in the grasping action generation network to obtain the teaching environment state image features. The grasping action is predicted based on the teaching environment state image features and the teaching grasping action pixel coordinates to obtain multiple predicted grasping actions.

[0122] Specifically, the target task to be executed includes multiple action sequences. At each moment of executing the target task, the grasping action generation network generates a grasping action based on the current environment state image o t and all possible grasping actions a pick , and predict the action value corresponding to each predicted grasping action, that is, the Q value, and select the grasping action with the largest Q value for execution. After executing the grasping action, the environmental state of the workspace will be updated, and the predicted grasping action and the corresponding Q value will be recorded and stored to accumulate experience. In addition, the action sequence target state fusion algorithm can also be introduced to stabilize the training process of the grasping action generation network, and the following steps will be introduced in detail.

[0123] Furthermore, the predicted grasping action value and the teaching environment state image features are mapped to obtain the grasping action value corresponding to the teaching grasping action pixel coordinates in the teaching environment state image, that is, the Q value of the predicted grasping action corresponding to each teaching grasping pixel coordinate. Through the above steps, the grasping action generation network learns the action strategy characteristics of executing the target task in the process of approximating the maximum action Q value through the fully convolutional neural network.

[0124] The grasping action generation network provided in the embodiment of the present application takes the robot arm expert teaching training data as input, and converts the prediction of the optimal grasping action posture of the robot arm into a prediction problem of the pixel-level Q-value function, thereby improving the efficiency of grasping action strategy learning, and ultimately achieving grasping of the target object by outputting the optimal grasping pixel coordinates.

[0125] In this embodiment, the preset probability distribution function is a Softmax function, and the pixel-level probability distribution operation is performed by the Softmax function. Based on the Softmax function and the grasping action value corresponding to each predicted grasping action, the grasping success rate corresponding to each teaching grasping pixel coordinate is calculated. According to the grasping success rate corresponding to each teaching grasping pixel coordinate in the teaching environment state image, a probability distribution characteristic diagram related to the grasping success rate is obtained.

[0126] Specifically, the probability distribution feature map related to the grasping success rate is a pixel heat map, that is, the teaching environment state image o t The pixel thermal value is used to represent the grasping success rate corresponding to the current pixel coordinate. The higher the thermal value, the greater the success rate of grasping action performed at the current pixel coordinate position.

[0127] Based on the pixel coordinates of the grasping action corresponding to the maximum grasping success rate in the probability distribution feature map, the final result is determined in the teaching environment state image o t The teaching grasping action information in the , the teaching grasping action information represents the grasping action strategy learned from the expert teaching training data, specifically the teaching grasping action pixel coordinates (u pick ,v pick ).

[0128] In some embodiments, see Figure 4 , Figure 4 is a schematic diagram of a grab action generation network provided in an embodiment of the present application. It can be understood that, Figure 4 A schematic diagram of the network test and application phase for the grasping action generation, that is, a schematic diagram of the grasping action generation network after training is completed. In the test and application phase, the teaching grasping action information ( Figure 4 (not shown) to perform the target task.

[0129] Specifically, the grasping action generation network is an hourglass codec architecture constructed by a 43-layer residual network (Residual Network, ResNet). The residual network contains 12 residual blocks, which can accelerate the training process of the network. It also includes a splicing module and a probability distribution function (Softmax function).

[0130] Furthermore, the residual network includes a full convolution layer (encoder) and a deconvolution layer (decoder). The full convolution layer is used to extract features from the input environment state image, and the deconvolution layer obtains a pixel-level semantic segmentation image with the same size and resolution as the input environment state image through upsampling. The full convolution layer specifically includes a 1*1 convolution layer with a convolution kernel width and height of 1 pixel, a Rectified Linear Unit (ReLU) activation function, a 2*2 convolution layer with a convolution kernel width and height of 2 pixels, and an identity block.

[0131] In order to minimize the potential resolution loss of the network, the convolution stride is set to 8. The convolution stride in the encoder is 2, and in the decoder there are 3 layers of bilinear upsampling layers (UpSampling). In addition, each convolution layer after the first layer has a dilated convolution and is interleaved with the ReLU activation function before the last layer to increase the prediction receptive field coverage of each pixel.

[0132] During the training phase, the input of the grasping action generation network is the teaching environment state image and the teaching grasping action posture. In the actual application process, the input of the grasping action generation network is the environment state image. The grasping action generation network can generate grasping actions according to the environment state image for the new application scenario where the target task is located, and output the optimal grasping action information under the current environment state, that is, the grasping action pixel coordinates. Furthermore, the image of the grasping target object is obtained by image capture based on the grasping action information as the input of the placement action generation network.

[0133] In step 303 of some embodiments, the placement action generation network is an action Q value function network constructed based on a twin neural network, wherein the twin neural network includes two identical network structures, each of which is the same as the network structure of the grabbing action generation network. The placement action generation network can generate placement action information based on the environment state image and the grabbing action information generated by the grabbing action network to obtain the first placement action information.

[0134] Specifically, one input of the placement action generation network is the captured target object image; the other input is the same as the input of the capture action generation network, which is the environment state image. The first action placement information includes the optimal placement action pixel coordinates (u place ,v place ) and placement angle θ place The coordinate system of the pixel coordinates of the placement action is transformed to obtain the first end position information of the robot arm. The first end position information of the robot arm is the position (x place ,y place ,zplace ,θ place ).

[0135] In some embodiments, the training method of the placement action generation network includes but is not limited to the following steps:

[0136] Acquire the teaching grasping action information and the teaching environment state image generated by the grasping action generation network;

[0137] Image extraction is performed with the pixel coordinates of the grasping action as the center to obtain the image of the target object;

[0138] Extracting features of the target object image to obtain the target object image features;

[0139] Acquire the teaching grasping action information and the teaching environment state image generated by the grasping action generation network;

[0140] Perform image extraction based on the taught grasping action information to obtain the target object image;

[0141] Extracting features of the target object image to obtain the target object image features;

[0142] Extract fusion features of the teaching environment state image to obtain fusion features of the teaching environment state image;

[0143] Predict placement actions based on the image features of the target object and the fusion features of the teaching environment state image to obtain multiple predicted placement actions;

[0144] Based on a preset placement action value function, the placement action value corresponding to executing each predicted placement action is obtained to obtain the placement action value corresponding to each pixel coordinate;

[0145] Based on the probability distribution function and the placement action value, the placement success rate corresponding to each pixel coordinate is calculated to obtain a probability distribution feature map related to the placement success rate;

[0146] Based on the pixel coordinates corresponding to the maximum placement success rate in the probability distribution characteristic map related to the placement success rate, the teaching placement action information in the teaching environment state image is determined, and the teaching placement action information includes the teaching placement action pixel coordinates and the teaching placement action angle.

[0147] In this embodiment, the training goal of the placement action generation network is to learn the placement action strategy features for executing the target task based on the teaching environment state image features and the teaching grasping action information, that is, to learn the mapping relationship from the teaching environment state image features to the teaching grasping action pixel coordinates.

[0148] Specifically, the teaching grasping action information and the teaching environment state image generated by the grasping action generation network are obtained, and the teaching grasping action information includes the target object image obtained by image extraction with the teaching grasping action pixel coordinates as the center.

[0149] During the training phase of the placement action generation network, one input of the placement action generation network is the captured target object image, and the other input is the same as the input of the capture action generation network, which is the teaching environment state image. Feature extraction is performed on the target object image to obtain the target object image features. Feature extraction is performed on the teaching environment state image to obtain the teaching environment state image features. The twin neural network uses the teaching environment state image features as feature templates and performs feature template matching with the target object image features.

[0150] The placement action is predicted based on the image features of the target object and the image features of the teaching environment state to obtain multiple predicted placement actions. The placement action value corresponding to each predicted placement action is calculated based on the preset placement action value function, that is, the Q value corresponding to each predicted placement action is calculated.

[0151] Furthermore, similar to the output of the grabbing action generation network, the placement action generation network can also output a pixel heat map related to the placement success rate. Based on the Softmax function, a pixel-level probability distribution operation is performed to calculate the placement success rate corresponding to each pixel coordinate, and a probability distribution feature map related to the placement success rate is obtained. Based on the teaching placement pixel coordinates corresponding to the maximum placement success rate (i.e., the maximum probability) in the probability distribution feature map related to the placement success rate, the teaching placement action information in the teaching environment state image is determined. In addition, the teaching placement action information also includes the teaching placement action angle.

[0152] Next, the method for estimating the angle of the teaching placement action will be introduced in detail. In an embodiment of the present application, the method for estimating the angle of the placement action is optimized on the Siam-BM Tracker model, which is a visual target tracking model based on a twin neural network. The Siam-BM Tracker model rotates the candidate matching target k times, while the matching template is not rotated. The embodiment of the present application rotates the matching template k times. Since the size of the matching template is smaller than the candidate matching target, the placement angle estimation method provided in the embodiment of the present application can improve the calculation speed.

[0153] Specifically, the placement action angle estimation method provided in the embodiment of the present application includes two processes: defining the image features of the captured target object as query features, and discretizing the query features into 72 angles, so as to achieve fine adjustment of the placement action angle; further, through the cross-correlation function in the correlation filter module, the correlation response values ​​of the 72 rotation angle features and the global visual features of the captured target object are calculated respectively, and the calculation formula is as follows:

[0154] Q place (τ|o t ,T pick )=γφ(o t [T pick ])*ψ(o t )[τ]+b

[0155] Among them, ψ(o t )[τ] represents the global visual feature map, φ(o t [T pick ]) represents the query feature map, i.e., the features of the captured target object, Q place (τ|o t ,T pick ) represents the maximum Q value of the placement action obtained in the state of the grasping action information corresponding to the rotation angle τ in the environment state image. The scalar parameters γ and b are introduced to make the score range suitable for logistic regression, and their values ​​can be defined according to actual needs.

[0156] The final output of the placement action generation network is the pixel-level action Q value related to the placement success rate, calculated as follows:

[0157]

[0158] Among them, Q place (τ i |o t ,T pick ) represents the environment state image o t In the process, the grasping action T is executed pick Under the condition of place The pixel coordinates (u, v) corresponding to the maximum value of i is the rotation angle corresponding to the i-th rotation, T place It is the 3D pose coordinate of the robot end effector (such as a vacuum gripper) when performing the placement action.

[0159] In some embodiments, see Figure 5 , Figure 5 is a schematic diagram of a placement action generation network provided in an embodiment of the present application. It can be understood that, Figure 5A schematic diagram of the network test and application phase for the placement action generation is generated, that is, a schematic diagram of the placement action generation network after training is completed. In the test and application phase, the teaching placement action information ( Figure 5 (not shown) to perform the target task.

[0160] Specifically, the placement action generation network includes a first feature extraction subnetwork and a second feature extraction subnetwork. The first feature extraction subnetwork and the second feature extraction subnetwork constitute a twin neural network. The two share weight parameters. The twin neural network is used to extract the features of the captured target object and the features of the environmental state image to compare the similarity of the features.

[0161] Furthermore, the network structure of the first feature extraction subnetwork and the second feature extraction subnetwork is the same. Taking the first feature extraction subnetwork as an example, the first feature extraction subnetwork is composed of a residual network with 8 steps and 43 layers, and the last layer has no nonlinear activation function. Specifically, the residual network includes a full convolution layer and a deconvolution layer, and the full convolution layer specifically includes a 1X1 convolution layer, a ReLU activation function, an identity block, and a 2X2 convolution layer. In addition, the first feature extraction subnetwork also includes a probability distribution function, namely the Softmax function. Among them, the functions of each part in the network are the same as those of each part in the grasping action generation network, which will not be repeated here.

[0162] In step 203 of some embodiments, placing action information is generated based on the environment state image and the grabbing action information based on a preset placing action generation network, and obtaining the first placing action information includes but is not limited to the following steps:

[0163] Extract features from the environment state image based on the first feature extraction subnetwork to obtain first feature information;

[0164] Based on the second feature extraction sub-network, feature extraction is performed on the grasping action information to obtain second feature information;

[0165] Perform feature matching on the first feature information and the second feature information based on the first correlation filter subnetwork to obtain a first feature matching result;

[0166] The placement action information is generated based on the first feature matching result and the grabbing action information to obtain the first placement action information.

[0167] Specifically, the placement action generation network includes a first feature extraction subnetwork and a second feature extraction subnetwork. Based on the first feature extraction subnetwork, feature extraction is performed on the environment state image to obtain first feature information, where the first feature information is the global visual perception state of the workspace. tBased on the convolutional layer of the second feature extraction subnetwork, feature extraction is performed on the grasping action information to obtain second feature information, where the second feature information is feature information of the grasped target object determined by the grasping action generation network.

[0168] The placement action generation network also includes a first correlation filter subnetwork, which is a correlation filter module. The correlation filter module is used to match the two features output by the twin neural network to determine the first placement action information under global visual observation. The first placement action information is specifically the first placement action pixel coordinates (u1, v1) and the placement action angle θ1. The detailed matching steps and angle estimation method have been introduced in the previous steps and will not be repeated here.

[0169] In step 204 of some embodiments, the first placement action information and the grabbing action information are input into a preset local structure learning network for position optimization to obtain the second placement action information, including but not limited to the following steps:

[0170] Perform image extraction according to the first placement pixel coordinates to determine a local placement position image;

[0171] Based on the third feature extraction sub-network, feature extraction is performed on the image of the grasped target object to obtain the image features of the grasped target object;

[0172] Perform feature extraction on the local placement position image based on the third feature extraction sub-network to obtain local placement position image features;

[0173] Perform feature matching on the captured target object image features and the local placement position image features based on the second correlation filter subnetwork to obtain a second feature matching result;

[0174] The position is optimized based on the second feature matching result to obtain second placement action information.

[0175] Specifically, the first placement action information includes the first placement pixel coordinates, and the local placement position image with a size of s×s is cropped with the first placement pixel coordinates as the center. In the embodiment of the present application, the value of s is 128. Please continue to refer to Figure 2 , Figure 2 The schematic diagram of the composition of the local structure learning network is shown in FIG. 4 , which includes a third feature extraction subnetwork and a second correlation filtering subnetwork.

[0176] It should be noted that the placement action generation network and the local structure learning network are independent during the training period, but the same expert teaching training data is used in the training of a single target task, and the training input of the local structure learning network depends on the output of the placement action generation network.

[0177] In some embodiments, see Figure 6 , Figure 6 It is a schematic diagram of the locally optimized action strategy learning model provided in the embodiment of the present application. In the global perception stage, the captured target object image is rotated and mapped to four feature spaces through the twin neural network to improve the speed of action strategy inference; and the global environment visual feature extraction is performed on the depth height map and RGB height map under the current environment state, and finally the full perception prediction result (u1, v1, θ1) is output.

[0178] The local structure learning network has two inputs. One input is the local placement position image; the other input is the grasping target object image determined by the grasping action generation network. The rotation mapping space of the grasped target object image consists of 72 rotation angles, which can reduce the deviation of the placement action pixel coordinates and optimize the placement action angle.

[0179] The output of the placement action generation network represents the output of the global perception stage, specifically the first placement action pixel coordinates (u1, v1) and the first placement action angle θ1 corresponding to the global placement action position, that is, the full perception prediction result (u1, v1, θ1). Further, the local structure learning network is used to perform local perception optimization to determine the second placement action pixel coordinates (u2, v2) and the second placement action angle θ2 corresponding to the local placement action position, that is, to obtain the local perception prediction result (u2, v2, θ2).

[0180] In this embodiment, the local structure learning network includes a third feature extraction subnetwork, based on which feature extraction is performed on the captured target object image to obtain the captured target object image features; based on the third feature extraction subnetwork, feature extraction is performed on the local placement position image to obtain the local placement position image features. The structure of the third feature extraction subnetwork is the same as that of the first feature extraction subnetwork, and will not be repeated here.

[0181] Furthermore, based on the second correlation filter subnetwork, feature matching is performed on the image features of the captured target object and the image features of the local placement position to obtain a second feature matching result, and the position is optimized based on the second feature matching result to obtain the second placement action information. The second correlation filter subnetwork has the same working principle as the first correlation filter subnetwork, and the output of the local structure learning network is the placement action pixel coordinates (u2, v2) and the placement action angle θ2 that maximize the placement action Q value function. Furthermore, the coordinate system can be transformed according to the placement action pixel coordinates (u2, v2) and the placement action angle θ2 to obtain the optimized position and posture of the end effector of the robotic arm to control the robotic arm to place the target object.

[0182] In step 205 of some embodiments, the target placement action information is determined based on the first placement action information obtained by the placement action generation network and the second placement action information determined by the local structure learning network. The first placement action information is (u1, v1) and θ1, the second placement action information is (u2, v2) and θ2, and the target placement action information is

[0183] The robot arm is controlled to place the target object according to the target placement action information. The target placement action information can be converted into a coordinate system to obtain the execution posture of the end of the robot arm to place the target object.

[0184] In some embodiments, extracting features from the teaching environment state image to obtain teaching environment state image features includes but is not limited to the following steps:

[0185] Extract features of the current teaching environment state image to obtain features of the current teaching environment state image;

[0186] Extracting features of the target teaching environment state image to obtain features of the target teaching environment state image;

[0187] The current teaching environment state image features and the target teaching environment state image features are fused to obtain the teaching environment state image fusion features.

[0188] In this application, when learning the grab and place strategy from the expert teaching data of multiple sequence actions, it is necessary to consider not only the accuracy of the single action strategy, but also the impact of the single action on the overall task. Each step of the action strategy affects the global strategy, and the actions with lower sequence levels directly affect the learning effect of the subsequent action strategy. Therefore, when learning the action strategy of multiple sequence tasks from expert teaching task data, the optimal action strategy characteristics in the current state can be learned, and the grab and place can be successfully performed, thereby effectively achieving the next optimal target state and ultimately achieving the overall goal of the task. Based on this, the embodiment of the present application provides a target state fusion guidance method.

[0189] Specifically, in the current visual state, observe o t Based on this, an additional target state element o is introduced g For visual guidance. g From the current state s t The present application embodiment provides three target state sampling methods, that is, three target teaching environment state image sampling methods. Among them, the current state s t The next state s t+1 , a random state s of the future state sequence i(t<i<k), and the final state s of a single task sequence k , where k represents the total length of actions in a single task sequence.

[0190] Furthermore, the key to training the action strategy learning model guided by the sequence target state fusion is to fuse the image data. There are generally two ways to process image data, which are divided into early data-level fusion and late feature-level fusion. The early data-level fusion method is to first fuse the current state visual image and the target state visual image according to different image channels, and then extract features.

[0191] In this embodiment, the current teaching environment state image and the target teaching environment state image are data fused and then input into the grasping action generation network for training. The data fusion method can learn the grasping action strategy characteristics of the fusion state through the grasping action generation network to improve the success rate of the grasping action.

[0192] Another image data processing method is to extract the features of the environment state image respectively, and fuse the extracted features using the Hadamard Product method. In the embodiment of the present application, the first feature extraction sub-network of the placement action generation network is used to extract the features of the current teaching environment state image to obtain the features of the current teaching environment state image, and the second feature extraction sub-network of the placement action generation network is used to extract the features of the target teaching environment state image to obtain the features of the target teaching environment state image, and then the features of the current teaching environment state image and the features of the target teaching environment state image are fused to obtain the features of the teaching environment state image in the fused state.

[0193] The feature fusion method can combine the grasping action information output by the grasping action generation network to extract the features of the grasped target object under the feature fusion environment state image, and then input it into the relevant filtering module to output the optimal placement action information, thereby improving the success rate of the placement action.

[0194] In some embodiments, see Figure 7 , Figure 7 is a schematic diagram of a model framework for target state fusion guidance provided by an embodiment of the present application. Specifically, for the grasping action generation network, the current teaching environment state image o t and the target teaching environment state image o t+1 Perform data fusion and input the data fusion result into the grasping action generation network. It can be understood that the target state element o g Including the target teaching environment state image o t+1 .

[0195] For the placement action generation network, the current teaching environment state image ot Perform feature extraction and teach the target environment state image o t+1 Feature extraction is performed, the two extracted features are fused, and the feature fusion result and the grasping action information obtained by the grasping action generation network are input into the first correlation filter submodule for similarity comparison. Figure 7 The current teaching environment status image in o t and the target teaching environment state image o t+1 Both include an RGB height map and a depth height map (not shown in the figure).

[0196] In some embodiments, see Figure 8 , Figure 8 : is a schematic diagram of a target state fusion guided grasping action generation network provided in an embodiment of the present application. The formula for data fusion is as follows:

[0197] Q pick (o t ,o g )=Φ pick (o t )☉Φ goal (o g )

[0198] Among them, Q pick (o t ,o g ) is the current teaching environment state image o t and the target teaching environment state image o t+1 The Q value of the grasping action under fusion, ☉ indicates feature fusion, specifically the Hadamard product, Φ pick (o t ) represents the features of the target object captured in the teaching environment state image, Φ goal (o g ) represents the state image o of the target teaching environment t+1 The features of the target object to be grasped in the process are finally output, and the action pixel prediction related to the grasping success rate is finally output. The optimal grasping action position pixel coordinates are generated by the following formula:

[0199]

[0200] in, Indicates the environment state image o t and the target environment state image o g Under the fusion condition, the pixel coordinate corresponding to the maximum value of the grasp action value function, T pick It is the three-dimensional pose coordinate of the end effector of the robot arm (such as a vacuum gripper) when performing a grasping action.

[0201] In some embodiments, see Fig. 9 , Fig. 9 It is a schematic diagram of the target state fusion guided placement action generation network provided in the embodiment of the present application. As mentioned in the above steps, the placement action generation network is built based on the twin network, and the placement action strategy generated by the global visual state is optimized through the local structure learning network, including the optimization of the placement action pixel coordinates and the placement action angle.

[0202] Therefore, the learning of the placement action strategy can be divided into two stages. The first stage is to learn the strategy based on the global visual state, and the second stage is to refine the action strategy of the first stage through the local visual state. g When , only the first stage of target fusion is performed, because the first stage is the global perception stage, the goal is to select the best strategy to reach the next target state according to the guidance of the target state, and the second stage is just the optimization of the first stage strategy.

[0203] Specifically, the learning process of the target state fusion-guided placement action generation network is divided into three steps: the current teaching environment state image can be extracted by the first feature extraction subnetwork and the second feature extraction subnetwork of the twin neural network respectively. t and the target teaching environment state image o t+1 The state features are fused, and the feature-level fusion formula is as follows:

[0204] φ fuse (o t )=ψ place (o t )☉ψ goal (o t )

[0205] Among them, ☉ represents feature fusion, specifically the Hadamard product, ψ place (o t ) represents the teaching environment status image o t The characteristics of the target object placed in goal (o g ) represents the image features of the target teaching environment state.

[0206] Furthermore, the local structure learning network performs placement action optimization, and the placement action value function formula is as follows:

[0207] Q place (Δτ|o t ,T place )=γφ crop (o t )[T place ]*φ fuse (o t)[Δτ]+b

[0208] Among them, the pixel coordinates of the grab action (u pick ,v pick ) In the feature fusion state, the corresponding feature φ of the captured target object crop (o t )[T pick ], and then through the first correlation filter sub-network, the feature φ of the target object is captured crop (o t )[T pick ] and the global state fusion feature φ fuse (o t ) to perform cross-convolution matching to obtain the pixel coordinates corresponding to the global perception placement action under the global state feature fusion.

[0209] The image is cropped with this coordinate as the center to obtain the environmental state image feature φ under the global state fusion feature crop (o t )[T place ], Δτ represents the optimized angle value of the placement action obtained by local optimization, Q place (Δτ|o t ,T place ) represents the placement action information T obtained by global perception prediction place And teaching environment status image o t The Q value of the placement action corresponding to the estimated placement action angle deviation under the condition. The scalar parameters γ and b are introduced to make the score range suitable for logistic regression, and their values ​​can be defined according to actual needs.

[0210] Furthermore, the local perception prediction result (x', y', z') output by the final local structure learning network represents the end-arm position after the placement action is optimized. The optimized second placement action information is obtained by the following formula:

[0211]

[0212] Among them, T on the left side of the equation place represents the optimized placement action information, which is the placement action information T obtained by global perception prediction on the right side of the equation place And teaching environment status image o t Under the condition, among the predicted multiple placement action optimization angle values, the placement action optimization angle value corresponding to the maximum action Q value is obtained. According to the optimized placement action angle, the placement action angle in the placement action information obtained by global perception prediction is adjusted.

[0213] In some embodiments, the method for obtaining expert teaching training data further includes:

[0214] The grasping action generation network and the placing action generation network are trained based on the expert teaching training data to obtain the target training loss;

[0215] Calculate the sampling probability of each action sequence in executing the target task based on the target training loss and the reward value corresponding to each action sequence;

[0216] Action sequences are sampled in expert teaching training data based on sampling probability, and the grasping action generation network and the placing action generation network are trained according to the sampled action sequences.

[0217] Specifically, in deep learning model training, it is generally believed that different label samples have almost the same contribution value to model training. However, in the object placement method based on a robotic arm provided in an embodiment of the present application, the expert teaching task data contains multiple action sequences, and the difficulty coefficient of the action strategy increases with the increase in the number of task sequences, especially assembly tasks and stacking tasks. When the number of executed action sequences increases, the predictable space for placing actions becomes smaller, and the previous action strategy will affect subsequent actions, which will lead to difficulties in learning placement strategies. Combined with the idea of ​​active learning, different sample data in the data set have different values. On this basis, the action strategy model can be trained with data of higher values, or the model can be trained with less sample data, so as to obtain better performance.

[0218] Therefore, in order to learn more valuable strategic knowledge from small sample expert teaching task data, the embodiment of the present application optimizes the sample sampling method in the action strategy model training process, and designs a sample sampling method based on teaching action rewards. The expert teaching training data includes multiple action sequences required to perform the target task, and each action sequence corresponds to the reward value corresponding to the action sequence in the process of performing the target task. In other words, in the process of collecting the action trajectory of the expert teaching data, the greater the reward value of an action, the greater the probability that the sequence action is sampled for training and updating the action strategy model. The failed behavior during the expert guidance period has no knowledge contribution to the update of the action strategy model, so it is not sampled. The sampling probability calculation formula for the expert teaching data provided in the embodiment of the present application is as follows:

[0219]

[0220] Where L represents the target training loss of the entire action strategy model obtained during the training of the grab action generation network and the placement action generation network for a single target task, and its initial value is 1; r irepresents the reward value when the action sequence level is i in a single teaching task, N represents the total number of teaching action sequences in a single target task; p(i) represents the probability of the action being sampled in the i-th sequence. As can be seen from the formula, as the number of training iterations increases, the loss gradually decreases and gradually approaches The sampling method is changed to uniform sampling.

[0221] The embodiment of the present application effectively solves the problem that uniform sampling can only obtain limited action strategy knowledge, is more valuable for optimizing the action strategy learning model, and can efficiently train the model with small sample expert teaching data.

[0222] For further information, see Fig.10 , Fig.10 It is a schematic diagram of the expert teaching data enhancement module provided in the embodiment of the present application. The expert teaching data enhancement module includes a reward optimization sampling module and a random rigid transformation module. The reward optimization sampling module is used to sample the initial expert teaching training data according to the aforementioned reward optimization sampling method and sampling formula, and perform random rigid transformation on the obtained valuable expert teaching data to obtain more valuable teaching data, and finally obtain expert teaching training data to train and update the action strategy model.

[0223] Please refer to Fig.11 The embodiment of the present application further provides an object placement device based on a robotic arm, which can implement the above-mentioned object placement method based on a robotic arm, and the device includes:

[0224] An acquisition unit 1110 is used to acquire target expert teaching data for performing a target task, the target expert teaching data including an environment state image, the environment state image including a target object required for performing the target task; a first generation unit 1120 is used to generate grasping action information for the environment state image based on a preset grasping action generation network, and obtain grasping action information of the target object; a second generation unit 1130 is used to generate placement action information for the environment state image and grasping action information based on a preset placement action generation network, and obtain first placement action information, the placement action generation network being a network constructed based on a twin network; an optimization unit 1140 is used to input the first placement action information and the grasping action information into a preset local structure learning network for position optimization, and obtain second placement action information; a determination unit 1150 is used to determine target placement action information based on the first placement action information and the second placement action information, so as to control the robotic arm to place the target object according to the target placement action information.

[0225] The specific implementation of the robot-based object placement device is basically the same as the specific implementation of the robot-based object placement method described above, and will not be repeated here.

[0226] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned object placement method based on the robotic arm when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0227] See also Fig.12 , Fig.12 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0228] The processor 1201 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0229] The memory 1202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1202, and the processor 1201 calls and executes the object placement method based on the robot arm in the embodiments of this application;

[0230] Input / output interface 1203, used to implement information input and output;

[0231] The communication interface 1204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0232] A bus 1205 that transmits information between various components of the device (e.g., the processor 1201, the memory 1202, the input / output interface 1203, and the communication interface 1204);

[0233] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0234] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned robot-based object placement method is implemented.

[0235] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0236] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0237] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0238] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0239] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0240] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0241] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0242] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0243] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0244] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0245] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0246] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for placing an object based on a robotic arm, characterized in that: The method comprises: Acquire target expert teaching data for performing a target task, wherein the target expert teaching data includes an environment state image, and the environment state image includes a target object required for performing the target task; Generating grasping action information for the environment state image based on a preset grasping action generation network to obtain grasping action information of the target object; Generating placement action information for the environment state image and the grabbing action information based on a preset placement action generation network to obtain first placement action information, wherein the placement action generation network is a network constructed based on a twin network; Inputting the first placement action information and the grabbing action information into a preset local structure learning network for position optimization to obtain second placement action information; Target placement action information is determined based on the first placement action information and the second placement action information, so as to control the robot arm to place the target object according to the target placement action information.

2. The method according to claim 1, characterized in that: The placement action generation network includes a first feature extraction subnetwork, a second feature extraction subnetwork and a first correlation filtering subnetwork. The placement action generation network based on the preset placement action generation network generates placement action information for the environment state image and the grabbing action information to obtain first placement action information, including: Performing feature extraction on the environment state image based on the first feature extraction subnetwork to obtain first feature information; Extracting features from the grasping action information based on the second feature extraction subnetwork to obtain second feature information; Performing feature matching on the first feature information and the second feature information based on the first correlation filter subnetwork to obtain a first feature matching result; The placement action information is generated based on the first feature matching result and the grabbing action information to obtain first placement action information.

3. The method according to claim 2, characterized in that The first placement action information includes first placement pixel coordinates, the grabbing action information includes grabbing a target object image, the local structure learning network includes a third feature extraction subnetwork and a second correlation filtering subnetwork, and the first placement action information and the grabbing action information are input into a preset local structure learning network for position optimization to obtain second placement action information, including: Perform image extraction according to the first placement pixel coordinates to determine a local placement position image; Performing feature extraction on the grasped target object image based on the third feature extraction subnetwork to obtain grasped target object image features; Performing feature extraction on the local placement position image based on the third feature extraction subnetwork to obtain local placement position image features; Performing feature matching on the captured target object image feature and the local placement position image feature based on a second correlation filter subnetwork to obtain a second feature matching result; Position optimization is performed based on the second feature matching result to obtain second placement action information.

4. The method according to claim 1, characterized in that: The grasping action generation network includes a full convolutional layer, and the training method of the grasping action generation network includes: Acquire expert teaching training data, wherein the expert teaching training data includes a teaching environment state image and a teaching grasping action posture; Performing coordinate system conversion on the posture of the teaching grasping action to obtain pixel coordinates of the teaching grasping action; Extracting features of the teaching environment state image based on the full convolution layer to obtain teaching environment state image features; Predicting the grasping action according to the teaching environment state image features and the teaching grasping action pixel coordinates to obtain multiple predicted grasping actions; Based on a preset grasping action value function, obtaining a grasping action value corresponding to executing each of the predicted grasping actions, and obtaining a grasping action value corresponding to each pixel coordinate; Based on a preset probability distribution function and the grasping action value, the grasping success rate corresponding to each pixel coordinate is calculated to obtain a probability distribution characteristic graph related to the grasping success rate; Based on the pixel coordinates corresponding to the maximum grasping success rate in the probability distribution characteristic map related to the grasping success rate, the teaching grasping action information in the teaching environment state image is determined.

5. The method according to claim 4, characterized in that The training method of the placement action generation network includes: Acquire the teaching grasping action information and the teaching environment state image generated by the grasping action generation network; Perform image extraction according to the teaching grasping action information to obtain a target object image; Extracting features of the target object image to obtain target object image features; Extracting fusion features of the teaching environment state image to obtain fusion features of the teaching environment state image; Predicting a placement action based on the target object image features and the teaching environment state image fusion features to obtain a plurality of predicted placement actions; Based on a preset placement action value function, obtaining a placement action value corresponding to executing each of the predicted placement actions, and obtaining a placement action value corresponding to each pixel coordinate; Based on the probability distribution function and the placement action value, calculating the placement success rate corresponding to each pixel coordinate, and obtaining a probability distribution feature graph related to the placement success rate; Based on the pixel coordinates corresponding to the maximum placement success rate in the probability distribution characteristic map related to the placement success rate, the teaching placement action information in the teaching environment state image is determined, and the teaching placement action information includes the teaching placement action pixel coordinates and the teaching placement action angle.

6. The method according to claim 5, characterized in that The teaching environment state image includes a current teaching environment state image and a target teaching environment state image, wherein the target teaching environment state image represents an environment state image after each action is performed, and the fusion feature extraction of the teaching environment state image to obtain the fusion feature of the teaching environment state image includes: Extracting features of the current teaching environment state image to obtain features of the current teaching environment state image; Extracting features of the target teaching environment state image to obtain features of the target teaching environment state image; The current teaching environment state image feature and the target teaching environment state image feature are feature-fused to obtain the teaching environment state image fusion feature.

7. The method according to claim 4, characterized in that The expert teaching training data includes a plurality of action sequences required to perform the target task, each of the action sequences corresponds to a reward value corresponding to the action sequence taken during the process of performing the target task, and the method for obtaining the expert teaching training data includes: Training the grab action generation network and the placement action generation network based on the expert teaching training data to obtain a target training loss; Calculating a sampling probability of each of the action sequences in executing the target task according to the target training loss and the reward value corresponding to each action sequence; Based on the sampling probability, action sequences are sampled in the expert teaching training data, and the grasping action generation network and the placing action generation network are trained according to the sampled action sequences.

8. A method and device for placing an object by a robot arm, characterized in that: The device comprises: An acquisition unit, used for acquiring target expert teaching data for performing a target task, wherein the target expert teaching data includes an environment state image, and the environment state image includes a target object required for performing the target task; A first generating unit, configured to generate grasping action information for the environment state image based on a preset grasping action generating network, so as to obtain grasping action information of the target object; A second generating unit is used to generate placement action information for the environment state image and the grabbing action information based on a preset placement action generating network to obtain first placement action information, wherein the placement action generating network is a network constructed based on a twin network; an optimization unit, configured to input the first placement action information and the grabbing action information into a preset local structure learning network for position optimization to obtain second placement action information; A determination unit is used to determine target placement action information based on the first placement action information and the second placement action information, so as to control the robot arm to place the target object according to the target placement action information.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the robot-based object placement method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the robot-arm-based object placement method according to any one of claims 1 to 7 is implemented.

Images