Mechanical arm-based object placement method and device, electronic device, and storage medium

Abstract

Embodiments of the present application provide a kind of object placement method and device based on mechanical arm, electronic equipment and storage medium, belong to the technical field of imitation learning.The method comprises: obtaining target expert teaching data for executing target task;Based on the generation of grabbing action information of environment state image by grabbing action generation network, obtain grabbing action information;Based on the generation of placing action information, obtain first placing action information, and placing action generation network is the network based on twin network construction;First placing action information and grabbing action information are input into the position optimization of pre-set local structure learning network, and second placing action information is obtained;Determine target placing action information based on first placing action information and second placing action information, and place target object according to target placing action information.The accuracy of object placement when mechanical arm executes task can be improved in the embodiments of the present application.

Description

Technical Field

[0001] This application relates to the fields of reinforcement learning and imitation learning technology, and in particular to a method and apparatus for object placement based on a robotic arm, an electronic device, and a storage medium. Background Technology

[0002] A robotic arm is an automated mechanical device capable of precisely positioning itself at a point in three-dimensional (or two-dimensional) space according to received instructions to perform a specific task. To enable robotic arms to be better applied to complex tasks involving multiple sequences of actions, such as assembly and stacking, the relevant technologies typically learn motion trajectory features from expert-trained tasks, construct strategy learning models for multiple sequences of actions such as grasping, classification, and assembly using neural networks, and control the robotic arm to place objects in complex tasks based on the trained model.

[0003] However, in the process of training the action policy learning model, it is quite difficult to collect a large amount of high-precision expert teaching task data. Furthermore, if less expert teaching task data is used for model training, the generalization degree of the policy model will be low, resulting in low accuracy of robotic arm grasping and placement, and a low success rate in executing multi-sequence tasks. Summary of the Invention

[0004] The main objective of this application is to propose a method and apparatus for object placement based on a robotic arm, an electronic device, and a storage medium that can efficiently learn the motion trajectory features of multiple sequence tasks even when acquiring a small amount of expert teaching task data, thereby improving the accuracy of the robotic arm in placing objects and increasing the success rate of executing multiple sequence tasks.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for object placement based on a robotic arm, the method comprising:

[0006] Acquire target expert teaching data for performing the target task, the target expert teaching data including an environmental state image, the environmental state image including the target object required to perform the target task;

[0007] Based on a preset grasping action generation network, grasping action information is generated from the environmental state image to obtain the grasping action information of the target object.

[0008] The placement action information is generated based on the environmental state image and the grasping action information by a preset placement action generation network to obtain the first placement action information. The placement action generation network is a network built based on Siamese network.

[0009] The first placement action information and the grasping action information are input into a preset local structure learning network for position optimization to obtain the second placement action information;

[0010] Target placement action information is determined based on the first placement action information and the second placement action information, so as to control the robotic arm to place the target object according to the target placement action information.

[0011] In some embodiments, the placement action generation network includes a first feature extraction subnetwork, a second feature extraction subnetwork, and a first correlation filtering subnetwork. The placement action generation network generates placement action information based on the environmental state image and the grasping action information to obtain first placement action information, including:

[0012] Based on the first feature extraction sub-network, feature extraction is performed on the environmental state image to obtain first feature information;

[0013] Based on the second feature extraction sub-network, feature extraction is performed on the grasping action information to obtain the second feature information;

[0014] Based on the first correlation filter sub-network, feature matching is performed on the first feature information and the second feature information to obtain the first feature matching result;

[0015] Based on the first feature matching result and the grasping action information, placement action information is generated to obtain the first placement action information.

[0016] In some embodiments, the first placement action information includes first placement pixel coordinates, the grasping action information includes a grasped target object image, the local structure learning network includes a third feature extraction subnetwork and a second correlation filtering subnetwork, and the step of inputting the first placement action information and the grasping action information into a preset local structure learning network for position optimization to obtain second placement action information includes:

[0017] Image extraction is performed based on the first placement pixel coordinates to determine the local placement position image;

[0018] Based on the third feature extraction sub-network, feature extraction is performed on the image of the target object to be captured, and the image features of the target object are obtained.

[0019] Based on the local placement position image of the third feature extraction subnetwork, feature extraction is performed to obtain the local placement position image features;

[0020] Based on the second correlation filter subnetwork, feature matching is performed on the image features of the captured target object and the image features of the local placement position to obtain the second feature matching result;

[0021] Position optimization is performed based on the second feature matching result to obtain the second placement action information.

[0022] In some embodiments, the grasping action generation network includes fully convolutional layers, and the training method of the grasping action generation network includes:

[0023] Acquire expert teaching training data, which includes images of the teaching environment and poses of the teaching grasping action;

[0024] The teaching grasping action pose is transformed by coordinate system to obtain the pixel coordinates of the teaching grasping action;

[0025] Based on the fully convolutional layer, feature extraction is performed on the teaching environment state image to obtain the teaching environment state image features;

[0026] Based on the features of the teaching environment state image and the pixel coordinates of the teaching grasping action, grasping action prediction is performed to obtain multiple predicted grasping actions;

[0027] Based on the preset grasping action value function, the grasping action value corresponding to each predicted grasping action is obtained, and the grasping action value corresponding to each pixel coordinate is obtained.

[0028] Based on the preset probability distribution function and the value of the grasping action, the grasping success rate corresponding to each pixel coordinate is calculated, and a probability distribution feature map related to the grasping success rate is obtained.

[0029] Based on the pixel coordinates corresponding to the maximum value of the grasping success rate in the probability distribution feature map related to the grasping success rate, the teaching grasping action information in the teaching environment state image is determined.

[0030] In some embodiments, the training method for the placement action generation network includes:

[0031] Acquire the teaching grasping action information and the teaching environment state image generated by the grasping action generation network;

[0032] Based on the taught grasping action information, image extraction is performed to obtain the target object image;

[0033] Feature extraction is performed on the target object image to obtain the target object image features;

[0034] The teaching environment state image is subjected to fusion feature extraction to obtain the teaching environment state image fusion feature;

[0035] Based on the fusion features of the target object image and the teaching environment state image, a placement action is predicted to obtain multiple predicted placement actions.

[0036] Based on the preset placement action value function, the placement action value corresponding to each predicted placement action is obtained, and the placement action value corresponding to each pixel coordinate is obtained.

[0037] Based on the probability distribution function and the placement action value, the placement success rate corresponding to each pixel coordinate is calculated to obtain a probability distribution feature map related to the placement success rate.

[0038] Based on the pixel coordinates corresponding to the maximum placement success rate in the probability distribution feature map related to the placement success rate, the teaching placement action information in the teaching environment state image is determined. The teaching placement action information includes the pixel coordinates of the teaching placement action and the angle of the teaching placement action.

[0039] In some embodiments, the teaching environment state image includes a current teaching environment state image and a target teaching environment state image, wherein the target teaching environment state image represents the environment state image after each action is performed, and the step of extracting fusion features from the teaching environment state image to obtain teaching environment state image fusion features includes:

[0040] Feature extraction is performed on the current teaching environment state image to obtain the current teaching environment state image features;

[0041] Feature extraction is performed on the target teaching environment state image to obtain the target teaching environment state image features;

[0042] The current teaching environment state image features and the target teaching environment state image features are fused to obtain the teaching environment state image fusion features.

[0043] In some embodiments, the expert teaching training data includes multiple action sequences required to perform the target task, each action sequence corresponding to a reward value taken during the execution of the target task, and the method for obtaining the expert teaching training data includes:

[0044] The grasping action generation network and the placement action generation network are trained based on the expert teaching training data to obtain the target training loss;

[0045] The sampling probability of each action sequence in the execution of the target task is calculated based on the target training loss and the reward value corresponding to each action sequence.

[0046] Based on the sampling probability, action sequences are sampled from the expert teaching training data, and the grasping action generation network and the placement action generation network are trained according to the sampled action sequences.

[0047] To achieve the above objectives, a second aspect of the present application provides an object placement device based on a robotic arm, which is applied to the method described in the first aspect.

[0048] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0049] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0050] This application proposes a robotic arm-based object placement method, apparatus, electronic device, and storage medium. It acquires target expert teaching data for performing a target task. This target expert teaching data includes an environmental state image, which includes the target object required to perform the target task. Based on a preset grasping action generation network, grasping action information is generated from the environmental state image to obtain grasping action information for the target object. Then, based on a preset placement action generation network, placement action information is generated from the environmental state image and the grasping action information to obtain first placement action information. Since the success rate of the current action affects the execution of subsequent actions when the robotic arm performs the target task, ultimately affecting the overall success rate of the task, the generation of the first placement action information is predicated on successfully identifying the target object to be grasped and successfully executing the grasping action. The placement action generation network in this application uses the grasping action information as input to generate the first placement action information, ensuring the accuracy of the placement action and improving the success rate of performing the target task. The placement action generation network in this application is based on a Siamese network, which can match the features of the grasped target object with the features of the environmental state image. It can more efficiently learn the trajectory features of experts performing target tasks based on expert teaching data, thus improving the efficiency of action strategy learning. Furthermore, the generated first placement action information and grasping action information are input into a preset local structure learning network for further local position optimization. Based on the first placement action information, the placement action is locally optimized according to the initially determined first placement action information and grasping action information to obtain second placement action information, improving the accuracy of the placement action. Finally, the target placement action information is determined based on the first and second placement action information, and the robotic arm is controlled to place the target object according to the target placement action information. Therefore, this application can further improve the success rate of the robotic arm performing multi-sequence tasks while improving the accuracy of the placement action. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of a model frame corresponding to the object placement method provided in the embodiments of this application;

[0052] Figure 2 This is a flowchart of an object placement method provided in an embodiment of this application;

[0053] Figure 3 This is a schematic diagram of an environmental state image provided in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of a crawling action generation network provided in an embodiment of this application;

[0055] Figure 5 This is a schematic diagram of the placement action generation network provided in an embodiment of this application;

[0056] Figure 6 This is a schematic diagram of a locally optimized action strategy learning model provided in an embodiment of this application;

[0057] Figure 7 This is a schematic diagram of a model framework for target state fusion guidance provided in an embodiment of this application;

[0058] Figure 8 This is a schematic diagram of a target state fusion guided grasping action generation network provided in an embodiment of this application;

[0059] Figure 9 This is a schematic diagram of the target state fusion guided placement action generation network provided in an embodiment of this application;

[0060] Figure 10 This is a schematic diagram of the expert teaching data enhancement module provided in an embodiment of this application;

[0061] Figure 11 This is a schematic diagram of a robotic arm-based object placement device provided in an embodiment of this application;

[0062] Figure 12 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0066] First, let's analyze some of the terms used in this application:

[0067] Reinforcement Learning (RL) is an important branch of machine learning that aims to enable an agent to learn optimal decision-making strategies through interaction with its environment. Reinforcement learning involves two basic elements: states and actions. Performing a certain action in a given state constitutes a strategy. The goal of reinforcement learning is to continuously explore and learn to obtain the optimal strategy.

[0068] Fully Convolutional Networks (FCNs) are neural networks used for semantic segmentation, capable of adapting to input images of varying sizes. A fully convolutional NNN mainly consists of two parts: a fully convolutional part and a deconvolutional part. The fully convolutional part, often found in classic convolutional neural networks (such as ResNet), extracts features from the input image; the deconvolutional part upsamples the input image to obtain a semantically segmented image of the same size as the original input image. The advantage of FCNs lies in replacing the fully connected layers of a traditional convolutional neural network with convolutional layers, thus transforming the neural network's output from image classification prediction into a pixel-level heatmap.

[0069] Markov Decision Process (MDP): Robot grasping and placement tasks can be represented as Markov decision processes, usually denoted by a quadruple (S, A, P, R), where S represents the current state space, A represents the action space, P represents the transition probability, and R represents the reward function.

[0070] Deep Q-Network (DQN): This type of network uses a deep neural network as a function approximator to estimate the Q-value function. The input to a DQN is the current state of the environment, and the output is the Q-value estimate for all possible actions. A DQN can obtain an end-to-end mapping function Q(S, A; θ) from the state space S to the action space A.

[0071] Siamese Network: A Siamese network consists of two or more identical networks. The basic idea is to simultaneously feed input data into two identical neural networks, which share the same weights and parameters. By learning the representation of the input data in these two networks, the Siamese network can calculate the similarity between the two input data sets. The Q-value function is a measure of the expected long-term cumulative reward for performing an action in a given state.

[0072] Based on this, embodiments of this application provide a method and apparatus for object placement based on a robotic arm, an electronic device, and a storage medium, which can efficiently learn the motion trajectory features of multi-sequence tasks even when acquiring a small amount of expert teaching task data, thereby improving the accuracy of robotic arm grasping and placement, and thus increasing the success rate of performing multi-sequence tasks.

[0073] The object placement method based on a robotic arm provided in this application relates to the fields of reinforcement learning and imitation learning technologies. This robotic arm-based object placement method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing image restoration methods, but is not limited to the above forms.

[0074] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0075] Before detailing the methods provided in the embodiments of this application, the application scenarios of the object placement method based on a robotic arm provided in this application will be described in detail. In this application, the robotic arm can perform tasks such as classification, stacking, matching, and assembly. Classification tasks include, for example, classifying multiple common objects according to factors such as color or shape, or grasping and placing specified objects into a container (e.g., a plate); stacking tasks include, for example, stacking scattered blocks into a pyramid according to specified colors; matching tasks include, for example, placing objects of different shapes into molds of corresponding shapes; and assembly tasks include, for example, assembling parts of some products. The robotic arm can perform more complex motion tasks, ensuring that it can be applied to more scenarios.

[0076] The object placement method based on a robotic arm provided in this application essentially constructs a motion policy learning model based on imitation learning. The trained motion policy learning network controls the robotic arm to perform a specific task, which includes multiple action sequences, primarily grasping and placing actions. The motion policy learning network learns the actions to perform the task based on expert teaching data, reducing the model's exploration cost.

[0077] To illustrate the action strategy learning model provided in the embodiments of this application, the state space and action space are first introduced.

[0078] State space refers to the set of all possible visual states in the working environment of a robotic arm, which can make corresponding action decisions based on the current state. For a specific task, the corresponding working environment is fixed. For example, in a pyramid stacking task, the number of objects required and the shape of the pyramid are fixed.

[0079] The state space provided in this embodiment is a visually perceived image of the robotic arm's workspace when performing multiple sequence tasks. t For example, an RGB-D image (including RGB image and depth map) at time t is acquired using a camera to represent the current state s. t RGB-D images contain color and depth information, where RGB represents the information from the red, green, and blue color channels, and D represents the depth information. This allows for more accurate location and identification of target objects. Depth images provide distance information to objects, helping to address issues such as changes in lighting conditions and improving the accuracy and robustness of object detection.

[0080] In some embodiments, in order to reduce the influence of environmental factors such as object occlusion, depth cameras are fixedly installed at positions 45° above, 45° above to the left, and 45° above to the right of the experimental platform for performing multi-sequence tasks, respectively, to obtain RGB-D images from different perspectives.

[0081] Furthermore, the RGB image and depth map are converted into 3D point clouds, and then the 3D point clouds are orthogonally projected to convert them into a height map viewed from above in the gravity direction. Finally, the height maps from different perspectives are fused to reconstruct a global projection. The environmental state image acquisition method provided in this application overcomes the appearance distortion problem caused by images acquired by ordinary cameras, enabling the action policy learning model to learn complete object shape and color features. It should be noted that the teaching environment state images in the expert teaching training data used to train the action policy learning model are also obtained using the above acquisition method, and will not be elaborated upon here.

[0082] The motion space refers to the set of all possible actions a robotic arm can take in any given environmental state. The motion space includes the grasping action (a). pick and placement action a place The placement action is contingent upon the successful execution of the grab action.

[0083] When collecting expert teaching training data and testing the motion strategy learning model, the grasping and placing actions are represented by the poses of the robotic arm's end effector (including the grasping pose and the placing pose). Expert teaching training data can be randomly generated through a simulation environment corresponding to a specific task to avoid interference from environmental noise, or the trajectory of the task can be collected by a physical robotic arm.

[0084] Compared to related technologies that save expert teaching task training data as videos to learn the semantic features of continuous motion trajectories in the expert teaching training data, this application embodiment uses the robotic arm itself as the teaching expert to collect grasping and placing actions. The motion trajectory is autonomously planned by the robotic arm's inverse kinematics algorithm module. That is, the pose of the robotic arm's end effector and the corresponding visual perception image when performing grasping and placing actions are obtained, without the need to collect the motion trajectory from the grasping position to the placing position. The entire teaching data acquisition process is more efficient.

[0085] During the training phase of the motion policy learning model, the grasping and placement poses of the robotic arm in the expert teaching task training data are converted into pixel coordinates (u,v) and angle θ of the environmental state image in the camera coordinate space. These environmental state image pixel coordinates (u,v) and angle θ are then used as model input. During the testing phase of the motion policy learning model, the environmental state image from the target expert teaching data is used as model input. The motion policy learning model predicts the grasping and placement actions, determines the optimal grasping and placement pixel coordinates (u,v) and placement angle θ, and then converts them to the pose (x,y,z,θ) of the robotic arm's end effector.

[0086] The trained action policy learning model can generate the optimal action policy for performing the target task based on the environmental state image of the workspace, and output the best grasping and placing actions. The mapping relationship between the environmental state image and the action policy is as follows:

[0087] f(o t )→a t =(T pick ,T place )∈Α

[0088] Where A represents the space of all possible actions predicted by the action policy learning model, o t Represents an environmental state image, a t T represents a specific action space predicted based on the environmental state image at time t. pick It is the pose (x) of the end effector (e.g., vacuum gripper) of the robotic arm when performing a grasping action. pick ,y pick ,z pick ,θ pick ), T place It is the pose (x) of the vacuum gripper at the end of the robotic arm during the placement action. place ,y place ,z place ,θ place ).

[0089] In some embodiments, see Figure 1 , Figure 1 This is a schematic diagram of a model framework corresponding to the object placement method provided in this application embodiment. The action strategy learning model provided in this application embodiment includes an expert teaching data augmentation module, a global grasping and placement action learning module, and a local structure learning network. The expert teaching data augmentation module is used to perform reward-optimized sampling and data augmentation on the collected expert teaching training data to obtain target expert data. The reward-optimized sampling method will be described in detail later.

[0090] Specifically, the global grasping and placement action learning module includes a grasping action generation network and a placement action generation network. The grasping action generation network is a grasping action Q-value function network built on a fully convolutional neural network. It is used to learn the policy features of grasping actions based on expert teaching training data. Then, the trained grasping action generation network is applied to a new working environment to test the network's performance.

[0091] The global grasping and placement action learning module also includes a placement action generation network. This network is a dual-network model constructed from a two-stream feedforward fully convolutional network. The placement action generation network performs feature template matching between the features of the grasped target object and the global environment state features. During the model testing phase, the global environment state features are the image features of the teaching environment state image; during the testing phase, they are the image features of the environment state image. A similarity comparator can be established between the determined features of the grasped target object and the features of the global environment state. Combined with the placement strategy provided in the expert teaching training data, the optimal placement strategy can be derived.

[0092] The detailed structures of the grasping action generation network and the placement action generation network will be explained later. The local structure learning network includes a third feature extraction subnetwork and a second correlation filtering subnetwork, which will be explained in detail later.

[0093] In some embodiments, see Figure 2 , Figure 2 This is a flowchart of an object placement method provided in an embodiment of this application. The object placement method based on a robotic arm provided in this embodiment includes, but is not limited to, the following steps 201 to 205:

[0094] Step 201: Obtain target expert teaching data for performing the target task;

[0095] Step 202: Generate grasping action information from the environmental state image based on the preset grasping action generation network to obtain the grasping action information of the target object.

[0096] Step 203: Based on the preset placement action generation network, place action information is generated from the environmental state image and the grasping action information to obtain the first placement action information;

[0097] Step 204: Input the first placement action information and the grasping action information into a preset local structure learning network for position optimization to obtain the second placement action information;

[0098] Step 205: Determine target placement action information based on the first placement action information and the second placement action information, so as to control the robotic arm to place the target object according to the target placement action information.

[0099] Specifically, based on the action strategies learned from expert teaching training data, the optimal action strategy for performing the target task can be derived from the environmental state image. The target task set in the expert teaching training data is the same as the target task set in the target expert teaching data.

[0100] In step 201 of some embodiments, the target expert teaching data is the expert teaching data required during model testing and model application. It should be noted that the expert teaching training data is used to train the action policy learning model, and the target expert teaching data is used to test the action policy learning model. Both the training set and the test set are expert teaching data performing the same target task, but the expert teaching data in the training set differs from that in the test set. The method for collecting the target expert teaching data has been described in the preceding steps and will not be repeated here.

[0101] Specifically, the target expert teaching data includes environmental state images, which represent the working area of ​​the robotic arm when performing the target task and the global observation image of the target object required to perform the target task. For example, if a classification task needs to be performed, a current visual image of objects of different shapes or colors on the experimental platform is acquired, showing the characteristics of the target object and its placement in the task execution area.

[0102] In some embodiments, see Figure 3 , Figure 3 This is a schematic diagram of an environmental state image provided in an embodiment of this application. The environmental state image 300 includes an RGB height map 301 and a depth height map 302. In the RGB height map 301, smaller rectangles of different colors represent target objects that need to be grasped and placed to perform the task, and brown rectangles represent the execution area of ​​the target task. In the depth height map 302, solid yellow squares represent the target objects corresponding to those in the RGB height map 301, and yellow lines represent the execution area of ​​the target task.

[0103] Specifically, the environmental state image has a pixel resolution of 240×240, and each pixel can represent a three-dimensional space of 3.125×3.125mm. During testing and application, the action policy learning model can generate the optimal grasping and placing actions for the current environmental state based on the environmental state image and the action policy features learned during the training phase from expert teaching.

[0104] In step 202 of some embodiments, the preset grasping action generation network can generate the optimal grasping action under the current environmental state based on the environmental state image, and obtain grasping action information. The grasping action information is represented as the pixel coordinates (u,v) ∈ o of the grasping action in the environmental state image. t The end effector of the robotic arm used in this embodiment is a vacuum gripper, so the angle θ of the gripping motion does not need to be considered. pick .

[0105] Furthermore, the vacuum gripper used in this application can avoid collisions caused by the close arrangement of target objects and poor gripping angles during the gripping action. Therefore, the optimal gripping angle θ only needs to be considered after the target objects to be gripped have been determined and the optimal placement action position has been calculated by the placement action generation network. place .

[0106] After determining the pixel coordinates of the grasping action, the pixel coordinates of the grasping action can be transformed into a coordinate system according to the hand-eye calibration method of the robotic arm and camera, that is, transformed from the camera coordinate system to the robotic arm coordinate system, to obtain the three-dimensional coordinates of the robotic arm end-effector pose (x, y, y). pick ,y pick ,z pick ).

[0107] The capture action generation network is essentially a pixel-level representation of the action Q-value function network, as shown in the following equation:

[0108]

[0109] Among them, Q pick ((u,v)|o t ) indicates the environmental state image o t In the middle, the grabbing action value function Q pick The pixel coordinates (u,v) corresponding to the maximum value of T pick These are the three-dimensional pose coordinates of the robotic arm's end effector (e.g., a vacuum gripper) during the grasping action. (In the environmental state image o) t In this context, the pixel distribution of the grabbing action corresponding to the successful grabbing of the target can be multimodal, especially when there are multiple instances of the same target object in the work scenario of performing the target task, or when the shape of the target object is symmetrical.

[0110] Furthermore, the target object is grasped based on the determined grasping action pixel coordinates (u, v), and the feature information of the grasped target object is obtained based on the grasping action pixel coordinates. The feature information of the target object is the environmental state image o centered at the grasping action pixel coordinates. t The target object image φ(o) is extracted by image cropping. t [T pick In this embodiment of the application, the size of the captured target object image is 64x64, and the feature information of the captured target object will be used as an input to the placement action generation network.

[0111] In some embodiments, the training method for the grabbing action generation network includes, but is not limited to, the following steps:

[0112] Acquire expert teaching and training data, which includes images of the teaching environment and poses of the teaching grasping action.

[0113] The coordinate system is transformed to obtain the pixel coordinates of the teaching grasping action pose;

[0114] Feature extraction of the teaching environment state image is performed based on a fully convolutional layer to obtain the features of the teaching environment state image;

[0115] Based on the features of the teaching environment state image and the pixel coordinates of the teaching grasping action, the grasping action is predicted to obtain multiple predicted grasping actions;

[0116] Based on the preset grasping action value function, the grasping action value corresponding to each predicted grasping action is obtained, and the grasping action value corresponding to each pixel coordinate is obtained.

[0117] Based on the preset probability distribution function and the value of the grabbing action, the grabbing success rate corresponding to each pixel coordinate is calculated, and a probability distribution feature map related to the grabbing success rate is obtained.

[0118] Based on the pixel coordinates corresponding to the maximum success rate in the probability distribution feature map related to the success rate of grasping, the teaching grasping action information in the teaching environment state image is determined.

[0119] In this embodiment, the grasping action generation network is actually an action Q-value function network built on a fully convolutional neural network, that is, the Q-value function is estimated through a fully convolutional neural network. During the model training phase, the input to the grasping action generation network is expert teaching training data, which consists of the trajectory data of the robotic arm's end effector optimal for performing the target task. The training objective of the grasping action generation network is to learn the mapping relationship from the features of the teaching environment state image to the pixel coordinates of the taught grasping action. By continuously adjusting the internal weight parameters through predicting the Q-value of the action, it minimizes the error between the predicted grasping action pixel coordinates and the actual taught grasping action pixel coordinates.

[0120] Specifically, expert teaching training data includes images of the teaching environment and poses for the teaching grasping action. The images of the teaching environment are the visual perception images of the teaching expert when performing the target task. t It also includes RGB heightmap and depth heightmap.

[0121] The teaching grasping action pose is the pose (x, y, z) of the vacuum gripper at the end of the robotic arm when performing the grasping action. A coordinate system transformation is performed on the teaching grasping action pose to obtain the corresponding pixel coordinates (u, v) of the teaching grasping action in the height map. Further, image features are extracted from the teaching environment state image through a fully convolutional layer in the grasping action generation network to obtain the teaching environment state image features. Based on the teaching environment state image features and the pixel coordinates of the teaching grasping action, grasping actions are predicted, resulting in multiple predicted grasping actions.

[0122] Specifically, the target task to be performed includes multiple action sequences. At each moment of performing the target task, the action generation network captures the current environmental state image. t and all possible grabbing actions predicted. pick The system predicts the action value (Q-value) for each predicted grasping action and selects the grasping action with the highest Q-value for execution. After the grasping action is executed, the workspace environment state is updated, and the predicted grasping action and its corresponding Q-value are recorded and stored to accumulate experience. Furthermore, an action sequence-target state fusion algorithm can be introduced to stabilize the training process of the grasping action generation network; these steps will be described in detail later.

[0123] Furthermore, by mapping the predicted grasping action value to the features of the teaching environment state image, the grasping action value corresponding to the pixel coordinates of the teaching grasping action in the teaching environment state image is obtained, that is, the Q-value of the predicted grasping action corresponding to each teaching grasping pixel coordinate. Through the aforementioned steps, the grasping action generation network learns the action policy features for performing the target task while approximating the maximum action Q-value through a fully convolutional neural network.

[0124] The grasping action generation network provided in this application takes expert teaching training data of the robotic arm as input, transforms the prediction of the optimal grasping action pose of the robotic arm into the prediction problem of pixel-level Q-value function, improves the efficiency of grasping action strategy learning, and finally realizes the grasping of the target object by outputting the optimal grasping pixel coordinates.

[0125] In this embodiment, the preset probability distribution function is the Softmax function, which is used to perform pixel-level probability distribution operations. Based on the Softmax function and the grasping action value corresponding to each predicted grasping action, the grasping success rate corresponding to each taught grasping pixel coordinate is calculated. Based on the grasping success rate corresponding to each taught grasping pixel coordinate in the teaching environment state image, a probability distribution feature map related to the grasping success rate is obtained.

[0126] Specifically, the probability distribution feature map related to the success rate of grasping is a pixel heatmap, that is, it is obtained through the teaching environment state image. t The pixel heat value is used to represent the success rate of grabbing at the current pixel coordinate. The higher the heat value, the greater the success rate of performing the grabbing action at the current pixel coordinate.

[0127] Based on the pixel coordinates of the grasping action corresponding to the maximum grasping success rate in the probability distribution feature map, the state image o in the teaching environment is finally determined. t The teaching grasping action information in the data represents the grasping action strategy learned from expert teaching training data, specifically the pixel coordinates (u) of the teaching grasping action. pick ,v pick ).

[0128] In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the crawling action generation network provided in an embodiment of this application. It is understood that... Figure 4 This diagram illustrates the testing and application phases of the grasping action generation network, specifically the network after training. During the testing and application phases, the network needs to utilize the learned grasping action information from the training phase. Figure 4 (Not shown in the image) to perform the target task.

[0129] Specifically, the grab action generation network is an hourglass encoding and decoding architecture built from a 43-layer residual network (ResNet). The residual network contains 12 residual blocks, which can accelerate the training process of the network. It also includes a splicing module and a probability distribution function (Softmax function).

[0130] Furthermore, the residual network includes fully convolutional layers (encoders) and deconvolutional layers (decoders). The fully convolutional layers extract features from the input environment state image, while the deconvolutional layers upsample the input environment state image to obtain a pixel-level semantic segmentation image with the same size and resolution. Specifically, the fully convolutional layers include 1x1 convolutional layers with kernels of 1 pixel width and height, Rectified Linear Unit (ReLU) activation functions, 2x2 convolutional layers with kernels of 2 pixels width and height, and identity blocks.

[0131] To minimize potential resolution loss in the network, a convolution stride of 8 is set. This includes a stride of 2 in the encoder and 3 layers of bilinear upsampling in the decoder. Furthermore, each convolutional layer after the first layer features dilated convolutions, interleaved with the ReLU activation function preceding the last layer, to increase the predicted receptive field coverage per pixel.

[0132] During the training phase, the input to the grasping action generation network is the teaching environment state image and the teaching grasping action pose. In practical applications, the input to the grasping action generation network is the environment state image. The network can generate grasping actions based on the environment state image, adapting to new application scenarios of the target task, and outputting the optimal grasping action information (i.e., the grasping action pixel coordinates) for the current environment. Further, an image of the grasped target object is extracted based on the grasping action information and used as the input to the placement action generation network.

[0133] In step 303 of some embodiments, the placement action generation network is an action Q-value function network constructed based on a Siamese neural network. The Siamese neural network includes two identical network structures, each identical to the network structure of the grasping action generation network. The placement action generation network can generate placement action information based on the environmental state image and the grasping action information generated by the grasping action network, thus obtaining the first placement action information.

[0134] Specifically, one input to the placement action generation network is the image of the target object to be grasped; the other input is the same as the input to the grasping action generation network, which is the environment state image. The first action placement information includes the optimal placement action pixel coordinates (u) in the environment state image. place ,v place and placement angle θ place A coordinate system transformation is performed on the pixel coordinates of the placement action to obtain the first end effector position information of the robotic arm. This first end effector position information represents the pose (x, y) of the robotic arm's end effector under global visual observation. place ,y place ,zplace ,θ place ).

[0135] In some embodiments, the training method for the placement action generation network includes, but is not limited to, the following steps:

[0136] Acquire teaching grasping action information and teaching environment state images generated by the grasping action generation network;

[0137] Image extraction is performed centered on the pixel coordinates of the grabbing action to obtain the image of the target object;

[0138] Feature extraction is performed on the target object image to obtain the target object image features;

[0139] Acquire teaching grasping action information and teaching environment state images generated by the grasping action generation network;

[0140] Image extraction is performed based on the teaching grasping action information to obtain the target object image;

[0141] Feature extraction is performed on the target object image to obtain the target object image features;

[0142] The fusion features of the teaching environment state images are extracted to obtain the fusion features of the teaching environment state images;

[0143] The placement action is predicted by fusing the features of the target object image and the state image of the teaching environment, resulting in multiple predicted placement actions.

[0144] Based on the preset placement action value function, the placement action value corresponding to each predicted placement action is obtained, and the placement action value corresponding to each pixel coordinate is obtained.

[0145] Based on the probability distribution function and the value of the placement action, the placement success rate corresponding to each pixel coordinate is calculated, and a probability distribution feature map related to the placement success rate is obtained.

[0146] Based on the pixel coordinates corresponding to the maximum placement success rate in the probability distribution feature map related to the placement success rate, the teaching placement action information in the teaching environment state image is determined. The teaching placement action information includes the pixel coordinates of the teaching placement action and the angle of the teaching placement action.

[0147] In this embodiment, the training objective of the placement action generation network is to learn the placement action strategy features for executing the target task based on the features of the teaching environment state image and the teaching grasping action information, that is, to learn the mapping relationship from the features of the teaching environment state image to the pixel coordinates of the teaching grasping action.

[0148] Specifically, the teaching grasping action information and teaching environment state image generated by the grasping action generation network are obtained. The teaching grasping action information includes the target object image obtained by image extraction centered on the pixel coordinates of the teaching grasping action.

[0149] During the training phase of the placement action generation network, one input to the placement action generation network is the image of the target object to be grasped, and the other input is the same as that of the grasping action generation network, which is the teaching environment state image. Feature extraction is performed on the target object image to obtain target object image features. Feature extraction is also performed on the teaching environment state image to obtain teaching environment state image features. The Siamese neural network uses the teaching environment state image features as feature templates and performs feature template matching with the target object image features.

[0150] Placement actions are predicted based on the image features of the target object and the state image features of the teaching environment, resulting in multiple predicted placement actions. The placement action value corresponding to each predicted placement action is calculated according to a preset placement action value function, i.e., the Q-value corresponding to each predicted placement action is calculated.

[0151] Furthermore, similar to the output of the grasping action generation network, the placement action generation network can also output a pixel heatmap related to the placement success rate. Pixel-level probability distribution operations are performed using the Softmax function to calculate the placement success rate corresponding to each pixel coordinate, resulting in a probability distribution feature map related to the placement success rate. Based on the teaching placement pixel coordinates corresponding to the maximum placement success rate (i.e., the maximum probability) in the probability distribution feature map related to the placement success rate, the teaching placement action information in the teaching environment state image is determined. In addition, the teaching placement action information also includes the teaching placement action angle.

[0152] The following section details the method for estimating the placement action angle. In this embodiment, the placement action angle estimation method is optimized based on the Siam-BM Tracker model, a visual target tracking model based on Siamese neural networks. The Siam-BM Tracker model performs k rotations on the candidate matching target, while the matching template remains unchanged. In this embodiment, the matching template is rotated k times. Since the size of the matching template is smaller than the candidate matching target, the placement angle estimation method provided in this embodiment can improve the computation speed.

[0153] Specifically, the placement action angle estimation method provided in this application embodiment includes two processes: defining the image features of the captured target object as query features, which can be discretized into 72 angles to achieve fine-grained adjustment of the placement action angle; further, through the cross-correlation cross-convolution function in the correlation filtering module, calculating the correlation response values ​​of the 72 rotation angle features of the captured target object and the global visual features respectively, using the following calculation formula:

[0154] Q place (τ|o t ,T pick )=γφ(o t [T pick ])*ψ(o t )[τ]+b

[0155] Wherein, ψ(o t [τ] represents the global visual feature map, φ(o t [T pick ]) represents the query feature map, i.e., the features of the target object to be crawled, Q place (τ|o t ,T pick The value represents the maximum Q-value of the placement action obtained under the state of grasping action information corresponding to the rotation angle τ in the environmental state image. The scalar parameters γ and b are introduced to make the score range suitable for logistic regression; their values ​​can be defined according to actual needs.

[0156] The final output of the placement action generation network is a pixel-level action Q-value related to the placement success rate, calculated as follows:

[0157]

[0158] Among them, Q place (τ i |o t ,T pick ) indicates the environmental state image o t In the middle, the grabbing action T is executed. pick Under the condition of placing the action value function Q place The pixel coordinates (u,v) corresponding to the maximum value of τ i Let T be the rotation angle corresponding to the i-th rotation. place It is the three-dimensional pose coordinate of the end effector (such as a vacuum gripper) of the robotic arm when performing the placement action.

[0159] In some embodiments, see Figure 5 , Figure 5 This is a schematic diagram of the placement action generation network provided in an embodiment of this application. It can be understood that... Figure 5This diagram illustrates the testing and application phases of the placement action generation network, specifically the network after training. During the testing and application phases, the network needs to utilize the placement action information learned during training. Figure 5 (Not shown in the image) to perform the target task.

[0160] Specifically, the placement action generation network includes a first feature extraction subnetwork and a second feature extraction subnetwork. The first feature extraction subnetwork and the second feature extraction subnetwork constitute a Siamese neural network, which shares weight parameters. The Siamese neural network is used to extract the features of the captured target object and the features of the environmental state image to compare the similarity of the features.

[0161] Furthermore, the first and second feature extraction subnetworks have the same network structure. Taking the first feature extraction subnetwork as an example, it consists of a residual network with 43 layers and an 8-step stride, with no non-linear activation function in the last layer. Specifically, the residual network includes fully convolutional layers and deconvolutional layers. The fully convolutional layers include a 1x1 convolutional layer, a ReLU activation function, an identity block, and a 2x2 convolutional layer. In addition, the first feature extraction subnetwork also includes a probability distribution function, namely the Softmax function. The roles of each part in the network are the same as those in the action generation network, and will not be elaborated further here.

[0162] In step 203 of some embodiments, placement action information is generated based on the environmental state image and grasping action information using a preset placement action generation network to obtain first placement action information, including but not limited to the following steps:

[0163] Based on the first feature extraction sub-network, feature extraction is performed on the environmental state image to obtain the first feature information;

[0164] The second feature extraction subnetwork is used to extract features from the grasping action information to obtain the second feature information;

[0165] Based on the first correlation filter subnetwork, feature matching is performed on the first feature information and the second feature information to obtain the first feature matching result;

[0166] The first placement action information is generated based on the first feature matching result and the grasping action information.

[0167] Specifically, the placement action generation network includes a first feature extraction subnetwork and a second feature extraction subnetwork. Based on the first feature extraction subnetwork, features are extracted from the environmental state image to obtain first feature information, which represents the global visual perception state of the workspace. tThe second feature information is obtained by extracting features from the grasping action information using the convolutional layers of the second feature extraction sub-network. This second feature information is the feature information of the target object to be grasped, as determined by the grasping action generation network.

[0168] The placement action generation network also includes a first correlation filtering sub-network, which is a correlation filtering module. This module matches two features output by the Siamese neural network to determine the first placement action information under global visual observation. Specifically, the first placement action information consists of the pixel coordinates (u1, v1) and the placement action angle θ1. The detailed matching steps and angle estimation methods have been described previously and will not be repeated here.

[0169] In step 204 of some embodiments, the first placement action information and the grasping action information are input into a preset local structure learning network for position optimization to obtain the second placement action information, including but not limited to the following steps:

[0170] Image extraction is performed based on the first placement pixel coordinates to determine the local placement position image;

[0171] Based on the third feature extraction sub-network, feature extraction is performed on the image of the target object to obtain the image features of the target object.

[0172] Feature extraction is performed on the local placement location image based on the third feature extraction subnetwork to obtain the local placement location image features;

[0173] Based on the second correlation filter sub-network, feature matching is performed on the image features of the captured target object and the image features of the local placement position to obtain the second feature matching result;

[0174] Position optimization is performed based on the second feature matching result to obtain the second placement action information.

[0175] Specifically, the first placement action information includes the first placement pixel coordinates, and a local placement position image of size s×s is cropped centered on the first placement pixel coordinates. In this embodiment, the value of s is 128. Please continue to see... Figure 2 , Figure 2 The diagram shows the composition of the local structure learning network, which includes a third feature extraction subnetwork and a second correlation filtering subnetwork.

[0176] It should be noted that the placement action generation network and the local structure learning network are independent during training, but they use the same expert-taught training data for training on a single target task, and the training input of the local structure learning network depends on the output of the placement action generation network.

[0177] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram of a locally optimized action policy learning model provided in an embodiment of this application. In the global perception stage, the captured target object image is rotated and mapped to four feature spaces through a Siamese neural network to improve the speed of action policy inference; and global environmental visual features are extracted from the depth height map and RGB height map under the current environmental state, and finally the full perception prediction result (u1, v1, θ1) is output.

[0178] The local structure learning network has two inputs: one is the local placement position image; the other is the target object image to be grasped determined by the grasping action generation network. The rotation mapping space of the target object image consists of 72 rotation angles, which can reduce the deviation of the pixel coordinates of the placement action and optimize the placement action angle.

[0179] The output of the placement action generation network represents the output of the global perception stage, specifically the pixel coordinates (u1, v1) and angle θ1 of the first placement action corresponding to the global placement action position, i.e., the global perception prediction result (u1, v1, θ1). Further, a local perception optimization is performed through a local structure learning network to determine the pixel coordinates (u2, v2) and angle θ2 of the second placement action corresponding to the local placement action position, thus obtaining the local perception prediction result (u2, v2, θ2).

[0180] In this embodiment, the local structure learning network includes a third feature extraction subnetwork. Features are extracted from the image of the target object to be grasped, resulting in the image features of the target object. Features are also extracted from the local placement position image, resulting in the local placement position image features. The structure of the third feature extraction subnetwork is the same as that of the first feature extraction subnetwork, and will not be described again here.

[0181] Furthermore, feature matching is performed on the image features of the captured target object and the image features of the local placement position based on the second correlation filter sub-network to obtain the second feature matching result. Position optimization is then performed based on the second feature matching result to obtain the second placement action information. The second correlation filter sub-network works on the same principle as the first correlation filter sub-network. The output of the local structure learning network is the placement action pixel coordinates (u2, v2) and the placement action angle θ2 that maximize the Q-value function of the placement action. Further, coordinate system transformation can be performed based on the placement action pixel coordinates (u2, v2) and the placement action angle θ2 to obtain the optimized pose of the robotic arm end effector, thereby controlling the robotic arm to place the target object.

[0182] In step 205 of some embodiments, target placement action information is determined based on first placement action information obtained by the placement action generation network and second placement action information determined by the local structure learning network. The first placement action information is (u1, v1) and θ1, the second placement action information is (u2, v2) and θ2, and the target placement action information is...

[0183] The robotic arm is controlled to place the target object based on the target placement action information. The target placement action information can be transformed into a coordinate system to obtain the execution pose of the robotic arm end effector, so as to place the target object.

[0184] In some embodiments, feature extraction of the teaching environment state image to obtain the teaching environment state image features includes, but is not limited to, the following steps:

[0185] Feature extraction is performed on the current teaching environment state image to obtain the current teaching environment state image features;

[0186] Feature extraction is performed on the target teaching environment state image to obtain the target teaching environment state image features;

[0187] The features of the current teaching environment state image and the target teaching environment state image are fused to obtain the teaching environment state image fusion features.

[0188] In this application, learning grasping and placement strategies from expert teaching data of multi-sequence actions considers not only the accuracy of individual action strategies but also their impact on the overall task. Each action strategy influences the global strategy, and actions at lower sequence levels directly affect the learning effect of subsequent action strategies. Therefore, when learning action strategies for multi-sequence tasks from expert teaching task data, the optimal action strategy features for the current state can be learned, enabling successful grasping and placement, thereby effectively achieving the next optimal target state and ultimately realizing the overall task objective. Based on this, embodiments of this application provide a target state fusion guidance method.

[0189] Specifically, observing o in the current visual state t Introduce an additional target state element o on top of the existing one. g Used for visual guidance. Among them, o g From the current state s t The sampled state is obtained from a future state within the task action sequence. This application provides three sampling methods for target states, i.e., three methods for sampling target teaching environment state images. These include the current state s. t The next state s t+1 A random state s in a future state sequence i(t < i < k), and the final state s of a single task sequence. k , where k represents the total length of actions in a single task sequence.

[0190] Furthermore, the key to training the sequence-target-state fusion-guided action policy learning model lies in fusing image data. There are generally two methods for image data processing: pre-processing data-level fusion and post-processing feature-level fusion. Pre-processing data-level fusion first fuses the current state visual image and the target state visual image according to different image channels, and then extracts features.

[0191] In this embodiment, the current teaching environment state image and the target teaching environment state image are fused together and then input into the grasping action generation network for training. This data fusion method allows the grasping action generation network to learn the grasping action strategy features of the fused state, thereby improving the success rate of grasping actions.

[0192] Another image data processing method involves extracting features from the environmental state images separately and then fusing the extracted features using the Hadamard Product method. In this embodiment, the first feature extraction subnetwork of the placement action generation network extracts features from the current teaching environment state image to obtain the current teaching environment state image features. The second feature extraction subnetwork of the placement action generation network extracts features from the target teaching environment state image to obtain the target teaching environment state image features. Finally, the current teaching environment state image features and the target teaching environment state image features are fused to obtain the fused teaching environment state image features.

[0193] The feature fusion method can extract the features of the target object in the grasping environment state image by combining the grasping action information output by the grasping action generation network. Then, it is input into the correlation filtering module to output the optimal placement action information, thereby improving the success rate of placement action.

[0194] In some embodiments, see Figure 7 , Figure 7 This is a schematic diagram of a model framework for target state fusion guidance provided in an embodiment of this application. Specifically, for the grasping action generation network, the current teaching environment state image is generated in the preceding stage. t Image of the target teaching environment t+1 Data fusion is performed, and the fused results are input into the crawling action generation network. It can be understood that the target state element o... g Including target teaching environment state images. t+1 .

[0195] For the placement action generation network, the current teaching environment state image ot Feature extraction and analysis of the target teaching environment state image. t+1 Feature extraction is performed, the two extracted features are fused, and the feature fusion result and the crawling action information obtained by the crawling action generation network are input into the first correlation filtering submodule for similarity comparison. Figure 7 The current teaching environment state image in o t Image of the target teaching environment t+1 Both include RGB heightmaps and depth heightmaps (not shown in the figure).

[0196] In some embodiments, see Figure 8 , Figure 8 This is a schematic diagram of the target state fusion guided crawling action generation network provided in an embodiment of this application. The data fusion formula is as follows:

[0197] Q pick (o t ,o g )=Φ pick (o t )☉Φ goal (o g )

[0198] Among them, Q pick (o t ,o g (Image o representing the current teaching environment state) t Image of the target teaching environment t+1 The Q-value of the grasping action under fusion, where ☉ indicates feature fusion, specifically the Hadamard product, Φ pick (o t ) represents the features of the target object captured in the state image of the teaching environment, Φ goal (o g () indicates the state image of the target teaching environment. t+1 The system captures the features of the target object and ultimately outputs the predicted action pixels related to the capture success rate. The optimal capture action position pixel coordinates are generated using the following formula:

[0199]

[0200] in, Indicates the environmental state image o t and target environment state image o g Under fusion conditions, the pixel coordinates corresponding to the maximum value of the grasping action value function, T pick It is the three-dimensional pose coordinates of the end effector (such as a vacuum gripper) of the robotic arm when performing a grasping action.

[0201] In some embodiments, see Figure 9 , Figure 9 This is a schematic diagram of the target state fusion guided placement action generation network provided in an embodiment of this application. As mentioned in the preceding steps, the placement action generation network is built based on a Siamese network. It optimizes the placement action strategy generated by the global visual state through a local structure learning network, including the optimization of the placement action pixel coordinates and placement action angle.

[0202] Therefore, the learning of the placement action policy can be divided into two stages: the first stage learns the policy based on the global visual state, and the second stage refines the action policy from the first stage using local visual states. A target state o is introduced into the placement action generation network. g At this time, only the first stage of target fusion is performed, because the first stage is the global perception stage, and the goal is to select the best strategy to reach the next target state based on the guidance of the target state. The second stage is just an optimization of the strategy in the first stage.

[0203] Specifically, the learning process of the target state fusion-guided placement action generation network consists of three steps: the current teaching environment state image can be extracted through the first feature extraction subnetwork and the second feature extraction subnetwork of the Siamese neural network, respectively. t Image of the target teaching environment t+1 The state features are identified and fused. The feature-level fusion formula is as follows:

[0204] φ fuse (o t )=ψ place (o t )☉ψ goal (o t )

[0205] Where ☉ represents feature fusion, specifically Hadamard product, ψ place (o t () indicates the state image o in the teaching environment. t The characteristics of the target object placed in the middle, ψ goal (o g ) represents the features of the target teaching environment state image.

[0206] Furthermore, the local structure learning network optimizes the placement action, and the value function formula for the placement action is as follows:

[0207] Q place (Δτ|o t ,T place )=γφ crop (o t )[T place ]*φ fuse (o t)[Δτ]+b

[0208] Among them, the pixel coordinates of the grabbing action (u) pick ,v pick In the feature fusion state, the corresponding feature φ of the target object to be grasped crop (o t )[T pick Then, through the first correlation filter sub-network, the features φ of the target object are captured. crop (o t )[T pick ] Fusion feature φ with global state fuse (o t Perform cross-correlation cross-convolution matching to obtain the pixel coordinates corresponding to the global perception placement action under global state feature fusion.

[0209] Image cropping is performed using this coordinate as the center to obtain the environmental state image features φ under global state fusion features. crop (o t )[T place ], Δτ represents the optimized placement angle value obtained through local optimization, Q place (Δτ|o t ,T place ) represents the placement action information T obtained from global perception prediction. place and teaching environment status images o t The Q-value of the placement action corresponding to the estimated placement action angle deviation under the given conditions. Scalar parameters γ and b are introduced to make the score range suitable for logistic regression; their values ​​can be defined according to actual needs.

[0210] Furthermore, the final local perception prediction result (x', y', z') output by the local structure learning network represents the end-effector pose of the robotic arm after the placement action optimization. The optimized second placement action information is obtained through the following formula:

[0211]

[0212] Where, T on the left side of the equation place This represents the optimized placement action information, which is obtained by predicting the placement action information T in global perception using the right side of the equation. place and teaching environment status images o t Under the given conditions, the optimal placement action angle corresponding to the maximum Q-value among the predicted multiple placement action optimization angle values ​​is obtained. Based on the optimized placement action angle, the placement action angles in the global perception prediction information are adjusted.

[0213] In some embodiments, the method for obtaining expert teaching training data further includes:

[0214] The grasping action generation network and the placement action generation network are trained based on expert teaching training data to obtain the target training loss;

[0215] The sampling probability of each action sequence in the execution of the target task is calculated based on the target training loss and the reward value corresponding to each action sequence.

[0216] Action sequences are sampled from the expert teaching training data based on sampling probability, and the grasping action generation network and the placement action generation network are trained based on the sampled action sequences.

[0217] Specifically, in deep learning model training, it is generally assumed that different labeled samples contribute almost equally to model training. However, in the object placement method based on a robotic arm provided in this application, the expert teaching task data contains multiple action sequences, and the difficulty coefficient of the action strategy increases with the number of task sequences, especially for assembly and stacking tasks. When the number of action sequences increases, the predictability space of placement actions decreases, and the previous action strategy affects subsequent actions, which leads to difficulties in learning placement strategies. Combining the idea of ​​active learning, different sample data in the dataset have different values. Based on this, the action strategy model can be trained with data of higher values ​​or with fewer sample data, thereby obtaining better performance.

[0218] Therefore, in order to learn more valuable strategic knowledge from small-sample expert teaching task data, this application optimizes the sample sampling method in the action policy model training process and designs a sample sampling method based on teaching action rewards. Expert teaching training data includes multiple action sequences required to perform the target task, and each action sequence corresponds to a reward value taken during the execution of the target task. That is, during the collection of action trajectories in the expert teaching data, the larger the reward value of an action, the greater the probability that the sequence of actions will be sampled for training and updating the action policy model. Failures during expert guidance do not contribute to the updating of the action policy model and are therefore not sampled. The sampling probability calculation formula for expert teaching data provided in this application is as follows:

[0219]

[0220] Where L represents the target training loss of the entire action policy model obtained during the training of the grasping action generation network and the placement action generation network for a single target task, with an initial value of 1; r iLet represent the reward value when the action sequence is of level i in a single teaching task, and N represent the total number of teaching action sequences in a single target task; p(i) represents the probability that the action in the i-th sequence is sampled. As can be seen from the formula, the loss gradually decreases and gradually approaches the expected value as the number of training iterations increases. The sampling method was changed to uniform sampling.

[0221] The embodiments of this application effectively solve the problem that uniform sampling can only obtain limited action policy knowledge, which is more valuable for optimizing action policy learning models and can efficiently train models with small sample expert teaching data.

[0222] Further, please see Figure 10 , Figure 10 This is a schematic diagram of the expert teaching data augmentation module provided in this application embodiment. The expert teaching data augmentation module includes a reward optimization sampling module and a random rigid transformation module. The reward optimization sampling module is used to sample the initial expert teaching training data according to the aforementioned reward optimization sampling method and sampling formula, and to perform random rigid transformation on the obtained valuable expert teaching data to obtain more valuable teaching data, and finally obtain expert teaching training data for training and updating the action policy model.

[0223] Please see Figure 11 This application also provides an object placement device based on a robotic arm, which can implement the above-described object placement method based on a robotic arm. The device includes:

[0224] The acquisition unit 1110 is used to acquire target expert teaching data for performing the target task. The target expert teaching data includes an environmental state image, which includes the target object required to perform the target task. The first generation unit 1120 is used to generate grasping action information from the environmental state image based on a preset grasping action generation network to obtain grasping action information of the target object. The second generation unit 1130 is used to generate placement action information from the environmental state image and the grasping action information based on a preset placement action generation network to obtain first placement action information. The placement action generation network is a network constructed based on a Siamese network. The optimization unit 1140 is used to input the first placement action information and the grasping action information into a preset local structure learning network for position optimization to obtain second placement action information. The determination unit 1150 is used to determine the target placement action information based on the first placement action information and the second placement action information, so as to control the robotic arm to place the target object according to the target placement action information.

[0225] The specific implementation of the robotic arm-based object placement device is basically the same as the specific implementation of the robotic arm-based object placement method described above, and will not be repeated here.

[0226] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described object placement method based on a robotic arm. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0227] Please see Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0228] The processor 1201 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0229] The memory 1202 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 to execute the object placement method based on the robotic arm in the embodiments of this application.

[0230] The input / output interface 1203 is used to implement information input and output;

[0231] The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0232] Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204);

[0233] The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0234] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described object placement method based on a robotic arm.

[0235] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0236] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0237] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0238] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0239] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0240] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0241] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0242] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0243] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0244] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0245] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0246] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for object placement based on a robotic arm, characterized in that, The method includes: Acquire target expert teaching data for performing the target task, the target expert teaching data including an environmental state image, the environmental state image including the target object required to perform the target task; Based on a preset grasping action generation network, grasping action information is generated from the environmental state image to obtain the grasping action information of the target object. The placement action information is generated based on the environmental state image and the grasping action information by a preset placement action generation network to obtain the first placement action information. The placement action generation network is a network built based on Siamese network. The first placement action information and the grasping action information are input into a preset local structure learning network for position optimization to obtain the second placement action information; Target placement action information is determined based on the first placement action information and the second placement action information, so as to control the robotic arm to place the target object according to the target placement action information. Wherein, the first placement action information includes first placement pixel coordinates, the grasping action information includes grasping target object image, the local structure learning network includes a third feature extraction subnetwork and a second correlation filtering subnetwork, and the step of inputting the first placement action information and the grasping action information into a preset local structure learning network for position optimization to obtain second placement action information includes: Image extraction is performed based on the first placement pixel coordinates to determine the local placement position image; Based on the third feature extraction sub-network, feature extraction is performed on the image of the target object to be captured, and the image features of the target object are obtained. Based on the local placement position image of the third feature extraction subnetwork, feature extraction is performed to obtain the local placement position image features; Based on the second correlation filter subnetwork, feature matching is performed on the image features of the captured target object and the image features of the local placement position to obtain the second feature matching result; Position optimization is performed based on the second feature matching result to obtain the second placement action information; The grasping action generation network includes fully convolutional layers, and the training method for the grasping action generation network includes: Acquire expert teaching training data, which includes images of the teaching environment and poses of the teaching grasping action; The teaching grasping action pose is transformed by coordinate system to obtain the pixel coordinates of the teaching grasping action; Based on the fully convolutional layer, feature extraction is performed on the teaching environment state image to obtain the teaching environment state image features; Based on the features of the teaching environment state image and the pixel coordinates of the teaching grasping action, grasping action prediction is performed to obtain multiple predicted grasping actions; Based on the preset grasping action value function, the grasping action value corresponding to each predicted grasping action is obtained, and the grasping action value corresponding to each pixel coordinate is obtained. Based on the preset probability distribution function and the value of the grasping action, the grasping success rate corresponding to each pixel coordinate is calculated, and a probability distribution feature map related to the grasping success rate is obtained. Based on the pixel coordinates corresponding to the maximum value of the grasping success rate in the probability distribution feature map related to the grasping success rate, the teaching grasping action information in the teaching environment state image is determined.

2. The method according to claim 1, characterized in that, The placement action generation network includes a first feature extraction subnetwork, a second feature extraction subnetwork, and a first correlation filtering subnetwork. The placement action generation network generates placement action information based on the environmental state image and the grasping action information to obtain first placement action information, including: Based on the first feature extraction sub-network, feature extraction is performed on the environmental state image to obtain first feature information; Based on the second feature extraction sub-network, feature extraction is performed on the grasping action information to obtain the second feature information; Based on the first correlation filter sub-network, feature matching is performed on the first feature information and the second feature information to obtain the first feature matching result; Based on the first feature matching result and the grasping action information, placement action information is generated to obtain the first placement action information.

3. The method according to claim 1, characterized in that, The training method for the placement action generation network includes: Acquire the teaching grasping action information and the teaching environment state image generated by the grasping action generation network; Based on the taught grasping action information, image extraction is performed to obtain the target object image; Feature extraction is performed on the target object image to obtain the target object image features; The teaching environment state image is subjected to fusion feature extraction to obtain the teaching environment state image fusion feature; Based on the fusion features of the target object image and the teaching environment state image, a placement action is predicted to obtain multiple predicted placement actions. Based on the preset placement action value function, the placement action value corresponding to each predicted placement action is obtained, and the placement action value corresponding to each pixel coordinate is obtained. Based on the probability distribution function and the placement action value, the placement success rate corresponding to each pixel coordinate is calculated to obtain a probability distribution feature map related to the placement success rate. Based on the pixel coordinates corresponding to the maximum placement success rate in the probability distribution feature map related to the placement success rate, the teaching placement action information in the teaching environment state image is determined. The teaching placement action information includes the pixel coordinates of the teaching placement action and the angle of the teaching placement action.

4. The method according to claim 3, characterized in that, The teaching environment state image includes a current teaching environment state image and a target teaching environment state image. The target teaching environment state image represents the environment state image after each action is performed. The step of extracting fusion features from the teaching environment state images to obtain fusion features of the teaching environment state images includes: Feature extraction is performed on the current teaching environment state image to obtain the current teaching environment state image features; Feature extraction is performed on the target teaching environment state image to obtain the target teaching environment state image features; The current teaching environment state image features and the target teaching environment state image features are fused to obtain the teaching environment state image fusion features.

5. The method according to claim 1, characterized in that, The expert teaching training data includes multiple action sequences required to perform the target task, and each action sequence corresponds to a reward value taken during the execution of the target task. The method for obtaining the expert teaching training data includes: The grasping action generation network and the placement action generation network are trained based on the expert teaching training data to obtain the target training loss; The sampling probability of each action sequence in the execution of the target task is calculated based on the target training loss and the reward value corresponding to each action sequence. Based on the sampling probability, action sequences are sampled from the expert teaching training data, and the grasping action generation network and the placement action generation network are trained according to the sampled action sequences.

6. A method and apparatus for placing objects with a robotic arm, characterized in that, The device includes: An acquisition unit is configured to acquire target expert teaching data for performing a target task, the target expert teaching data including an environmental state image, the environmental state image including target objects required to perform the target task; The first generation unit is used to generate grasping action information from the environmental state image based on a preset grasping action generation network to obtain the grasping action information of the target object. The second generation unit is used to generate placement action information based on the environmental state image and the grasping action information according to a preset placement action generation network, so as to obtain the first placement action information. The placement action generation network is a network built based on Siamese network. An optimization unit is used to input the first placement action information and the grasping action information into a preset local structure learning network for position optimization to obtain the second placement action information. The determining unit is configured to determine target placement action information based on the first placement action information and the second placement action information, so as to control the robotic arm to place the target object according to the target placement action information; Wherein, the first placement action information includes first placement pixel coordinates, the grasping action information includes grasping target object image, the local structure learning network includes a third feature extraction subnetwork and a second correlation filtering subnetwork, and the step of inputting the first placement action information and the grasping action information into a preset local structure learning network for position optimization to obtain second placement action information includes: Image extraction is performed based on the first placement pixel coordinates to determine the local placement position image; Based on the third feature extraction sub-network, feature extraction is performed on the image of the target object to be captured, and the image features of the target object are obtained. Based on the local placement position image of the third feature extraction subnetwork, feature extraction is performed to obtain the local placement position image features; Based on the second correlation filter subnetwork, feature matching is performed on the image features of the captured target object and the image features of the local placement position to obtain the second feature matching result; Position optimization is performed based on the second feature matching result to obtain the second placement action information; The grasping action generation network includes fully convolutional layers, and the training method for the grasping action generation network includes: Acquire expert teaching training data, which includes images of the teaching environment and poses of the teaching grasping action; The teaching grasping action pose is transformed by coordinate system to obtain the pixel coordinates of the teaching grasping action; Based on the fully convolutional layer, feature extraction is performed on the teaching environment state image to obtain the teaching environment state image features; Based on the features of the teaching environment state image and the pixel coordinates of the teaching grasping action, grasping action prediction is performed to obtain multiple predicted grasping actions; Based on the preset grasping action value function, the grasping action value corresponding to each predicted grasping action is obtained, and the grasping action value corresponding to each pixel coordinate is obtained. Based on the preset probability distribution function and the value of the grasping action, the grasping success rate corresponding to each pixel coordinate is calculated, and a probability distribution feature map related to the grasping success rate is obtained. Based on the pixel coordinates corresponding to the maximum value of the grasping success rate in the probability distribution feature map related to the grasping success rate, the teaching grasping action information in the teaching environment state image is determined.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the object placement method based on a robotic arm as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the object placement method based on the robotic arm as described in any one of claims 1 to 5.

Images