Object grasping method, device, electronic device and storage medium

Abstract

The present application relates to a method, device, electronic device, storage medium, and computer program product for grasping an object. The method comprises: acquiring a first scene image of a first frame containing an object, and a second scene image of a subsequent second frame containing the object; acquiring a first candidate grasping posture set of the first frame and a second candidate grasping posture set of the second frame based on the first scene image and the second scene image, and acquiring the degree of association between the first candidate grasping posture set and the second candidate grasping posture set; determining a first grasping posture corresponding to the first frame from the first candidate grasping posture set, and determining a second grasping posture corresponding to the second frame from the second candidate grasping posture set based on the first grasping posture and the degree of association; and controlling an actuator to grasp a moving object based on the second grasping posture. The use of this method can improve the stability and accuracy of grasping moving objects.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a method, device, electronic device, storage medium, and computer program product for grasping an object. Background Art

[0002] Object grasping, a key issue in robotics, plays a crucial role in agriculture, industry, and the service industry. The mainstream approach to object grasping involves using sensors to capture a 2D or 3D image of the scene, using grasping posture detection technology to predict the position and orientation of the robotic arm's end-grip when grasping the object, and then executing the grasping action after motion planning.

[0003] In traditional technology, the grasping detection of moving objects generally converts the grasping of dynamic objects into the problem of object posture tracking. By continuously tracking the same object and predicting its posture in space, the moving object is grasped according to a manually pre-set grasping posture template. However, since the grasping posture of the object is manually set and the distribution is relatively sparse, it cannot change flexibly with the movement of the position, resulting in unsatisfactory grasping effect. Summary of the Invention

[0004] Based on this, it is necessary to provide an object grasping method, device, electronic device, computer-readable storage medium and computer program product that can improve the accuracy of object grasping in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for grasping an object, the method comprising:

[0006] Acquire a first scene image of a first frame including the object, and a second scene image of a subsequent second frame including the object;

[0007] Obtaining, based on the first scene image and the second scene image, a first candidate grasping posture set of the first frame and a second candidate grasping posture set of the second frame, and obtaining a degree of correlation between the first candidate grasping posture set and the second candidate grasping posture set;

[0008] Determine a first grasping posture corresponding to the first frame from the first candidate grasping posture set, and determine a second grasping posture corresponding to the second frame from the second candidate grasping posture set based on the first grasping posture and the degree of association;

[0009] The control execution mechanism grasps the moving object according to the second grasping posture.

[0010] In one embodiment, obtaining a first candidate grasping posture set for the first frame and a second candidate grasping posture set for the second frame based on the first scene image and the second scene image, and obtaining a correlation degree between the first candidate grasping posture set and the second candidate grasping posture set, includes:

[0011] Processing the first scene image and the second scene image through a grasp generation model to obtain a first candidate grasping posture set for the first frame and a second candidate grasping posture set for the second frame;

[0012] The first candidate grasping posture set and the second candidate grasping posture set are processed by a grasping association model to determine the degree of association between the first candidate grasping posture set and the second candidate grasping posture set.

[0013] In one embodiment, the training method of the crawling association model includes:

[0014] Obtain a first sample grasping posture set and a second sample posture grasping posture set of a sample data set;

[0015] Determining a grasping distance between any two sample grasping postures in the first sample grasping posture set and the second sample grasping posture set;

[0016] Comparing each of the grasping distances with a preset distance threshold to determine a sample target grasping posture set, where the sample target grasping posture set is a set consisting of sample grasping postures of the same type in the second sample posture grasping set and the first sample grasping posture set;

[0017] Determining a loss function value based on the first sample grasping posture set, the second sample grasping posture set, and the sample target grasping posture set;

[0018] According to the loss function value, the initial weight parameters are iterated to train and obtain the grasp association model.

[0019] In one embodiment, the first sample grasping posture set includes a first sample grasping posture prediction value, a first sample geometric feature vector, and a first sample candidate grasping feature vector associated with each first sample grasping posture; the second sample grasping posture set includes a second sample grasping posture prediction value, a second sample geometric feature vector, and a second sample candidate grasping feature vector associated with each second sample grasping posture;

[0020] The determining of the grasping distance between any two sample grasping postures in the first sample grasping posture set and the second sample grasping posture set includes:

[0021] Determining a first sample rotation amount and a first sample translation amount of the first sample grasping posture based on the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample candidate grasping feature vector;

[0022] determining a second sample rotation amount and a second sample translation amount of the second sample grasping posture according to the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector;

[0023] determining a rotation distance between the first sample grasping posture and the second sample grasping posture based on the first sample rotation amount and the second sample rotation amount;

[0024] determining a translation distance between the first sample grasping posture and the second sample grasping posture according to the first sample translation amount and the second sample translation amount;

[0025] A grasping distance between the first sample grasping posture and the second sample grasping posture is determined according to the rotation distance and the translation distance.

[0026] In one embodiment, the step of processing the first scene image and the second scene image using a grasping generation model to obtain a first candidate grasping pose set for the first frame and a second candidate grasping pose set for the second frame includes:

[0027] Acquire a first scene point cloud of the first scene image;

[0028] Performing point cloud scene analysis on the first scene point cloud to obtain a plurality of first scene seed points;

[0029] Acquire a second scene point cloud of the second scene image;

[0030] Performing point cloud scene analysis on the second scene point cloud to obtain a plurality of second scene seed points;

[0031] Performing feature extraction processing on each first scene seed point to obtain a corresponding first grasping result, where the first grasping result includes a first grasping posture prediction value, a first geometric feature vector, and a first candidate grasping feature vector;

[0032] Feature extraction processing is performed on each second scene seed point to obtain a corresponding second grasping result, where the first grasping result includes a second grasping posture prediction value, a second geometric feature vector, and a second candidate grasping feature vector.

[0033] In one embodiment, the first candidate grasping posture set includes a first grasping posture prediction value, a first geometric feature vector, and a first candidate grasping feature vector associated with each first grasping posture; the second candidate grasping posture set includes a second grasping posture prediction value, a second geometric feature vector, and a second candidate grasping feature vector associated with each grasping posture;

[0034] The processing of the first candidate grasping posture set and the second candidate grasping posture set by using a grasping association model to determine the degree of association between the first candidate grasping posture set and the second candidate grasping posture set includes:

[0035] Performing color feature processing on the first grasping posture to obtain a first color feature vector corresponding to the first grasping posture;

[0036] performing color feature processing on the second grasping posture to obtain a second color feature vector corresponding to the second grasping posture;

[0037] Performing vector processing on the first grasping posture prediction value, the first geometric feature vector, and the first color feature vector, and splicing the resultant vectors onto the first candidate grasping feature vector to obtain a first initial spliced feature vector;

[0038] performing vector processing on the second grasping posture prediction value, the second geometric feature vector, and the second color feature vector, and splicing the resultant vectors onto the second candidate grasping feature vector to obtain a second initial spliced feature vector;

[0039] Processing the first initial splicing feature vector and the second initial splicing feature vector respectively to obtain a corresponding first target splicing feature vector and a second target splicing feature vector;

[0040] Based on the first target splicing feature vector and the second target splicing feature vector, a correlation degree between the first candidate grasping posture set and the second candidate grasping posture set is obtained.

[0041] In one embodiment, for any of the first grasping postures and the second grasping postures, the corresponding color feature vector is obtained by the following steps:

[0042] Determine the color information of the seed point of the grasping posture;

[0043] Sampling the color information of the seed point to obtain an initial color vector;

[0044] The initial color vectors are sequentially subjected to feature processing to obtain corresponding color feature vectors.

[0045] In one embodiment, the correlation degree is a correlation matrix; and obtaining the correlation degree between the first candidate grasping posture set and the second candidate grasping posture set based on the first target splicing feature vector and the second target splicing feature vector includes:

[0046] determining a cosine similarity between the first target splicing feature vector and the second target splicing feature vector;

[0047] According to the cosine similarity, a correlation matrix is obtained.

[0048] In one embodiment, the degree of association is an association matrix;

[0049] The determining, based on the first grasping posture and the association degree, a second grasping posture corresponding to the second frame from the second candidate grasping posture set includes:

[0050] Based on the number of rows of the association matrix where the first grasping posture is located, the grasping posture with the highest grasping association degree in the second candidate grasping posture set is determined as the second grasping posture of the second frame.

[0051] In a second aspect, the present application further provides a device for grasping an object. The device comprises:

[0052] An image acquisition module, configured to acquire a first scene image of a first frame including the object, and a second scene image of a subsequent second frame including the object;

[0053] a prediction module, configured to obtain, based on the first scene image and the second scene image, a first set of candidate grasping postures for the first frame and a second set of candidate grasping postures for the second frame, and obtain a degree of correlation between the first set of candidate grasping postures and the second set of candidate grasping postures;

[0054] a posture determination module, configured to determine a first grasping posture corresponding to the first frame from the first candidate grasping posture set, and determine a second grasping posture corresponding to the second frame from the second candidate grasping posture set based on the first grasping posture and the degree of association;

[0055] The grasping module is used to control the actuator to grasp the moving object according to the second grasping posture.

[0056] In a third aspect, the present application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the object grasping method described above when executing the computer program.

[0057] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the object grasping method described above.

[0058] In a fifth aspect, the present application further provides a computer program product, which includes a computer program that implements the steps of the object grasping method when executed by a processor.

[0059] The above-mentioned object grasping method, device, electronic device, storage medium and computer program product obtain a first scene image of a first frame containing an object, and a second scene image of a subsequent second frame containing the object; based on the first scene image and the second scene image, obtain a first candidate grasping posture set of the first frame and a second candidate grasping posture set of the second frame, and obtain the degree of association between the first candidate grasping posture set and the second candidate grasping posture set; determine a first grasping posture corresponding to the first frame from the first candidate grasping posture set, and based on the first grasping posture and the degree of association, determine a second grasping posture corresponding to the second frame from the second candidate grasping posture set; and control the actuator to grasp the moving object according to the second grasping posture. Among them, based on the acquired first scene image and the second scene image, the first candidate grasping posture set of the first frame and the second candidate grasping posture set of the second frame are obtained, and the degree of correlation between the first candidate grasping posture set and the second candidate grasping posture set is obtained, so as to determine the similarity between different candidate grasping postures in two adjacent frames. Therefore, the second grasping posture of the second frame can be inferred based on the first grasping posture and the degree of correlation of the first frame, thereby avoiding the grasping posture jump problem in the current grasping detection algorithm and improving the stability and accuracy of grasping. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 FIG. 1 is a diagram illustrating an application environment of a method for grasping an object in one embodiment;

[0061] Figure 2 is a flowchart of a method for grasping an object in one embodiment;

[0062] Figure 3 1 is a flow chart of a method for grasping an object in one embodiment;

[0063] Figure 4 is a flow chart of a method for grasping an object in another embodiment;

[0064] Figure 5 is a flow chart of a method for grasping an object in another embodiment;

[0065] Figure 6 is a flowchart of a method for grasping an object in another embodiment;

[0066] Figure 7 is a flow chart of a method for grasping an object in another embodiment;

[0067] Figure 8 is a structural block diagram of a device for grasping an object in one embodiment;

[0068] Figure 9 FIG. 1 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0070] The object grasping method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, Figure 1 The application environment shown includes an electronic device 102, an image acquisition device 104, and an actuator 106. The image acquisition device 104 and the actuator 106 are respectively connected to the electronic device 102 for communication. The image acquisition device 104 can be an RGB-D camera (depth camera), and the actuator 106 can be a robotic arm. When there is a need to grasp an object, the image acquisition device 104 can obtain a first scene image and a second scene image of the current scene (the scene image can include a color map and a depth map), and output the obtained first scene image and second scene image to the electronic device 102. The electronic device 102 processes the first scene image and the second scene image to determine a second grasping posture. Finally, the electronic device 102 controls the actuator 106 to grasp the moving object according to the second grasping posture.

[0071] In one embodiment, electronic device 102 obtains a first scene image of a first frame containing an object, and a second scene image of a subsequent second frame containing the object; based on the first scene image and the second scene image, obtains a first set of candidate grasping postures for the first frame and a second set of candidate grasping postures for the second frame, and obtains a degree of correlation between the first set of candidate grasping postures and the second set of candidate grasping postures; determines a first grasping posture corresponding to the first frame from the first set of candidate grasping postures, and based on the first grasping posture and the degree of correlation, determines a second grasping posture corresponding to the second frame from the second set of candidate grasping postures; and controls an actuator to grasp the moving object based on the second grasping posture. Electronic device 102 may be a terminal device, which may be, but is not limited to, various control chips, personal computers, laptops, smartphones, tablet computers, and portable wearable devices. In other embodiments, electronic device 102 may also be a server, which may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0072] In one embodiment, Figure 2 As shown in FIG, it is a general flow chart of the object grasping method provided in an embodiment of the present application:

[0073] Among them, the overall flow chart of this embodiment may include the following modules: A: data acquisition and processing module; B: grasping generation module; C: grasping association module; the grasping association module may include C1 color feature aggregation submodule and C2 association matrix generation submodule (not shown in the figure); D: network training module, the network training module may include (D1: grasping distance definition; D2: supervised contrast learning; D3: training details (not shown in the figure)).

[0074] Among them, the specific processing process of each module roughly includes the following steps: the data acquisition and processing module A can obtain a data set, which can store scene images taken by the camera in a variety of scenes, and perform data processing on two adjacent frames of scene images in the same scene in the data set to obtain the target point cloud input to the grasping generation module B, which is processed by the grasping generation module B to obtain the grasping posture prediction value, geometric feature vector and candidate grasping feature vector. The grasping generation module B can input the above three parameters into the grasping association module C, and obtain the grasping association value after processing by the grasping association module C. The three parameters generated by the grasping generation module B and the grasping association value between the grasping postures obtained in the grasping association module C are input together into the network training module D. During training, the loss value is obtained by calculation, and the loss value and the weight parameters generated by random initialization are input into the optimizer to obtain new weight parameters. The network training module D sends the above new weight parameters to the grasping association module C to update the weight parameters of the multi-layer perceptron in the grasping association module C.

[0075] The above process is then repeated based on the data in the dataset. Each time, a weight parameter is obtained after network training, and the weights of the multilayer perceptron in the grasp association module C are updated. Training is stopped after a certain number of iterations, and the number of iterations can be set. The weights in the trained object pose prediction model are no longer updated, meaning that the weight parameters in the subsequent grasp association module C do not need to be changed.

[0076] In one embodiment, Figure 3 As shown, a method for grasping an object is provided, and the method is applied to Figure 1 The electronic device in FIG is taken as an example to illustrate, including the following steps:

[0077] Step S302 : acquiring a first scene image of a first frame including an object and a second scene image of a subsequent second frame including the object.

[0078] The first scene image of the first frame may be an image captured by the camera at the current moment when there is a need to grasp an object, and the second scene image of the second frame may be an image captured by the camera at the next moment after the current moment, wherein the second frame may be the next frame after the first frame. The first scene image and the second scene image may include a moving object to be grasped.

[0079] Step S304 : acquiring a first candidate grasping posture set of the first frame and a second candidate grasping posture set of the second frame according to the first scene image and the second scene image, and acquiring a correlation degree between the first candidate grasping posture set and the second candidate grasping posture set.

[0080] Among them, the first candidate grasping posture set and the second candidate grasping posture set are composed of grasping postures, and the grasping posture can be the position and orientation of the actuator in three-dimensional space when grasping a moving object. For example, if the actuator structure is a robotic arm, the grasping posture can be the position and orientation of the robotic arm's gripper in three-dimensional space, where the position and orientation can be the translation amount and rotation amount.

[0081] Among them, the first candidate grasping posture set is composed of multiple first grasping postures, and the first grasping posture can be determined after the electronic device processes the first scene image of the first frame; the second candidate grasping posture set is composed of multiple second grasping postures, and the second grasping posture can be determined after the electronic device processes the second scene image; the degree of association between the first candidate grasping posture set and the second candidate grasping posture set can be the degree of similarity between each first grasping posture and the second grasping posture; the degree of similarity between the first grasping posture and the second grasping posture can be presented in the form of a correlation matrix.

[0082] In one embodiment, based on the first scene image and the second scene image, a first candidate grasping posture set of the first frame and a second candidate grasping posture set of the second frame are obtained, and the degree of association between the first candidate grasping posture set and the second candidate grasping posture set is obtained, including: processing the first scene image and the second scene image through a grasping generation model to obtain the first candidate grasping posture set of the first frame and the second candidate grasping posture set of the second frame; processing the first candidate grasping posture set and the second candidate grasping posture set through a grasping association model to determine the degree of association between the first candidate grasping posture set and the second candidate grasping posture set.

[0083] Among them, after the electronic device obtains the first scene image and the second scene image, it can obtain a grasping generation model and a grasping association model. The grasping generation model processes the first scene image and the second scene image to obtain a first candidate grasping posture set and a second candidate grasping posture set. Then, the grasping association model further processes the first candidate grasping posture set and the second candidate grasping posture set to obtain the degree of association between the first candidate grasping posture set and the second candidate grasping posture set.

[0084] Step S306 : determining a first grasping posture corresponding to the first frame from the first candidate grasping posture set, and determining a second grasping posture corresponding to the second frame from the second candidate grasping posture set based on the first grasping posture and the correlation degree.

[0085] The second grasping posture is the grasping posture finally selected by the electronic device to grasp the moving object. When determining the second grasping posture corresponding to the second frame, the electronic device may determine it based on the degree of association and the first grasping posture of the first frame.

[0086] Step S308: Control the actuator to grasp the moving object according to the second grasping posture.

[0087] After determining the second grasping posture, the electronic device can control the actuator to grasp the moving object according to the second grasping posture.

[0088] In the above-mentioned object grasping method, a first scene image of a first frame containing the object and a second scene image of a subsequent second frame containing the object are obtained; a first set of candidate grasping postures for the first frame and a second set of candidate grasping postures for the second frame are obtained based on the first scene image and the second scene image, and a degree of correlation between the first set of candidate grasping postures and the second set of candidate grasping postures is obtained; a first grasping posture corresponding to the first frame is determined from the first set of candidate grasping postures, and a second grasping posture corresponding to the second frame is determined from the second set of candidate grasping postures based on the first grasping posture and the degree of correlation; and an actuator is controlled to grasp the moving object based on the second grasping posture. The method comprises the following steps: according to the first scene image and the second scene image, the first set of candidate grasping postures for the first frame and the second set of candidate grasping postures for the second frame are obtained, and the degree of correlation between the first set of candidate grasping postures and the second set of candidate grasping postures is obtained, thereby determining the degree of similarity between different candidate grasping postures in two adjacent frames. Thus, the second grasping posture for the second frame can be inferred based on the first grasping posture and the degree of correlation of the first frame, thereby avoiding the grasping posture jump problem in current grasping detection algorithms and improving the stability and accuracy of grasping.

[0089] In one embodiment, when the electronic device determines the grasping posture for grasping a moving object, when the first frame is the initial frame, it can also directly process only the first scene image of the first frame. Specifically, the electronic device can determine the corresponding three-dimensional coordinates (x, y, z) of any pixel point (u, v) in the first scene image in the camera coordinate system, wherein the determined three-dimensional coordinates are the point cloud. Furthermore, the electronic device performs voxel sampling on the obtained point cloud to obtain an initial frame point cloud, and the voxel side length can be 0.005m. Finally, the electronic device determines the initial frame seed point from the initial frame point cloud through graspability sampling, analyzes each initial frame seed point, and obtains multiple grasping postures. Each grasping posture can carry a grasping score. The electronic device grasps the moving object according to the grasping score of each grasping posture and the grasping posture with the highest grasping score.

[0090] When determining the three-dimensional coordinates, the electronic device may use the following formula:

[0091]

[0092] Among them, f x and f y Respectively represent the focal length of the camera on the u-axis and v-axis of the pixel coordinate system, (c x ,c y ) can represent the coordinates of the principal point in the pixel coordinate system, s represents the image depth value scaling ratio, d represents the depth value corresponding to the point, and (u, v) can be any pixel point in the first scene image.

[0093] In the above embodiment, when the first frame is the initial frame, the electronic device can directly analyze and determine the grasping posture corresponding to the first scene image of the first frame based on the first scene image of the first frame, and then grasp the moving object according to the grasping score of each grasping posture according to the grasping posture with the highest grasping score, which ensures a faster reasoning speed to a certain extent and can meet the needs of real-time tracking.

[0094] In which, the degree of association between the first candidate grasping posture set and the second candidate grasping posture set is determined by a grasping association model. In one embodiment, the training method of the grasping association model includes: obtaining a first sample grasping posture set and a second sample posture grasping set of a sample data set; determining the grasping distance between any two sample grasping postures in the first sample grasping posture set and the second sample posture grasping set; comparing each grasping distance with a preset distance threshold to determine a sample target grasping posture set, and the sample target grasping posture set is a set consisting of sample grasping postures of the same type in the second sample posture grasping set and the first sample grasping posture set; determining a loss function value based on the first sample grasping posture set, the second sample posture grasping set and the sample target grasping posture set; iterating the initial weight parameters according to the loss function value, and training to obtain the grasping association model.

[0095] Among them, the grasping association model can be a model pre-trained by an electronic device. After the electronic device preliminarily processes the first scene image and the second scene image, it obtains a first candidate grasping posture set and a second candidate grasping posture set. The first candidate grasping posture set and the second candidate grasping posture set are input into the grasping association model to obtain the degree of association.

[0096] Among them, when the electronic device trains the grasping association model, it needs to obtain a sample data set, which may include a first sample grasping posture set and a second sample posture grasping set. The first sample grasping posture set can be obtained by the electronic device after processing the first sample scene image, and the second sample grasping posture set can be obtained by the electronic device after processing the second sample scene image. The first sample scene image and the second sample scene image are images of two adjacent frames in the same scene.

[0097] Among them, the grasping distance can be used to represent the value of the similarity between any two sample grasping postures in the first sample grasping posture set and the second sample posture grasping set. In this embodiment, the smaller the grasping distance, the higher the similarity between the two sample grasping postures. For grasping postures on different objects, their grasping distances can be set to positive infinity. The preset distance threshold is a pre-set threshold. The electronic device can compare the grasping distance with the preset distance threshold to determine the sample target grasping posture set. Specifically, for any two sample grasping postures g1 and g2 in the first sample grasping posture set and the second sample posture grasping set, the electronic device will mark them as the same category if and only if the distance d(g1, g2) ≤ σ, where σ is a pre-set threshold. For example, given two point clouds C A and C B , electronic device predicted C A The first sample grasping posture set is C B The second sample grasping posture set is Point Cloud C B Zhongyu The set of grasping postures for target samples of the same class can be expressed as

[0098] After obtaining the sample target grasping posture set, the electronic device can determine the loss function value based on the first sample grasping posture set, the second sample posture grasping set and the sample target grasping posture set, wherein the electronic device can determine the degree of correlation between the first sample grasping posture set and each sample grasping posture in the first sample grasping posture set based on the first sample grasping posture set and the second sample posture grasping set, as well as the degree of correlation between the sample grasping posture set and each sample grasping posture in the first sample grasping posture set, and then determine the loss function value in combination with a pre-constructed loss function formula.

[0099] After obtaining the loss function value, the electronic device can input the loss function value and the initial weight parameters into the optimizer of the initial neural network model (Adam optimizer can be used in this model, but is not limited to this optimizer) to obtain the updated value of the network weight parameters of the multilayer perceptron, so as to update the initial weight parameters and obtain the capture association model. In actual training, the above process will be repeated on different data in the data set, and the training will be stopped after several iterations. The number of iterations can be set by yourself. The goal of training is to reduce the loss function value as much as possible.

[0100] In one embodiment, the loss function is as follows:

[0101]

[0102] Among them, |P A (i)| represents the set P A (i) is the number of elements, log and exp represent the logarithmic function and exponential function respectively, s(g1, g2) represents the cosine similarity of the grasping postures g1 and g2 predicted by the grasping association module, τ is a pre-set parameter, in this embodiment σ = 0.1, τ = 0.1, the terminal can input the calculated degree of association into the preset loss function calculation formula to calculate the loss function value.

[0103] In the above embodiment, the electronic device uses supervised contrast learning to perform model training to obtain a target posture prediction model. Through the neural network model, it can be upgraded without significantly changing the model structure during iteration, and has a faster reasoning speed, which can meet the needs of real-time tracking.

[0104] In one embodiment, Figure 4 As shown, the first sample grasping posture set includes the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample candidate grasping feature vector associated with each first sample grasping posture; the second sample grasping posture set includes the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector associated with each second sample grasping posture; determining the grasping distance between any two sample grasping postures in the first sample grasping posture set and the second sample posture grasping set includes the following steps:

[0105] Step S402 : determining a first sample rotation amount and a first sample translation amount of the first sample grasping posture based on the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample candidate grasping feature vector.

[0106] Among them, the first sample rotation and the first sample translation are the position and orientation of the first sample grasping posture determined by the electronic device. Specifically, the electronic device can determine the first sample rotation and the first sample translation of the first sample grasping posture based on the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample candidate grasping feature vector. For example, for the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample grasping feature vector on the same object, they are converted into a unified coordinate system and can be expressed as (R1, t1), where R1 is a matrix of size 3×3, representing the first sample rotation of the first sample grasping posture, and t1 is a vector of length 3, which can represent the first sample translation of the first sample grasping posture.

[0107] Step S404 : determining a second sample rotation amount and a second sample translation amount of the second sample grasping posture according to the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector.

[0108] Among them, the second sample rotation and the second sample translation are the position and orientation of the second sample grasping posture determined by the electronic device. Specifically, the electronic device can determine the second sample rotation and the second sample translation of the second sample grasping posture based on the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector. For example, for the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector on the same object, they are converted into a unified coordinate system and can be expressed as (R2, t2), where R2 is a matrix of size 3×3, representing the second sample rotation of the second sample grasping posture, and t2 is a vector of length 3, representing the second sample translation of the second sample grasping posture.

[0109] Step S406 : determining a rotation distance between the first sample grasping posture and the second sample grasping posture based on the first sample rotation amount and the second sample rotation amount.

[0110] When calculating the rotation distance, the electronic device may use the following formula:

[0111]

[0112] Wherein, ΔR is the rotation distance, trace(A) represents the trace of the matrix A, wherein the matrix A may be determined by the first sample rotation amount and the second sample rotation amount.

[0113] Step S408 : determining a translation distance between the first sample grasping posture and the second sample grasping posture according to the first sample translation amount and the second sample translation amount.

[0114] The electronic device may use the following formula to determine the translation distance:

[0115] Δt=||t1-t2||

[0116] Wherein, ||t|| represents the modulus of vector t, and ||t|| is determined by the first sample translation and the second sample translation.

[0117] Step S410 : determining a grasping distance between a first sample grasping posture and a second sample grasping posture according to the rotation distance and the translation distance.

[0118] When calculating the grasping distance, the electronic device may use the following formula:

[0119]

[0120] Among them, w max Indicates the maximum width of the gripper of the actuator (i.e., the robotic arm), and γ is the weight coefficient. The smaller the grasping distance d(g1, g2), the higher the similarity between the first sample seed point and the second sample seed point (i.e., the two grasping postures). For grasping postures on different objects, their grasping distances can be set to positive infinity. In this implementation, w can be set max =0.1m,γ=0.1.

[0121] In the above embodiment, the electronic device can improve the accuracy of subsequent real-time tracking by accurately calculating the grasping distance.

[0122] In one embodiment, a first scene image and a second scene image are processed by a grasping generation model to obtain a first candidate grasping posture set for the first frame and a second candidate grasping posture set for the second frame, including: obtaining a first scene point cloud of the first scene image; performing point cloud scene analysis on the first scene point cloud to obtain a plurality of first scene seed points; obtaining a second scene point cloud of the second scene image; performing point cloud scene analysis on the second scene point cloud to obtain a plurality of second scene seed points; performing feature extraction processing on each first scene seed point to obtain a corresponding first grasping result, the first grasping result including a first grasping posture prediction value, a first geometric feature vector and a first candidate grasping feature vector; performing feature extraction processing on each second scene seed point to obtain a corresponding second grasping result, the first grasping result including a second grasping posture prediction value, a second geometric feature vector and a second candidate grasping feature vector.

[0123] Among them, when determining the first scene point cloud, the electronic device can calculate the three-dimensional coordinates of any pixel point in the first scene image in the camera coordinate system, and then perform voxel sampling on each obtained three-dimensional coordinate to obtain the first scene point cloud. Furthermore, the electronic device uses graspable sampling to obtain multiple first scene seed points from the first scene point cloud.

[0124] Among them, when determining the second scene point cloud, the electronic device can calculate the three-dimensional coordinates of any pixel point in the second scene image in the camera coordinate system, and then perform voxel sampling on each obtained three-dimensional coordinate to obtain the second scene point cloud. Furthermore, the electronic device uses graspable sampling to obtain multiple second scene seed points from the second scene point cloud.

[0125] Among them, the first grasping result is the first grasping posture prediction value, the first geometric feature vector and the first candidate grasping feature vector obtained after processing the first scene image, and the second grasping result is the second grasping posture prediction value, the second geometric feature vector and the second candidate grasping feature vector obtained after processing the second scene image.

[0126] Specifically, the electronic device can use parallel processing to process the first scene image and the second scene image, that is, use the same processing method to process the first scene image and the second scene image at the same time, so as to obtain the first scene seed point corresponding to the first scene image and the second scene seed point corresponding to the second scene image, and then perform feature extraction processing on the first scene seed point and the second scene seed point respectively to obtain the first capture result and the second capture result.

[0127] In the above embodiment, the electronic device processes the first sample scene image and the second sample scene image of the same scene by capturing the generation model, thereby getting rid of the dependence on the object itself, and is still effective for unknown objects. There is no need to obtain object information and annotations in advance, which greatly broadens the application scenarios of the algorithm and improves the convenience of use.

[0128] In one embodiment, Figure 5 As shown, the first candidate grasping posture set includes a first grasping posture prediction value, a first geometric feature vector, and a first candidate grasping feature vector associated with each first grasping posture; the second candidate grasping posture set includes a second grasping posture prediction value, a second geometric feature vector, and a second candidate grasping feature vector associated with each grasping posture;

[0129] Processing a first candidate grasping posture set and a second candidate grasping posture set by a grasping association model to determine a degree of association between the first candidate grasping posture set and the second candidate grasping posture set includes the following steps:

[0130] Step S502 : performing color feature processing on the first grasping posture to obtain a first color feature vector corresponding to the first grasping posture.

[0131] Among them, color feature processing is the process of the electronic device obtaining the color information of the area where the first grasping posture is located. Specifically, for the first grasping posture, the electronic device can find the corresponding color from the first scene seed point, and then randomly sample the seed point colors in the cylindrical space near each first scene seed point according to the point cloud cutting method in the GSNet model. Finally, the first color feature vector is obtained through multi-layer perceptron and maximum pooling operations.

[0132] Step S504 : performing color feature processing on the second grasping posture to obtain a second color feature vector corresponding to the second grasping posture.

[0133] Among them, for the second grasping posture, the electronic device can find the corresponding color from the second scene seed point, and then randomly sample the seed point colors in the cylindrical space near each second scene seed point according to the point cloud cutting method in the GSNet model, and finally obtain the second color feature vector through multi-layer perceptron and maximum pooling operations.

[0134] Step S506 : performing vector processing on the first grasping posture prediction value, the first geometric feature vector, and the first color feature vector, and splicing them onto the first candidate grasping feature vector to obtain a first initial spliced feature vector.

[0135] Among them, the electronic device can update the first candidate grasping feature vector to obtain a first initial splicing feature vector. Specifically, the electronic device can flatten the obtained first color feature vector and the first geometric feature vector associated with the first grasping posture, and the first grasping posture prediction value (including grasping translation parameters (3-dimensional vector), grasping rotation parameters 3×3 rotation matrix) into a 9-dimensional vector, and then splice it to the corresponding first candidate grasping feature vector to obtain the first initial splicing feature vector.

[0136] Step S508 : performing vector processing on the second grasping posture prediction value, the second geometric feature vector, and the second color feature vector, and splicing them onto the second candidate grasping feature vector to obtain a second initial spliced feature vector.

[0137] Among them, the electronic device can update the second candidate grasping feature vector to obtain a second initial splicing feature vector. Specifically, the electronic device can flatten the obtained second color feature vector and the second geometric feature vector associated with the second grasping posture, and the second grasping posture prediction value (including grasping translation parameters (3-dimensional vector), grasping rotation parameters 3×3 rotation matrix) into a 9-dimensional vector, and then splice it to the corresponding second candidate grasping feature vector to obtain the second initial splicing feature vector.

[0138] Step S510 : Process the first initial splicing feature vector and the second initial splicing feature vector respectively to obtain corresponding first target splicing feature vector and second target splicing feature vector.

[0139] After obtaining the first initial splicing feature vector and the second initial splicing feature vector, the electronic device may use a multilayer perceptron to process the first initial splicing feature vector and the second initial splicing feature vector to obtain a first target splicing feature vector and a second target splicing feature vector.

[0140] Step S512 : obtaining a correlation degree between the first candidate grasping posture set and the second candidate grasping posture set based on the first target splicing feature vector and the second target splicing feature vector.

[0141] Among them, for each first grasping posture, the electronic device can obtain its corresponding first target splicing feature vector, and for each second grasping posture, the electronic device can obtain its corresponding second target splicing feature vector. Furthermore, the electronic device can calculate the correlation matrix between the first grasping posture and the second grasping posture based on the obtained first target splicing feature vectors and second target splicing feature vectors.

[0142] In one embodiment, for any of the first grasping postures and the second grasping postures, the corresponding color feature vector is obtained through the following steps: determining the seed point color information of the grasping posture; sampling the seed point color information to obtain an initial color vector; and performing feature processing on the initial color vector in turn to obtain a corresponding color feature vector.

[0143] Among them, the seed point color information is the color determined by the electronic device from the scene seed point, the initial color vector is obtained by the electronic device randomly sampling the seed point color information, and the feature processing can be to perform multi-layer perceptron and maximum pooling operations on the initial color feature vector in sequence to obtain a color feature vector.

[0144] In the above embodiment, the electronic device uses the geometric and color features of the local area to predict the degree of correlation between grasping postures, thus getting rid of the dependence on the object itself. It is still effective for unknown objects and does not require obtaining object information and annotations in advance. It greatly broadens the application scenarios of the algorithm and improves the convenience of use.

[0145] In one embodiment, the degree of association is an association matrix; based on the first target splicing feature vector and the second target splicing feature vector, the degree of association between the first candidate grasping posture set and the second candidate grasping posture set is obtained, including: determining the cosine similarity between the first target splicing feature vector and the second target splicing feature vector; and obtaining the association matrix according to the cosine similarity.

[0146] The cosine similarity may be a quantity used to characterize the degree of similarity between the first target splicing feature vector and the second target splicing feature vector. Specifically, when determining the cosine similarity, the electronic device may use the following formula to perform the calculation:

[0147]

[0148] Among them, f1 and f2 can represent the first target splicing feature vector and the second target splicing feature vector respectively, s(g1, g2) can be the cosine similarity, ||f|| represents the modulus of vector f, then ||f1|| can represent the modulus of vector f1, ||f2|| can represent the modulus of vector f2, and the distribution range of s(g1, g2) can be between [-1, 1]. The higher its value, the greater the degree of correlation between the two vectors.

[0149] In the above embodiment, the electronic device can accurately determine the correlation matrix by calculating the cosine similarity, thereby improving the accuracy of subsequent real-time tracking.

[0150] In one embodiment, Figure 6 FIG. 1 is a flowchart of an overall process of determining an association matrix in one embodiment.

[0151] in, Figure 6 The main focus is on the execution process of the crawl generation module (i.e., the crawl generation model) and the crawl association module (i.e., the crawl association model). First, for the crawl generation module, the following steps are specifically included:

[0152] 1) Obtaining the target point cloud output by the data acquisition and processing module. In this embodiment, the target point clouds are obtained from the first scene image and the second scene image of two adjacent frames captured by the camera (point cloud 1 and point cloud 2 in the figure);

[0153] 2) The grasp detection network analyzes the point cloud scene corresponding to the target point cloud to obtain several grasp posture prediction values. Among them, the grasp posture prediction value includes a translation vector of length 3 (xyz) and a 3×3 rotation matrix of grasp rotation parameters. The grasp posture prediction value can provide candidates for subsequent grasp tracking.

[0154] 3) The grasp generation module also generates the geometric feature vectors of the seed point and the feature vectors of the candidate grasps. To facilitate the correlation between candidate grasp poses, the geometric feature vectors of the seed point and the feature vectors of the candidate grasps can be extracted from the network. Each feature vector has a size of M × C, where the seed point is the anchor point for generating the grasp pose and C represents the length of the feature vector. In this embodiment, M = 1024 and C = 128 are used. The size of N is not fixed and is determined by the result of point cloud downsampling.

[0155] For the capture association module, the following steps are specifically included: the module includes C1 color feature aggregation submodule and C2 association matrix generation submodule;

[0156] a) According to the C1 color feature aggregation submodule, two sets of seed point color feature vectors are obtained. (In this step, it is done for point cloud 1 and point cloud 2 respectively), wherein the color information of the areas where adjacent grasping postures are located has a high similarity, so that the degree of correlation between grasping postures can be judged by the color information of the local area of the seed point, further improving the accuracy of the prediction. For a set of seed points, first find the corresponding color from the original point cloud, the size is M×3; then, according to the point cloud cropping method in the GSNet model, the seed point color in the cylindrical space near each seed point is randomly sampled to obtain a vector of size M×K×3, where K represents the number of samples in each cylindrical space; finally, after a multi-layer perceptron of size (2C, C, C) and a maximum pooling operation, a seed point color feature vector of size M×C is obtained. In this embodiment, the sampling number K is set to 16.

[0157] b) Input the seed point's geometric features, grasp features, and color feature vectors. Using the correlation matrix generation module, a correlation matrix is generated between the two sets of feature vectors. This correlation matrix is then transformed into an M×M correlation matrix for candidate grasp poses. This matrix is then used to track the grasp poses frame by frame and select the optimal pose. The specific implementation steps are as follows: The grasp feature vector is updated. The resulting color feature vector is concatenated with the seed point's geometric feature vector, the grasp translation parameters (a 3D vector), and the grasp rotation parameters (a 3×3 rotation matrix flattened into a 9D vector) to the corresponding grasp feature vector, resulting in an M×(3C+12) feature vector. This is then passed through a multilayer perceptron of size (3C, 3C, C) to obtain an M×C feature vector. This process extracts feature vectors for each point cloud, and the correlation matrix between the two sets of feature vectors is calculated.

[0158] Among them, for the grasping postures g1 and g2 belonging to two point clouds, their feature vectors are f1 and f2 respectively, and their correlation degree is defined as the cosine similarity between the vectors: Where ||f|| represents the modulus of vector f, and s(g1, g2) ranges from [-1, 1]. Higher values indicate a greater correlation between the two vectors. Ultimately, the network outputs an M×M correlation matrix. Based on the grasping posture selected in the previous frame, the most similar grasping posture in the current frame can be determined, enabling real-time tracking.

[0159] In one embodiment, the degree of association is an association matrix; based on the first grasping posture and the degree of association, a second grasping posture corresponding to the second frame is determined from a second candidate grasping posture set, including: based on the number of rows of the association matrix where the first grasping posture is located, the grasping posture with the highest grasping association degree in the second candidate grasping posture set is determined as the second grasping posture of the second frame.

[0160] When the electronic device selects the second grasping posture of the second frame according to the correlation matrix and the first grasping posture of the first frame, it can first determine the first candidate grasping posture set, that is, the first candidate grasping posture set can be expressed as And the second candidate grasping posture set, that is, the second candidate grasping posture set can be expressed as The association matrix predicted by the crawl association module can be S, with a size of M×M, and the value S in the i-th row and j-th column of the matrix ij express and The degree of crawling relevance, that is, If the i0th first grasping posture has been selected in the first frame As output, the second grasping posture of the second frame is output The selection method can be That is, from the number of rows in the association matrix where the first grasping posture is located, from all possible In , the maximum value of the number of columns with the highest degree of grasping association is selected as the second grasping posture.

[0161] In the above embodiment, the electronic device tracks the moving object using the correlation degree of the grasping posture, thereby avoiding the problem of grasping posture jump caused by using only the grasping detection algorithm, and improving the stability and accuracy of control.

[0162] In one embodiment, Figure 7 FIG. 1 is a flow chart of a method for grasping a moving object in one embodiment:

[0163] In this embodiment, when grasping a moving object, the following steps are mainly included:

[0164] 1. First, before grasping a moving object, you can train a grasping association model. When training a grasping association model, the following steps are specifically included:

[0165] Among them, when the electronic device trains the grasping association model, it needs to obtain a sample data set, which may include a first sample grasping posture set and a second sample posture grasping set. The first sample grasping posture set can be obtained by the electronic device after processing the first sample scene image, and the second sample grasping posture set can be obtained by the electronic device after processing the second sample scene image. The first sample scene image and the second sample scene image are images of two adjacent frames in the same scene.

[0166] Specifically, the electronic device can use parallel processing to process the first sample scene image and the second sample scene image, that is, use the same processing method to process the first sample scene image and the second sample scene image at the same time, so as to obtain the first sample scene seed point corresponding to the first sample scene image and the second sample scene seed point corresponding to the second sample scene image, and then perform feature extraction processing on the first sample scene seed point and the second scene sample seed point respectively to obtain the first sample capture result and the second sample capture result.

[0167] Among them, the first sample grasping result is the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample candidate grasping feature vector associated with the first sample grasping posture obtained after processing the first sample scene image, and the second grasping result is the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector associated with each second sample grasping posture obtained after processing the second scene image.

[0168] The electronic device can determine the first sample rotation and the first sample translation of the first sample grasping posture based on the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample candidate grasping feature vector. For example, for the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample grasping feature vector on the same object, they are converted into a unified coordinate system and can be expressed as (R1, t1), where R1 is a matrix of size 3×3, representing the first sample rotation of the first sample grasping posture, and t1 is a vector of length 3, which can represent the first sample translation of the first sample grasping posture.

[0169] The electronic device can determine the second sample rotation and the second sample translation of the second sample grasping posture based on the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector. For example, for the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector on the same object, they are converted into a unified coordinate system and can be expressed as (R2, t2), where R2 is a matrix of size 3×3, representing the second sample rotation of the second sample grasping posture, and t2 is a vector of length 3, representing the second sample translation of the second sample grasping posture.

[0170] Furthermore, the electronic device determines the rotation distance between the first sample grasping posture and the second sample grasping posture based on the first sample rotation amount and the second sample rotation amount, determines the translation distance between the first sample grasping posture and the second sample grasping posture based on the first sample translation amount and the second sample translation amount, determines the grasping distance between the first sample grasping posture and the second sample grasping posture based on the rotation distance and the translation distance, and then compares the grasping distance with the preset distance threshold to determine the sample target grasping posture set. Specifically, for any two sample grasping postures g1 and g2 in the first sample grasping posture set and the second sample posture grasping set, the electronic device marks them as the same category if and only if the distance d(g1, g2) ≤ σ, where σ is a pre-set threshold. For example, given two point clouds C A and C B , electronic device predicted C A The first sample grasping posture set is C B The second sample grasping posture set is Point Cloud C B Zhongyu The set of grasping postures for target samples of the same class can be expressed as

[0171] After obtaining the sample target grasping posture set, the electronic device can determine the loss function value based on the first sample grasping posture set, the second sample posture grasping set and the sample target grasping posture set, wherein the electronic device can determine the degree of correlation between the first sample grasping posture set and each sample grasping posture in the first sample grasping posture set based on the first sample grasping posture set and the second sample posture grasping set, as well as the degree of correlation between the sample grasping posture set and each sample grasping posture in the first sample grasping posture set, and then determine the loss function value in combination with a pre-constructed loss function formula.

[0172] After obtaining the loss function value, the electronic device can input the loss function value and the initial weight parameters into the optimizer of the initial neural network model (Adam optimizer can be used in this model, but is not limited to this optimizer) to obtain the updated value of the network weight parameters of the multilayer perceptron, so as to update the initial weight parameters and obtain the capture association model. In actual training, the above process will be repeated on different data in the data set, and the training will be stopped after several iterations. The number of iterations can be set by yourself. The goal of training is to reduce the loss function value as much as possible.

[0173] 2. In the actual grasping operation process, on the one hand, the electronic device can obtain the trained grasping generation model and grasping association model (the two can together form Figure 7 Grasping posture model in );

[0174] 3. On the other hand, the camera collects the scene depth RGB-D picture at the current moment, sets the current frame collected at the current moment as the initial frame, and calls the obtained grasp generation model and grasp association model according to the current scene;

[0175] 4. In the case where the current frame at the current moment is the initial frame, the electronic device can determine the corresponding three-dimensional coordinates (x, y, z) in the camera coordinate system for any pixel point (u, v) in the initial frame image, where the determined three-dimensional coordinates are the point cloud. Further, the electronic device performs voxel sampling on the obtained point cloud to obtain the initial frame point cloud, where the voxel side length can be 0.005m. Finally, the electronic device determines the initial frame seed point from the initial frame point cloud through graspability sampling, analyzes each initial frame seed point, and obtains multiple initial candidate grasping postures. Each initial candidate grasping posture can carry a grasping score. The electronic device determines the initial candidate grasping posture with the highest grasping score as the target grasping posture based on the grasping scores of the initial candidate grasping postures.

[0176] 5. On the other hand, when the current frame at the current moment is set as the initial frame, but the initial frame capture fails, the camera can capture the next frame of the scene RGB-D image after the current frame.

[0177] 6. Process the RGB-D image of the current frame and the RGB-D image of the next frame into a 3D point cloud, and input the above 3D point cloud into the trained grasp generation model and grasp association model;

[0178] 7. The grasping association model processes the point cloud of the current frame and the point cloud of the next frame in parallel to obtain the grasping posture prediction value, the first geometric feature vector and the first candidate grasping feature vector associated with the grasping posture. The grasping association model can calculate the grasping posture prediction value, the first geometric feature vector and the first candidate grasping feature vector associated with the grasping posture of the grasping generation module to obtain an association matrix. The association matrix is an association matrix for representing the degree of association between the grasping postures of the current frame and the next frame. Specifically, when generating the association matrix, the electronic device predicts the candidate grasping posture through the grasping generation module, and extracts the seed point geometric feature vector and the grasping feature vector, and further uses the grasping association module to obtain the seed point color vector, and then updates the grasping feature vector according to the seed point geometric feature vector and the seed point color vector to obtain the association matrix;

[0179] 8. According to the current frame grasping posture and the association matrix of the current frame, the target grasping posture of the next frame is selected. Specifically, when the electronic device selects the target grasping posture of the next frame according to the association matrix and the current frame grasping posture of the current frame, the grasping posture set of the current frame can be determined first. That is, the grasping posture set of the current frame can be expressed as The grasping posture sets of the next frame can be The association matrix predicted by the crawl association module can be S, with a size of M×M, and the value S in the i-th row and j-th column of the matrix ij express and The captured associated value, that is, If the i0th grasping posture has been selected in the previous frame As output, the grasping posture selection method for the next frame output can be That is, from the number of rows in the association matrix where the current frame grasp posture is located, from all possible In , the maximum value of the number of columns with the highest grasp correlation is selected as the target grasping posture.

[0180] 9. The robot arm adjusts its motion trajectory according to the predicted target grasping posture (i.e., the grasping posture with a high degree of grasping correlation);

[0181] 10. Whether the gripper reaches the target grasping posture;

[0182] 11. If the target grasping posture is reached, the robot arm grasps the moving object and places it at the specified location. If the target grasping posture is not reached, the robot continues to collect RGB-D images and re-predicts the target grasping posture.

[0183] 12. Check whether there is an object in the scene. If not, end the capture process. Otherwise, reacquire the RGB-D image of the new scene and start the capture.

[0184] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0185] Based on the same inventive concept, embodiments of the present application also provide an object grasping device for implementing the object grasping method described above. The solution to the problem provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of the embodiments of the grasping device for one or more objects provided below can be found in the above-mentioned limitations of the object grasping method and will not be repeated here.

[0186] In one embodiment, Figure 8 As shown, a device for grasping an object is provided, comprising: an image acquisition module, a prediction module, a posture determination module and a grasping module, wherein:

[0187] The image acquisition module 802 is configured to acquire a first scene image of a first frame including an object, and a second scene image of a subsequent second frame also including the object.

[0188] The prediction module 804 is used to obtain a first candidate grasping posture set of the first frame and a second candidate grasping posture set of the second frame based on the first scene image and the second scene image, and obtain the degree of correlation between the first candidate grasping posture set and the second candidate grasping posture set.

[0189] The posture determination module 806 is configured to determine a first grasping posture corresponding to the first frame from the first candidate grasping posture set, and determine a second grasping posture corresponding to the second frame from the second candidate grasping posture set based on the first grasping posture and the association degree.

[0190] The grasping module 808 is used to control the actuator to grasp the moving object according to the second grasping posture.

[0191] In one embodiment, the prediction module is further used to process the first scene image and the second scene image through a grasping generation model to obtain a first candidate grasping posture set of the first frame and a second candidate grasping posture set of the second frame; and process the first candidate grasping posture set and the second candidate grasping posture set through a grasping association model to determine the degree of association between the first candidate grasping posture set and the second candidate grasping posture set.

[0192] In one embodiment, the apparatus further comprises a model training module;

[0193] A model training module is used to obtain a first sample grasping posture set and a second sample posture grasping set of a sample data set; determine the grasping distance between any two sample grasping postures in the first sample grasping posture set and the second sample posture grasping set; compare each grasping distance with a preset distance threshold to determine a sample target grasping posture set, where the sample target grasping posture set is a set consisting of sample grasping postures of the same type in the second sample posture grasping set and the first sample grasping posture set; determine a loss function value based on the first sample grasping posture set, the second sample posture grasping set and the sample target grasping posture set; iterate the initial weight parameters according to the loss function value, and train to obtain a grasping association model.

[0194] In one embodiment, the first sample grasping posture set includes the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample candidate grasping feature vector associated with each first sample grasping posture; the second sample grasping posture set includes the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector associated with each second sample grasping posture; the model training module is further used to determine the first sample rotation amount and the first sample translation amount of the first sample grasping posture based on the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample candidate grasping feature vector; determine the second sample rotation amount and the second sample translation amount of the second sample grasping posture according to the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector; determine the rotation distance between the first sample grasping posture and the second sample grasping posture based on the first sample rotation amount and the second sample rotation amount; determine the translation distance between the first sample grasping posture and the second sample grasping posture according to the first sample translation amount and the second sample translation amount; and determine the grasping distance between the first sample grasping posture and the second sample grasping posture according to the rotation distance and the translation distance.

[0195] In one embodiment, the prediction module is also used to obtain a first scene point cloud of a first scene image; perform point cloud scene analysis on the first scene point cloud to obtain multiple first scene seed points; obtain a second scene point cloud of a second scene image; perform point cloud scene analysis on the second scene point cloud to obtain multiple second scene seed points; perform feature extraction processing on each first scene seed point to obtain a corresponding first grasping result, the first grasping result includes a first grasping posture prediction value, a first geometric feature vector and a first candidate grasping feature vector; perform feature extraction processing on each second scene seed point to obtain a corresponding second grasping result, the first grasping result includes a second grasping posture prediction value, a second geometric feature vector and a second candidate grasping feature vector.

[0196] In one embodiment, the first candidate grasping posture set includes a first grasping posture prediction value, a first geometric feature vector, and a first candidate grasping feature vector associated with each first grasping posture; the second candidate grasping posture set includes a second grasping posture prediction value, a second geometric feature vector, and a second candidate grasping feature vector associated with each grasping posture; the device further includes an association matrix processing module;

[0197] An association matrix processing module is used to perform color feature processing on the first grasping posture to obtain a first color feature vector corresponding to the first grasping posture; perform color feature processing on the second grasping posture to obtain a second color feature vector corresponding to the second grasping posture; perform vector processing on the first grasping posture prediction value, the first geometric feature vector and the first color feature vector, and splice them onto the first candidate grasping feature vector to obtain a first initial splicing feature vector; perform vector processing on the second grasping posture prediction value, the second geometric feature vector and the second color feature vector, and splice them onto the second candidate grasping feature vector to obtain a second initial splicing feature vector; process the first initial splicing feature vector and the second initial splicing feature vector respectively to obtain the corresponding first target splicing feature vector and second target splicing feature vector; based on the first target splicing feature vector and the second target splicing feature vector, obtain the degree of association between the first candidate grasping posture set and the second candidate grasping posture set.

[0198] In one embodiment, the correlation matrix processing module is further used to determine the seed point color information of the grasping posture; sample the seed point color information to obtain an initial color vector; and perform feature processing on the initial color vector in turn to obtain a corresponding color feature vector.

[0199] In one embodiment, the correlation matrix processing module is further configured to determine the cosine similarity between the first target splicing feature vector and the second target splicing feature vector; and obtain the correlation matrix according to the cosine similarity.

[0200] In one embodiment, the grasping module is further configured to determine the grasping posture with the highest grasping correlation in the second candidate grasping posture set as the second grasping posture of the second frame based on the row number of the correlation matrix where the first grasping posture is located.

[0201] Each module in the aforementioned object grasping device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in an electronic device in hardware form, or may be stored in a memory in the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0202] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for grasping an object is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0203] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0204] In one embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned object grasping method when executing the computer program.

[0205] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned object grasping method are implemented.

[0206] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned object grasping method when executed by a processor.

[0207] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0208] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0209] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0210] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for grasping an object, characterized in that: The method comprises: Acquire a first scene image of a first frame including the object, and a second scene image of a subsequent second frame including the object; Processing the first scene image and the second scene image through a grasp generation model to obtain a first candidate grasping posture set for the first frame and a second candidate grasping posture set for the second frame; Processing the first candidate grasping posture set and the second candidate grasping posture set through a grasping association model to determine a degree of association between the first candidate grasping posture set and the second candidate grasping posture set; Determining a first grasping posture corresponding to the first frame from the first candidate grasping posture set, and determining a second grasping posture corresponding to the second frame from the second candidate grasping posture set based on the first grasping posture and the degree of association; wherein the greater the cosine similarity between the feature vector of the first grasping posture and the feature vector of any second candidate grasping posture in the second candidate grasping posture set, the greater the degree of association, and selecting the second candidate grasping posture with the higher degree of association as the second grasping posture for the second frame; The control execution mechanism grasps the moving object according to the second grasping posture.

2. The method according to claim 1, characterized in that The training method of the crawling association model includes: Obtain a first sample grasping posture set and a second sample posture grasping posture set of a sample data set; Determining a grasping distance between any two sample grasping postures in the first sample grasping posture set and the second sample grasping posture set; Comparing each of the grasping distances with a preset distance threshold to determine a sample target grasping posture set, where the sample target grasping posture set is a set consisting of sample grasping postures of the same type in the second sample posture grasping set and the first sample grasping posture set; Determining a loss function value based on the first sample grasping posture set, the second sample grasping posture set, and the sample target grasping posture set; According to the loss function value, the initial weight parameters are iterated to train and obtain the grasp association model.

3. The method according to claim 2, characterized in that The first sample grasping posture set includes a first sample grasping posture prediction value, a first sample geometric feature vector, and a first sample candidate grasping feature vector associated with each first sample grasping posture; the second sample posture grasping set includes a second sample grasping posture prediction value, a second sample geometric feature vector, and a second sample candidate grasping feature vector associated with each second sample grasping posture; The determining of the grasping distance between any two sample grasping postures in the first sample grasping posture set and the second sample grasping posture set includes: Determining a first sample rotation amount and a first sample translation amount of the first sample grasping posture based on the first sample grasping posture prediction value, the first sample geometric feature vector, and the first sample candidate grasping feature vector; determining a second sample rotation amount and a second sample translation amount of the second sample grasping posture according to the second sample grasping posture prediction value, the second sample geometric feature vector, and the second sample candidate grasping feature vector; determining a rotation distance between the first sample grasping posture and the second sample grasping posture based on the first sample rotation amount and the second sample rotation amount; determining a translation distance between the first sample grasping posture and the second sample grasping posture according to the first sample translation amount and the second sample translation amount; A grasping distance between the first sample grasping posture and the second sample grasping posture is determined according to the rotation distance and the translation distance.

4. The method according to claim 1, characterized in that The step of processing the first scene image and the second scene image by using a grasping generation model to obtain a first candidate grasping posture set for the first frame and a second candidate grasping posture set for the second frame includes: Acquire a first scene point cloud of the first scene image; Performing point cloud scene analysis on the first scene point cloud to obtain a plurality of first scene seed points; Acquire a second scene point cloud of the second scene image; Performing point cloud scene analysis on the second scene point cloud to obtain a plurality of second scene seed points; Performing feature extraction processing on each first scene seed point to obtain a corresponding first grasping result, where the first grasping result includes a first grasping posture prediction value, a first geometric feature vector, and a first candidate grasping feature vector; Feature extraction processing is performed on each second scene seed point to obtain a corresponding second grasping result, where the first grasping result includes a second grasping posture prediction value, a second geometric feature vector, and a second candidate grasping feature vector.

5. The method according to claim 1, wherein The first candidate grasping posture set includes a first grasping posture prediction value, a first geometric feature vector, and a first candidate grasping feature vector associated with each first grasping posture; the second candidate grasping posture set includes a second grasping posture prediction value, a second geometric feature vector, and a second candidate grasping feature vector associated with each grasping posture; The processing of the first candidate grasping posture set and the second candidate grasping posture set by using a grasping association model to determine the degree of association between the first candidate grasping posture set and the second candidate grasping posture set includes: Performing color feature processing on the first grasping posture to obtain a first color feature vector corresponding to the first grasping posture; performing color feature processing on the second grasping posture to obtain a second color feature vector corresponding to the second grasping posture; Performing vector processing on the first grasping posture prediction value, the first geometric feature vector, and the first color feature vector, and splicing the resultant vectors onto the first candidate grasping feature vector to obtain a first initial spliced feature vector; performing vector processing on the second grasping posture prediction value, the second geometric feature vector, and the second color feature vector, and splicing the resultant vectors onto the second candidate grasping feature vector to obtain a second initial spliced feature vector; Processing the first initial splicing feature vector and the second initial splicing feature vector respectively to obtain a corresponding first target splicing feature vector and a second target splicing feature vector; Based on the first target splicing feature vector and the second target splicing feature vector, a correlation degree between the first candidate grasping posture set and the second candidate grasping posture set is obtained.

6. The method according to claim 5, characterized in that For any of the first grasping postures and the second grasping postures, the corresponding color feature vector is obtained by the following steps: Determine the color information of the seed point of the grasping posture; Sampling the color information of the seed point to obtain an initial color vector; The initial color vectors are sequentially subjected to feature processing to obtain corresponding color feature vectors.

7. The method according to claim 5, characterized in that The correlation degree is a correlation matrix; the correlation degree between the first candidate grasping posture set and the second candidate grasping posture set is obtained based on the first target splicing feature vector and the second target splicing feature vector, including: Determining a cosine similarity between the first target splicing feature vector and the second target splicing feature vector; According to the cosine similarity, a correlation matrix is obtained.

8. The method according to claim 1, characterized in that The degree of association is an association matrix; The determining, based on the first grasping posture and the association degree, a second grasping posture corresponding to the second frame from the second candidate grasping posture set includes: Based on the number of rows of the association matrix where the first grasping posture is located, the grasping posture with the highest grasping association degree in the second candidate grasping posture set is determined as the second grasping posture of the second frame.

9. A device for grasping an object, characterized in that: A method for grasping an object according to any one of claims 1 to 8; the device comprising: An image acquisition module, configured to acquire a first scene image of a first frame including the object, and a second scene image of a subsequent second frame including the object; a prediction module, configured to process the first scene image and the second scene image using a grasp generation model to obtain a first set of candidate grasping postures for the first frame and a second set of candidate grasping postures for the second frame; and process the first set of candidate grasping postures and the second set of candidate grasping postures using a grasp association model to determine a degree of association between the first set of candidate grasping postures and the second set of candidate grasping postures; a posture determination module, configured to determine a first grasping posture corresponding to the first frame from the first candidate grasping posture set, and determine a second grasping posture corresponding to the second frame from the second candidate grasping posture set based on the first grasping posture and the degree of association; wherein the greater the cosine similarity between the feature vector of the first grasping posture and the feature vector of any second candidate grasping posture in the second candidate grasping posture set, the greater the degree of association, and the second candidate grasping posture with the higher degree of association is selected as the second grasping posture of the second frame; The grasping module is used to control the actuator to grasp the moving object according to the second grasping posture.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images