Precise object grabbing and control method based on graph vector database

Abstract

The invention discloses an accurate object grabbing and control method based on a graph vector database, and the method comprises the steps: collecting a multi-view RGB-D image of a scene for color and depth RGB-D sensors at different positions, carrying out the grabbing pose estimation of a target object for the multi-view RGB-D image, and entering a target object placement stage; in the target object placement stage, a multi-view RGB-D image and a final state anchor point image are received, and reasoning is performed in combination with context data in a graph vector database, so that accurate end-to-end object control can be performed by the scheme; before reasoning, a current image and a final state anchor point vector are used for querying a graph vector database, context information related to current control is obtained, and the model reasoning accuracy is improved; furthermore, the pose estimation of the target object is separated into rotation estimation and translation estimation, and the algorithm complexity and the graph database state dimension are reduced. The method does not depend on a prior accurate target object model, and the requirements for accurate control and high stability of the target object in actual industrial application are met.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a precise object grasping and manipulation technology based on a graph vector database. Background Art

[0002] Object grasping and manipulation algorithms play a vital role in the current development of science and technology. First, these algorithms need to accurately estimate the position and posture of objects in three-dimensional space. This process involves converting the object from the world coordinate system to the camera coordinate system and estimating and tracking the six degrees of freedom (6DoF) of the object. These six degrees of freedom include the freedom of movement along the three rectangular coordinate axes of x, y, and z, and the freedom of rotation around these three coordinate axes, namely, front-to-back (x-axis), up-down (y-axis), left-to-right (z-axis) movement, and pitch, yaw, and roll rotation.

[0003] Once the 6DoF posture of the object is obtained, the algorithm will select the optimal grasping point according to the grasping strategy, and finally achieve precise control of the target object. These algorithms are widely used in modern industrial production, incoming material assembly, disordered sorting and grasping, etc. For example, in the field of disordered sorting and grasping, vision-based grasping and control algorithms can help robots grasp disordered objects in the material frame more efficiently, thereby significantly improving the grasping success rate. In the field of robot vision, these vision-based grasping and control algorithms can help robots better perceive the environment, thereby improving their autonomy and intelligence. In the field of motion control, these algorithms can help robots accurately control their motion trajectories, thereby improving motion accuracy and stability.

[0004] In general, object grasping and manipulation algorithms not only promote the development of robotics, but also show great potential and value in multiple industrial application scenarios. The continuous optimization and development of these algorithms will undoubtedly further improve the efficiency, accuracy and intelligence level of robotic systems.

[0005] Existing object grasping and manipulation methods include methods based on precise object models, methods based on generated object models, and end-to-end object manipulation methods.

[0006] 1. Methods based on accurate object models: This method is a classic method in traditional object grasping and manipulation, which relies on accurate modeling of the detailed geometric and physical characteristics of the target object. One implementation method is to use these models to fully sample the target object in the possible SE(3) space through rendering technology, extract sufficiently robust templates from them, and then roughly estimate the object's pose by matching the templates. Finally, the result is refined by the iterative closest point (ICP) algorithm to obtain the object's accurate pose. Next, the best grasping point is selected based on the estimated pose, and finally the object is grasped and manipulated. In addition, the accurate model of the target object can be used in combination with deep learning technology to train the target pose estimation algorithm. The robot then uses the trained model to accurately estimate the 6-degree-of-freedom (6DoF) pose of the object in the scene, thereby completing the grasping and manipulation of the object.

[0007] This type of method has high requirements for prior information, and it relies on accurate modeling of the detailed geometry and physical properties of the target object. This means that a large amount of object model information needs to be acquired and stored in advance, which is not only time-consuming and labor-intensive, but also has high requirements for data integrity and accuracy. Secondly, this type of method has poor environmental adaptability. Since it relies on pre-stored accurate models, it has poor adaptability when facing environmental changes or object deformation. When the shape or position of the object changes, it needs to be re-acquired and modeled, which reduces its application flexibility in dynamic and complex environments. Finally, its ability to handle irregular or unknown objects is also very limited. Methods based on precise models are mainly applicable to known and regular objects. For objects with complex shapes or lack of prior models, the accuracy and stability of their grasping and manipulation may be limited.

[0008] 2. Methods based on generative object models: Methods based on generative object models simplify the representation of objects by using generative models. These generative models generate approximate object models based on partial observation data, thereby reducing the dependence on precise geometric data. For example, point cloud data or depth maps can be used to generate a 3D model of an object. This method shows greater flexibility and adaptability when dealing with new objects and changing environments. After obtaining the generative model of the object, these models are used to estimate the six degrees of freedom (6DoF) posture of the object, and the posture estimation results of the object are corrected and optimized by comparing with the actual observation data. According to the generated object model and the posture estimation results, the optimal grasping point is selected, and the grasping action is performed according to the calculated grasping point and grasping path. Finally, the robot can show high adaptability in a changing environment while achieving accurate grasping and manipulation of the target object.

[0009] The first disadvantage of this type of method is that the accuracy of the generated model may be low. The method of generating object models relies on partial observation data to generate an approximate three-dimensional model of the object, which means that the generated model may not fully and accurately describe the geometric details of the object. This will cause errors in the grasping and manipulation process, thus affecting the success rate and accuracy of the operation. Secondly, this type of method has high computational complexity. The process of generating object models usually involves complex calculations, especially when the model needs to be generated and optimized in real time. This places high demands on the system's computing resources and response time, and may not perform well in high-real-time applications. Finally, the stability of the generated model is insufficient. The generated model may produce unstable results when dealing with unknown or complex objects. For example, the accuracy and stability of the generated model of some objects with complex shapes or partial occlusions may be poor, affecting the subsequent grasping and manipulation effects.

[0010] 3. End-to-end object manipulation method: The end-to-end object manipulation method uses advanced technologies such as deep learning and reinforcement learning to achieve object grasping and manipulation by directly mapping sensor data (such as images, depth maps) to control signals. The advantage of this method lies in its high adaptability and ability to handle complex environments. Since there is no need for intermediate object modeling, the end-to-end method can directly learn the characteristics and grasping strategies of objects and adapt to various objects and scenes.

[0011] Compared with the first and second methods, end-to-end object manipulation methods have shown significant advantages in adaptability and handling complex environments, but there are also some problems that cannot be ignored. The most prominent problems are the predictability of the model, as well as the accuracy and stability of object manipulation. Deep learning models are often regarded as "black boxes" and their internal decision-making processes are difficult to explain and understand. This is particularly critical for robot grasping and manipulation tasks, because the debugging and optimization process depends on a clear understanding of the model's decision path. In addition, end-to-end training methods face another important challenge in practical applications, that is, we cannot effectively constrain and control the manipulation posture of the final object. Since the end-to-end model learns directly from the data and generates control signals, the output of the model is difficult to accurately meet the specific needs of practical applications. This may lead to insufficient accuracy or instability in actual operations, thus affecting the reliability and effectiveness of the overall system. Summary of the invention

[0012] The technical problem to be solved by the present invention is to provide a robotic arm grasping and manipulation method that meets the high precision and high stability requirements in actual industrial applications for an end-to-end model.

[0013] The technical solution adopted by the present invention to solve the above problems is a method for accurately grasping and manipulating objects based on a graph vector database, comprising the following steps:

[0014] Color and depth RGB-D sensors at different positions collect multi-view RGB-D images of the scene; the grasping pose of the target object is estimated based on the multi-view RGB-D images; the robotic arm executes the grasping instruction on the target object according to the grasping pose estimation and enters the target object placement phase;

[0015] Target object placement stage: receiving multi-view RGB-D images and final state anchor images; the final state anchor image is the pre-collected image of the target object that is finally manipulated into place;

[0016] On the one hand, the frame data obtained by preprocessing and vector encoding of the multi-view RGB-D image and the final state anchor point features obtained by vector encoding of the final state anchor point image are output to the graph vector database; in the graph vector database, the feature vectors of the target object image in different placement states are used as graph nodes, and the three-dimensional transfer relationship between the feature vectors is used as the edge; the graph starting point determined based on the data sequence and the graph end point determined based on the final state anchor point feature are used to query the relevant state and transfer relationship in the graph vector database as the output context information; on the other hand, the data sequence composed of the frame data and the final state anchor point feature are feature extracted and fused to obtain the fused feature;

[0017] The target placement is estimated based on the context information and fusion features. The robot executes the placement instruction on the target object according to the target placement estimation and updates the placement completion status, which is then updated in the graph vector database until the target object is placed.

[0018] This paper proposes an innovative model-free grasping and manipulation scheme that does not rely on a priori accurate target object models. This scheme uses user-defined final state anchor images and combines context data in the graph vector database for reasoning, executes predictable controlled instructions in the end-to-end control of the robot arm, achieves grasping and precise manipulation of the target object, and updates and queries the graph vector database in real time during the grasping process to continuously improve the reasoning performance, overcoming the shortcomings of traditional algorithms that have poor generalization or cannot precisely manipulate objects.

[0019] Optionally, the graph nodes in the graph vector database are represented by six degrees of freedom, including the degrees of freedom of movement along the three rectangular coordinate axes of x, y, and z and the degrees of freedom of rotation around the three coordinate axes of x, y, and z.

[0020] Preferably, the target object placement stage is divided into rotation placement prediction and translation placement prediction. The rotation placement prediction is performed first. After the robot arm completes the rotation placement, the translation placement prediction is performed again. After the robot arm completes the translation placement, the target object placement is completed. This separates the prediction of rotational action and translational action, thereby greatly reducing the dimension and complexity of the vector database. For rotational placement prediction, the graph nodes in the graph vector database are represented by the rotational degrees of freedom around the three coordinate axes of x, y, and z; for translational placement prediction, the graph nodes in the graph vector database are represented by the movement degrees of freedom along the three coordinate axes of x, y, and z.

[0021] The beneficial effect of the present invention is that it does not rely on a priori accurate target object models, and exhibits excellent adaptability in complex scenes and new object processing, while also meeting the requirements for accurate manipulation and high stability of target objects in actual industrial applications. Through the present invention, the robot arm can complete various grasping tasks more flexibly and efficiently, improving production efficiency and operational reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 For example, the process of implementing the embodiment;

[0023] Figure 2 Flowchart of how the target grasp estimation module uses the fused image to obtain the grasp pose estimate;

[0024] Figure 3 It is a processing flow chart of the target object placement phase;

[0025] Figure 4 A flowchart of the target placement estimation module when performing rotation placement prediction;

[0026] Figure 5 Design a diagram for a graph database. DETAILED DESCRIPTION

[0027] The model-free grasping and manipulation algorithm implementation scheme proposed in the present invention does not rely on a priori precise target object models. It uses user-defined final state anchor vectors and combines them with context data in the graph vector database for reasoning, executes predictable controlled instructions in the end-to-end control of the robotic arm, and achieves grasping and precise manipulation of the target object.

[0028] Object grasping and manipulation algorithms such as Figure 1 As shown in the figure, the whole algorithm is divided into two parts, the target object grasping part (a) and the target object placement part (b). For the grasping process of a target object, the robot arm only needs to execute one grasping instruction to complete it, but for the placement process of the target object, the robot arm may need to execute multiple placement instructions to complete it. The details are as follows:

[0029] (1) Multiple color and depth RGB-D sensors at different locations capture multi-view RGB-D images of the scene ;

[0030] (2) Determine whether the current state of the target object is grasped. If so, proceed to step (5); if not, proceed to step (3) to trigger the target grasping estimation module;

[0031] (3) Target grasp estimation module for multi-view RGB-D images Perform fusion and reasoning to output the estimated grasping pose of the target object To the robotic arm; Represented by RGB-D image A collection of components;

[0032] (4) The robot arm estimates the grasping posture Execute the grab command on the target object and set the grab status to grab completed;

[0033] (5) Determine whether the current placement status of the target object is completed. If yes, proceed to step (8); if no, proceed to step (6) to trigger the target placement estimation module;

[0034] (6) Object placement estimation module for multi-view RGB-D images Perform fusion and reasoning to output the estimated placement position of the target object and placement status To the robot arm; placement status Used to indicate whether the target object has been placed, the initial placement status False, set the placement status after completion is True;

[0035] (7) The robot arm estimates the placement position Execute placement instructions on the target object and update the placement status And return to step (1);

[0036] (8) Reset the grabbing state and placement state of the target object and start grabbing the next target object.

[0037] Both the target grasping estimation module and the target placement estimation module are trained using machine learning.

[0038] Specifically, the processing flow of the target capture estimation module is as follows:

[0039] 1) Target grasp estimation module for multi-view RGB-D images Perform height map transformation and multi-view fusion to obtain a fused image , and then output the estimated grasping pose of the target object through inference To the robotic arm. Among them, the height map change and multi-view fusion are specifically: RGB-D images from multiple cameras All are transformed into a unified coordinate system to obtain the transformed images of each coordinate system , , is the coordinate system transformation matrix; then for the image Perform ROI cropping on the target area of ​​the image and remove irrelevant areas; finally, perform pixel alignment and fusion on the results of each perspective according to a given resolution to obtain a fused image .

[0040] 2) Target capture estimation module uses fused images Get the grasping posture estimate The specific process is as follows Figure 2 As shown:

[0041] (1) Fusion image Use rigid body transformation prediction to get the three-dimensional rigid body transformation matrix ;

[0042] (2) Using the rigid body transformation matrix For fused images Change the coordinate system to get the image , ;

[0043] (3) Use multi-layer perceptron MLP to image Perform preliminary extraction to obtain features ;

[0044] (4) Features Use feature transformation prediction to get feature transformation matrix , where the feature transformation prediction is the same as the rigid body transformation prediction;

[0045] (5) Using feature transformation coefficients Features Change the coordinate system to the feature , complete the normalization of features, ;

[0046] (6) Use MLP to analyze features Further extraction , the weight W of the grasping strategy used by the MLP when extracting features policy , which is obtained by training according to different grasping strategies during the training process; switching weights allows the grasping strategy to be changed at any time;

[0047] (7) Features Perform pooling operation to obtain global features ;

[0048] (8) Characteristics and global features Perform feature fusion to obtain fusion features ;

[0049] (9) Use the prediction head to fusion features , output grasp pose estimation ,That Including gripper position and jaw opening angle .

[0050] Preferably, in order to simplify the dimension of the estimation quantity and reduce the complexity of the subsequent vector database, the target placement estimation module divides the placement operation of the target object in the six degrees of freedom (6DoF) direction into two parts: rotational placement prediction (a) and translational placement prediction (b). Each part only needs to process the placement processing of three rotational degrees of freedom or three translational degrees of freedom. The state of each node in the graph vector database only needs three degrees of freedom data dimensions. Optionally, the target placement estimation module may also directly predict the rotational placement prediction and the translational placement prediction as a whole. The prediction method is the same, but each graph node in the graph vector database needs to be represented by six degrees of freedom, including the degrees of freedom of movement along the three rectangular coordinate axes of x, y, and z and the degrees of freedom of rotation around the three coordinate axes of x, y, and z.

[0051] The target placement estimation module includes a vector encoder, a feature extraction and fusion module, a rotation motion prediction module, a translation motion prediction module, and a graph vector database. The graph vector database is used to store the feature vectors of the target object image in different placement states and the three-dimensional transfer relationship between the feature features. The processing flow is as follows Figure 3 As shown:

[0052] (1) The target placement estimation module receives the current input multi-view RGB-D image ;

[0053] (2) Image Perform height map transformation and multi-view fusion to obtain a fused image ;

[0054] (3) In order to pass historical data to the subsequent network for better prediction, the fused image The historical image sequence is stitched together according to time to obtain a stitched image ;

[0055] (4) The target placement estimation module obtains the final state anchor image , and Output to the vector encoder; the final state anchor image is the pre-captured image of the target object that is finally manipulated into place;

[0056] (5) Image Input vector encoder, which is composed of multiple layers of convolutional encoders; vector encoder outputs the final state anchor feature To the graph vector database, the rotation placement prediction part (a) and the translation placement prediction part (b);

[0057] For the rotation placement prediction part (a):

[0058] (6) The target placement estimation module first determines whether the rotation placement is completed. If so, it proceeds to step (12), i.e., the translation placement prediction part (b). If not, it proceeds to step (7).

[0059] (7) Stitching images Each fused image in Perform dynamic ROI trimming to obtain the trimmed current frame data to be rotated , thus obtaining the trimmed data sequence to be rotated , the trimmed current frame data to be rotated The features of the current frame to be rotated are obtained by the vector encoder and then output to the image vector database. Output to the feature extraction and fusion module; Dynamic ROI trimming is trimming around the center point of the gripper;

[0060] (8) Feature extraction and fusion module of target placement estimation module for final state anchor feature Extract and fuse the features of the current frame data sequence to be rotated, and output the fused features to be rotated ;

[0061] (9) Graph vector database to receive the final state anchor feature The end point of the image, the current frame data to be rotated As the starting point of the graph, extract the relevant state and transition relationship in the vector database as the output context information to be rotated ;

[0062] (10) The rotation action prediction module receives the features to be rotated and fused and context information to be rotated Get the estimated next rotation action , and set the rotation placement state ; Rotation placement status Used to indicate whether the final state of rotation is completed; the next rotation action Output to the robot arm as a rotation placement instruction;

[0063] (11) The robot executes the rotation command and places the object according to the estimated rotation state. Update the rotation completion flag, then update the graph vector database, and finally return to step (1); the update in the graph vector database is specifically to update the state transfer to the database, build the graph path of the graph vector database according to the similarity, or insert and merge it into the current known path;

[0064] For the translation placement prediction part (b):

[0065] This part is consistent with the rotation prediction part (a), except that this part is used for the fused image. To perform static ROI pruning, the specific steps are as follows:

[0066] (12) The target placement estimation module first determines whether the translation placement is completed. If so, the placement is completed and the placement status after completion is set. If the result is True, go to step (13);

[0067] (13) Stitching images Each fused image in Perform static ROI trimming to obtain the trimmed current frame data to be translated , thus obtaining the trimmed data sequence to be translated , the trimmed current frame data to be translated After the vector encoder obtains the features of the current frame to be translated, it is output to the image vector database, and the data sequence to be translated is Output to the feature extraction and fusion module; static ROI pruning is pruning around static coordinates;

[0068] (14) Feature extraction and fusion module of target placement estimation module for final state anchor feature Extract and fuse the features of the current frame to be translated, and output the fused features to be translated ;

[0069] (15) Graph vector database to receive the final state anchor image The end point of the graph, the current frame data to be translated As the starting point of the graph, extract the relevant state and transition relationship in the vector database as the output context information to be translated ;

[0070] (16) The translation action prediction module receives the features to be translated and fused and context information to be translated Get the estimated next translation action , and set the translation placement state ; Translation placement state Used to indicate whether the final state of translation placement is completed; the next step of rotation action Output to the robot arm as a translation command;

[0071] (17) The robot executes the translation command and places the robot according to the estimated translation state. Update the translation completion flag, then update the graph vector database, and finally return to step (1).

[0072] Specifically, when the target placement estimation module performs rotation placement prediction, the specific process is as follows: Figure 4 As shown:

[0073] (1) Current frame data to be rotated and the final state anchor image After vector encoding, the features are obtained and And output to the image vector database;

[0074] (2) Characteristics As the starting point for querying the graph vector database, the feature As the end point of the graph query vector database, the graph vector database outputs the context information to be rotated ;

[0075] (3) Using the features to be rotated and context information to be rotated Use the data query function Query to construct the query output to the multimodal basic model;

[0076] (4) The multimodal basic model outputs the final inference result and rotated placement state . Reasoning results It is the currently predicted rotation action that the robot arm will perform next.

[0077] The target placement estimation module performs the same process as above when performing translation placement prediction, which will not be described here.

[0078] Specifically, the structure design of the graph vector database is as follows: Figure 5 As shown:

[0079] Taking rotation action prediction as an example, the graph vector database is designed to store the feature vectors of the target object image in different rotation states and the three-dimensional transfer relationship between these feature vectors. The feature vectors in different rotation states are graph nodes, and the three-dimensional transfer relationship is the edge. Figure 5 middle, For the Line The feature vector corresponding to the node of the column is the feature of the object image in a certain rotation state. From the rotating state To the rotating state The following transfer relationship. ,in stands for Special Orthogonal Group, where each transformation matrix Represents a rotation in three-dimensional space. The graph vector database can query and output related states and transition relationships according to the similarity of the input start point, end point and point index. The features in the database are updated in real time after each action performed by the robot arm. and status .

[0080] Specifically, the graph vector database query method is:

[0081] When the characteristics As the starting point for querying the graph vector database, the feature As the graph endpoint for querying the graph vector database, the query steps for the graph vector database are as follows:

[0082] 1) Use For each feature vector in the graph vector database Calculate cosine similarity :

[0083]

[0084] Represents the modulus of a vector, if If it is greater than a given threshold, then the To match the graph nodes, determine the is the graph starting point vector; if there is no matching graph node, when updating the graph vector database, As a new graph node;

[0085] 2) Use the same method For each vector in the vector database Calculate the cosine similarity if If it is greater than a given threshold, then the To match the graph nodes, determine the current is the graph endpoint vector; if there is no matching graph node, when updating the graph vector database, As a new graph node;

[0086] 3) Match all paths in the graph according to the determined graph start vector and graph end vector;

[0087] 4) Sort by the rotation distance of the path passed and return to the previous records, is the default value.

[0088] Specifically, the initialization and update method of the graph vector database is:

[0089] There are two ways to initialize:

[0090] 1) Keep the database empty

[0091] 2) Pre-populate the database using an update mechanism

[0092] The database is updated as follows:

[0093] 1) Record the starting point vector of the initial graph ;

[0094] 2) Wait for the robot to execute Finished, the starting point of the graph changes;

[0095] 3) The sensor captures new image data;

[0096] 4) Record the starting vector of the subsequent new graph , which is the end point vector of the previous graph;

[0097] 5) According to and Database vectors are matched and new graph paths are created or inserted and merged into existing known paths based on similarity.

[0098] The embodiment uses a user-defined final state anchor vector and combines it with contextual data in the graph vector database for reasoning, so that the solution can perform accurate end-to-end object manipulation. The vector database is continuously updated to improve the accuracy of model reasoning. Accurate manipulation of objects is achieved while ensuring generalization and model-free end-to-end reasoning.

Claims

1. A method for accurate object grasping and manipulation based on a graph vector database, characterized in that: Includes steps: Color and depth RGB-D sensors at different positions collect multi-view RGB-D images of the scene; the grasping pose of the target object is estimated based on the multi-view RGB-D images; the robotic arm executes the grasping instruction on the target object according to the grasping pose estimation and enters the target object placement phase; Target object placement stage: receiving multi-view RGB-D images and final state anchor images; the final state anchor image is the pre-collected image of the target object that is finally manipulated into place; On the one hand, the frame data obtained by preprocessing and vector encoding of the multi-view RGB-D image and the final state anchor point features obtained by vector encoding of the final state anchor point image are output to the graph vector database; in the graph vector database, the feature vectors of the target object image in different placement states are used as graph nodes, and the three-dimensional transfer relationship between the feature vectors is used as the edge; the graph starting point determined based on the data sequence and the graph end point determined based on the final state anchor point feature are used to query the relevant state and transfer relationship in the graph vector database as the output context information; on the other hand, the data sequence composed of the frame data and the final state anchor point feature are feature extracted and fused to obtain the fused feature; The target placement is estimated based on the context information and fusion features. The robot executes the placement instruction on the target object according to the target placement estimation and updates the placement completion status, which is then updated in the graph vector database until the target object is placed.

2. The method according to claim 1, characterized in that: The graph nodes in the graph vector database are represented by six degrees of freedom, including the freedom of movement along the three rectangular coordinate axes of x, y, and z and the freedom of rotation around the three coordinate axes of x, y, and z.

3. The method according to claim 1, characterized in that: When determining the starting point and the end point of the graph, the cosine similarity between the data sequence and the final state anchor point features and each graph node in the graph vector database is calculated to find the matching graph nodes as the starting point and the end point of the graph.

4. The method according to claim 2 or 3, characterized in that: The preprocessing of the perspective RGB-D images includes height map transformation and multi-view fusion as well as the splicing of historical image sequences in time.

5. The method according to claim 4, characterized in that: The target object placement stage is divided into rotation placement prediction and translation placement prediction. Rotation placement prediction is performed first. After the robot arm completes rotation placement, translation placement prediction is performed. After the robot arm completes translation placement, the target object placement is completed.

6. The method according to claim 5, characterized in that For rotational placement prediction, the graph nodes in the graph vector database are represented by the rotational degrees of freedom around the three coordinate axes x, y, and z; For translation placement prediction, graph nodes in the graph vector database are represented by the degrees of freedom of movement along the three coordinate axes: x, y, and z.

7. The method according to claim 6, characterized in that For rotation placement prediction, the preprocessing of the perspective RGB-D image includes dynamic image target area ROI pruning after the splicing of the historical image sequence according to time. Dynamic ROI pruning is performed around the center point of the gripper of the robot arm. For translational placement prediction, the preprocessing of the viewpoint RGB-D image includes static ROI pruning after the splicing of the historical image sequence in time; static ROI pruning is pruning around static coordinates.

8. The method according to claim 7, characterized in that: For rotation placement prediction, the multi-view RGB-D image is preprocessed to obtain the current frame data to be rotated and the data sequence to be rotated; On the one hand, the current frame data to be rotated obtained by preprocessing and vector encoding of the multi-view RGB-D image and the final state anchor point features obtained by vector encoding of the final state anchor point image are output to the graph vector database; the graph starting point determined based on the current frame data to be rotated and the graph end point determined based on the final state anchor point features are used to query the relevant states and transition relationships in the graph vector database as the output context information to be rotated; on the other hand, the data sequence to be rotated and the final state anchor point features are feature extracted and fused to obtain the fused features to be rotated; For translation placement prediction, the multi-view RGB-D images are preprocessed to obtain the current frame data to be translated and the sequence of data to be translated; On the one hand, the current frame data to be translated obtained by preprocessing and vector encoding of the multi-view RGB-D image and the final state anchor point features obtained by vector encoding of the final state anchor point image are output to the graph vector database; the graph starting point determined based on the current frame data to be translated and the graph end point determined based on the final state anchor point features are used to query the graph vector database for relevant states and transfer relationships as the output context information to be translated; on the other hand, the data sequence to be translated and the final state anchor point features are feature extracted and fused to obtain the fused features to be translated.

9. The method according to claim 8, characterized in that For rotation placement prediction, the target placement is estimated based on the context information to be rotated and the fused features to be rotated to obtain the next rotation action and the rotation placement state. The robot executes the rotation placement instruction on the target object and updates the rotation completion flag, which is then updated in the graph vector database until the target object completes the rotation placement of the final state, and then the translation placement prediction is performed; For translation placement prediction, the target placement is estimated based on the context information to be translated and the fusion features to be translated to obtain the next translation action and the translation placement state. The robotic arm executes the translation placement instruction on the target object and updates the translation completion flag, and then updates it in the graph vector database until the target object completes the translation placement of the final state and the target object is placed.

Images