An accurate object grasping and manipulation method based on a graph vector database

Abstract

A precise object grasping and manipulation method based on a graph vector database. It acquires multi-view RGB-D images of the scene from color and depth RGB-D sensors at different positions, and after estimating the grasping pose of the target object for the multi-view RGB-D images, it enters the target object placement stage. In the target object placement stage, it receives the multi-view RGB-D images and the final state anchor images and combines the context data in the graph vector database for reasoning so that the solution can perform precise end-to-end object manipulation. Before reasoning, it queries the graph vector database using the current image and the final state anchor vector to obtain the context information related to the current manipulation, improving the accuracy of model reasoning. Further, the pose estimation of the target object is separated into rotation estimation and translation estimation, reducing the algorithm complexity and the state dimension of the graph database. The present invention does not rely on a prior precise target object model and meets the requirements of precise manipulation and high stability of the target object in practical industrial applications.

Description

Technical Field

[0001] The present invention relates to image processing technology, and more particularly to an accurate object grasping and manipulation technology based on a graph vector database. Background Art

[0002] Object grasping and manipulation algorithms play a crucial role in the current technological development. First of all, these algorithms need to accurately estimate the position and pose of an object in three-dimensional space. This process includes transforming the object from the world coordinate system to the camera coordinate system and estimating and tracking the six degrees of freedom (6DoF) of the object. These six degrees of freedom include the translational degrees of freedom along the three rectangular coordinate axes x, y, and z, and the rotational degrees of freedom around these three axes, namely the translational movements of forward and backward (x-axis), up and down (y-axis), left and right (z-axis), and the rotational movements of pitch, yaw, and roll.

[0003] Once the 6DoF pose of the object is obtained, the algorithm will select the optimal grasping point according to the grasping strategy and finally achieve the accurate manipulation of the target object. These algorithms have a wide range of applications in modern industrial production, incoming material assembly, unordered sorting and grasping, etc. For example, in the field of unordered sorting and grasping, vision-based grasping and control algorithms can help robots grasp unordered objects in a bin more efficiently, thus significantly improving the grasping success rate. In the field of robot vision, these vision-based grasping and control algorithms can help robots better perceive the environment, thereby enhancing their autonomy and intelligence level. In the field of motion control, these algorithms can help robots accurately control their motion trajectories, thus improving the motion accuracy and stability.

[0004] Generally speaking, object grasping and manipulation algorithms not only promote the development of robot technology, but also show great potential and value in multiple industrial application scenarios. The continuous optimization and development of these algorithms will undoubtedly further improve the efficiency, accuracy, and intelligence level of the robot system.

[0005] Existing object grasping and manipulation methods include methods based on accurate object models, methods based on generated object models, and end-to-end object manipulation methods.

[0006] 1. Method based on an accurate object model: This method is a classic approach in traditional object grasping and manipulation. It relies on accurately modeling the detailed geometric and physical characteristics of the target object. One implementation is to use these models to fully sample the target object in the possible SE(3) space through rendering techniques, extract a sufficiently robust template from it, roughly estimate the pose of the object by matching the template, and finally refine the result through the Iterative Closest Point (ICP) algorithm to obtain the accurate pose of the object. Next, the best grasping points are selected based on the estimated pose, and finally, the grasping and manipulation of the object are achieved. Additionally, the accurate model of the target object can be utilized in combination with deep learning techniques to train the target pose estimation algorithm. The robot can then use the trained model to accurately estimate the 6-degree-of-freedom (6DoF) pose of the object in the scene, thus completing the grasping and manipulation of the object.

[0007] This type of method has high requirements for prior information. It depends on accurately modeling the detailed geometric and physical characteristics of the target object. This means that a large amount of object model information needs to be obtained and stored in advance, which is not only time-consuming and laborious but also has high requirements for the integrity and accuracy of the data. Secondly, this type of method has poor environmental adaptability. Since it depends on pre-stored accurate models, it has poor adaptability when facing environmental changes or object deformations. When the shape or position of the object changes, it is necessary to re-obtain and model, which reduces its application flexibility in dynamic and complex environments. Finally, its ability to handle irregular or unknown objects is also very limited. The method based on accurate models is mainly applicable to known and regular objects. For objects with complex shapes or lacking prior models, the accuracy and stability of their grasping and manipulation may be restricted.

[0008] 2. Method based on a generative object model: The method based on a generative object model simplifies the representation of the object by using generative models. These generative models generate approximate object models based on partially observed data, thus reducing the dependence on accurate geometric data. For example, point cloud data or depth maps can be used to generate a three-dimensional model of the object. This method shows greater flexibility and adaptability when dealing with new objects and changing environments. After obtaining the generative model of the object, these models are used to estimate the six-degree-of-freedom (6DoF) pose of the object. By comparing with the actual observed data, the pose estimation result of the object is corrected and optimized. Based on the generated object model and the pose estimation result, the optimal grasping points are selected, and the grasping action is executed according to the calculated grasping points and grasping paths. Finally, the robot can show a high degree of adaptability in a changing environment while achieving accurate grasping and manipulation of the target object.

[0009] The disadvantages of such methods are as follows. Firstly, the accuracy of the generated model may be relatively low. The method of generating an object model relies on partially observed data to generate an approximate three-dimensional model of the object, which means that the generated model may not be able to precisely describe the geometric details of the object completely. This can lead to errors during the grasping and manipulation process, thus affecting the success rate and accuracy of the operation. Secondly, the computational complexity of this type of method is relatively high. The process of generating an object model usually involves complex calculations, especially when real-time generation and optimization of the model are required. This places high demands on the system's computational resources and response time, and may not perform well in high-real-time applications. Finally, the stability of the generated model is insufficient. When dealing with unknown or complex objects, the generated model may produce unstable results. For example, for some objects with complex shapes or partially occluded parts, the accuracy and stability of their generated models may be poor, affecting the subsequent grasping and manipulation effects.

[0010] 3. End-to-end object manipulation method: The end-to-end object manipulation method utilizes advanced technologies such as deep learning and reinforcement learning to achieve object grasping and manipulation through direct mapping from sensor data (such as images, depth maps) to control signals. The advantage of this method lies in its high adaptability and ability to handle complex environments. Since there is no need for an intermediate object modeling process, the end-to-end method can directly learn the object's features and grasping strategies and adapt to various different objects and scenarios.

[0011] Compared with the first and second types of methods, although the end-to-end object manipulation method demonstrates significant advantages in terms of adaptability and handling complex environments, there are also some problems that cannot be ignored. The most prominent problems are the predictability of the model, as well as the precision and stability of object manipulation. Deep learning models are usually regarded as "black boxes", and their internal decision-making processes are difficult to explain and understand. This is particularly crucial for robot grasping and manipulation tasks because the debugging and optimization processes rely on a clear understanding of the model's decision-making path. In addition, the end-to-end training method faces another important challenge in practical applications, that is, we cannot effectively constrain and control the final manipulation posture of the object. Since the end-to-end model directly learns from the data and generates control signals, it is difficult for the model's output to precisely meet the specific requirements in practical applications. This may lead to situations of insufficient precision or instability in actual operations, thus affecting the reliability and effectiveness of the overall system. Summary of the Invention

[0012] The technical problem to be solved by the present invention is to provide a robotic arm grasping and manipulation method that meets the requirements of high precision and high stability in actual industrial applications for end-to-end models.

[0013] The technical solution adopted by the present invention to solve the above problems is an accurate object grasping and manipulation method based on a graph vector database, including the following steps:

[0014] RGB-D sensors of different positions collect multi-view RGB-D images of the scene; perform grasping pose estimation of the target object on the multi-view RGB-D images; the robotic arm executes the grasping instruction on the target object according to the grasping pose estimation and enters the target object placement stage;

[0015] Target object placement stage: Receive multi-view RGB-D images and the final state anchor image; the final state anchor image is an image of the target object finally manipulated in place collected in advance.

[0016] On the one hand, output the frame data obtained by preprocessing and vector encoding the multi-view RGB-D images and the final state anchor features obtained by vector encoding the final state anchor images to the graph vector database; in the graph vector database, the feature vectors of the target object images in different placement states are used as graph nodes, and the three-dimensional transfer relationship between the feature vectors is used as edges; query the relevant states and transfer relationships in the graph vector database with the graph start point determined based on the data sequence and the graph end point determined based on the final state anchor features as the output context information; on the other hand, perform feature extraction and fusion on the data sequence composed of the frame data and the final state anchor features to obtain the fusion features.

[0017] Perform target placement estimation according to the context information and the fusion features. The robotic arm executes the placement instruction on the target object according to the target placement estimation and updates the placement completion state, and then updates it in the graph vector database until the target object is placed.

[0018] The present invention proposes an innovative model-free grasping and manipulation scheme that does not rely on a priori precise target object models. This scheme performs inference by using the user-defined final state anchor image and combining the context data in the graph vector database, executes predictable controlled instructions in the end-to-end control of the robotic arm, realizes the grasping and precise manipulation of the target object, and continuously updates and queries the graph vector database during the grasping process to continuously improve the inference performance, overcoming the disadvantages of poor generalization or inability to precisely manipulate objects of traditional algorithms.

[0019] Optionally, the graph nodes in the graph vector database are represented by six degrees of freedom, including the translational degrees of freedom along the three rectangular coordinate axes of x, y, and z and the rotational degrees of freedom around the three coordinate axes of x, y, and z.

[0020] Preferably, the target object placement stage is divided into rotational placement prediction and translational placement prediction. First, the rotational placement prediction is performed. After the robotic arm completes the rotational placement, the translational placement prediction is then carried out. After the robotic arm completes the translational placement, the placement of the target object is completed. This separates the prediction of rotational and translational movements, thereby greatly reducing the dimension and complexity of the vector database. For rotational placement prediction, the graph nodes in the graph vector database are represented by the rotational degrees of freedom around the x, y, and z axes; for translational placement prediction, the graph nodes in the graph vector database are represented by the translational degrees of freedom along the x, y, and z axes.

[0021] The beneficial effects of the present invention are that it does not rely on a prior precise target object model, demonstrates excellent adaptability in complex scenarios and the handling of new objects, and also meets the requirements for precise manipulation and high stability of target objects in practical industrial applications. Through the present invention, the robotic arm can more flexibly and efficiently complete various grasping tasks, improving production efficiency and operation reliability. Description of the Drawings

[0022] Figure 1 For the embodiment process;

[0023] Figure 2 For the flowchart of the target grasping estimation module using the fused image to obtain the grasping pose estimation;

[0024] Figure 3 For the processing flowchart of the target object placement stage;

[0025] Figure 4 For the flowchart when the target placement estimation module performs rotational placement prediction;

[0026] Figure 5 For the schematic diagram of the graph database design. Detailed Embodiment

[0027] The implementation scheme of the model-free grasping and manipulation algorithm proposed by the present invention does not rely on a prior precise target object model. By using the user-defined final state anchor vector and combining the context data in the graph vector database for reasoning, predictable controlled instructions are executed in the end-to-end control of the robotic arm to achieve the grasping and precise manipulation of the target object.

[0028] The object grasping and manipulation algorithm is as Figure 1 shown. The entire algorithm is divided into two parts, the target object grasping part a and the target object placement part b. For the grasping process of a target object, the robotic arm only needs to execute one grasping instruction to complete it. However, for the placement process of the target object, the robotic arm may need to execute multiple placement instructions to complete it. Specifically as follows:

[0029] (1) Multiple color and depth RGB-D sensors at different positions collect multi-view RGB-D images of the scene ;

[0030] (2) Determine whether the grasping of the current target object is completed. If so, go to step (5). If not, go to step (3) to trigger the target grasping estimation module;

[0031] (3) The target grasping estimation module fuses and infers the multi-view RGB-D images and outputs the grasping pose estimation of the target object to the robotic arm; Denote the set composed of RGB-D images ;

[0032] (4) The robotic arm executes the grasping instruction on the target object according to the grasping pose estimation and sets the grasping state to grasping completed;

[0033] (5) Determine whether the placement state of the current target object is placement completed. If so, go to step (8). If not, go to step (6) to trigger the target placement estimation module;

[0034] (6) The target placement estimation module fuses and infers the multi-view RGB-D images and outputs the placement position estimation and the placement state to the robotic arm; The placement state is used to indicate whether the target object is placed completed. The initially set placement state is False, and the placement state after setting is True;

[0035] (7) The robotic arm executes the placement instruction on the target object according to the placement position estimation updates the placement state and returns to step (1);

[0036] (8) Reset the grasping state and placement state of the target object and start grasping the next target object.

[0037] Both the target grasping estimation module and the target placement estimation module are trained in a machine learning manner.

[0038] Specifically, the processing flow of the target grasping estimation module is as follows:

[0039] 1) The target grasping estimation module performs height map transformation and multi-view fusion on the multi-view RGB-D images to obtain a fused image , and then estimate the grasping pose of the target object through reasoning and output it to the robotic arm. Among them, the specific processes of height map change and multi-view fusion are as follows: Transform the RGB-D images under multiple cameras into a unified coordinate system to obtain the images after coordinate system transformation . , , is the coordinate system transformation matrix; then crop the ROI of the target area of the image to remove irrelevant areas; finally, align and fuse the results of each view according to the given resolution to obtain the fused image .

[0040] 2) The target grasping estimation module uses the fused image to obtain the grasping pose estimation , and the specific process is as Figure 2 shown:

[0041] (1) Use rigid body transformation prediction on the fused image to obtain the three-dimensional rigid body transformation matrix ;

[0042] (2) Use the rigid body transformation matrix to perform coordinate system transformation on the fused image to obtain the image , ;

[0043] (3) Use a multi-layer perceptron MLP to perform preliminary extraction on the image to obtain the feature ;

[0044] (4) Use feature transformation prediction on the feature to obtain the feature transformation matrix , where the feature transformation prediction is the same as the rigid body transformation prediction;

[0045] (5) Use the feature transformation coefficient to perform coordinate system transformation on the feature to the feature to complete the normalization of the feature, ;

[0046] (6) Use MLP to further extract the feature to obtain , and the grasping strategy weight W policy used by this MLP during feature extraction is trained according to different grasping strategies during the training process; switching the weight enables the grasping strategy to be changed at any time;

[0047] (7) For the feature Perform pooling operation to obtain global features ;

[0048] (8) Fuse the features and the global features to obtain fused features ;

[0049] (9) Use the prediction head to output the grasping pose estimation based on the fused features , which includes the gripper pose and the gripper opening angle .

[0050] Preferably, in order to simplify the dimension of the estimation quantity and reduce the complexity of the subsequent vector database, the target placement estimation module divides the placement operation of the six degrees of freedom (6DoF) direction of the target object into a rotational placement prediction part a and a translational placement prediction part b. Each part only needs to process the placement of three rotational degrees of freedom or three translational degrees of freedom. The state of each node in the graph vector database only requires three degrees of freedom data dimension. Optionally, the target placement estimation module may also directly predict the rotational placement prediction and the translational placement prediction as a whole. The prediction method is the same, but each graph node in the graph vector database needs to be represented by six degrees of freedom, including the translational degrees of freedom along the three right-angle coordinate axes x, y, z and the rotational degrees of freedom around the three coordinate axes x, y, z.

[0051] The target placement estimation module includes a vector encoder, a feature extraction and fusion module, a rotational action prediction module, a translational action prediction module, and a graph vector database. The graph vector database is used to store the feature vectors of the target object image in different placement states and the three-dimensional transfer relationship between the features. The processing flow is as Figure 3 shown:

[0052] (1) The target placement estimation module receives the currently input multi-view RGB-D image ;

[0053] (2) Perform height map transformation and multi-view fusion on the image to obtain a fused image ;

[0054] (3) In order to pass the historical data to the subsequent network for better prediction, splice the fused image by time to obtain a spliced image ;

[0055] (4) The target placement estimation module obtains the final state anchor image , and will Output to the vector encoder; the final state anchor image is a pre - acquired image of the target object finally manipulated into place.

[0056] (5) Input the image into the vector encoder, which consists of multiple layers of convolutional encoders; the vector encoder outputs the final state anchor features to the graph vector database, the rotational placement prediction part a, and the translational placement prediction part b.

[0057] For the rotational placement prediction part a:

[0058] (6) The target placement estimation module first determines whether the rotational placement is completed. If so, it enters step (12), that is, the translational placement prediction part b. If not, it enters step (7).

[0059] (7) For each fused image in the spliced image perform dynamic ROI pruning to obtain the pruned current frame data to be rotated , thus obtaining the pruned data sequence to be rotated . Output the pruned current frame data to be rotated to the graph vector database after obtaining the current frame features to be rotated through the vector encoder, and output the data sequence of frames to be rotated to the feature extraction and fusion module; the dynamic ROI pruning is performed around the center point of the gripper.

[0060] (8) The feature extraction and fusion module of the target placement estimation module performs feature extraction and fusion on the final state anchor features and the current data sequence of frames to be rotated, and outputs the fused features to be rotated ;

[0061] (9) The graph vector database takes the received final state anchor features as the graph end point, and the current frame data to be rotated as the graph start point, and extracts the relevant states and transition relationships in the vector database as the output context information to be rotated ;

[0062] (10) The rotational motion prediction module receives the fused features to be rotated and the context information to be rotated to obtain the estimated next rotational motion , and sets the rotational placement state ; the rotational placement state is used to indicate whether the rotational placement of the final state is completed; output the next rotational motion as the rotational placement instruction to the robotic arm.

[0063] The robotic arm executes a rotation instruction and updates the rotation completion flag according to the estimated rotation placement state and then updates it in the graph vector database, and finally returns to step (1); the update in the graph vector database is specifically to update the state transition to the database and construct or insert and merge into the current known path in the graph vector database according to the similarity;

[0064] For the translational placement prediction part b:

[0065] This part has the same process as the rotation prediction part a. The difference is that this part performs static ROI pruning on the fused image as follows:

[0066] (12) The target placement estimation module first determines whether the translational placement is completed. If so, the placement is completed and the placement state after completion is set to True. If not, it enters step (13);

[0067] (13) Perform static ROI pruning on each fused image in the stitched image to obtain the current translational frame data after pruning , thus obtaining the pruned translational data sequence . Output the current translational frame data after pruning to the graph vector database after obtaining the current translational frame features through the vector encoder, and output the translational data sequence to the feature extraction and fusion module; the static ROI pruning is performed around the static coordinates;

[0068] (14) The feature extraction and fusion module of the target placement estimation module performs feature extraction and fusion on the final state anchor feature and the current translational frame feature, and outputs the translational fusion feature ;

[0069] (15) The graph vector database takes the received final state anchor image as the graph end point and the current translational frame data as the graph start point, and extracts the relevant states and transition relationships in the vector database as the output translational context information ;

[0070] (16) The translational motion prediction module receives the translational fusion feature and the translational context information to obtain the estimated next translational motion , and sets the translational placement state ; Translation placement state Used to indicate whether the final state of translation placement is completed; the next rotation action Is output as a translation instruction to the robotic arm;

[0071] (17) The robotic arm executes the translation instruction and updates the translation completion flag according to the estimated translation placement state And updates it in the graph vector database, and finally returns to step (1).

[0072] Specifically, when the target placement estimation module performs rotation placement prediction, the specific process is as Figure 4 Shown:

[0073] (1) The current frame data to be rotated And the final state anchor image After vector encoding, features are obtained respectively And And output to the graph vector database;

[0074] (2) Feature Is used as the starting point of the graph for querying the graph vector database, and feature Is used as the ending point of the graph for querying the graph vector database, and the graph vector database outputs the context information to be rotated ;

[0075] (3) Using the context information to be rotated and the feature to be rotated And the context information to be rotated Use the data query function Query to construct a query and output it to the multi-modal basic model;

[0076] (4) The multi-modal basic model outputs the final inference result And the rotation placement state . The inference result Is the rotation action that the robotic arm will perform next for the current prediction.

[0077] The process of the target placement estimation module for translation placement prediction is the same as the above process and will not be elaborated here.

[0078] Specifically, the structure design of the graph vector database is as Figure 5 Shown:

[0079] Taking the rotation action prediction as an example, the graph vector database is designed to store the feature vectors of the target object image in different rotation states and the three-dimensional transfer relationship of these feature vectors. The feature vectors in different rotation states are the graph nodes, and the three-dimensional transfer relationship is the edge. Figure 5 In, Is the th row and the The eigenvector corresponding to the node in the column is the feature of the object image in a certain rotation state. is the transition relationship from the rotation state to the rotation state below. , where represents the Special Orthogonal Group, and each transformation matrix represents a rotation in three-dimensional space. The graph vector database can query and output relevant states and transition relationships according to the input start and end indices according to similarity. The features and states in the database are updated in real time after each action of the robotic arm.

[0080] Specifically, the method for querying the graph vector database is as follows:

[0081] When the feature is used as the graph start point for querying the graph vector database, and the feature is used as the graph end point for querying the graph vector database, the query steps for the graph vector database are as follows:

[0082] 1) Use to calculate the cosine similarity for each eigenvector in the graph vector database:

[0083]

[0084] represents taking the modulus of the vector. If is greater than a given threshold, then this is considered a matching graph node, and the current in the graph vector database is determined as the graph start point vector; if there is no matching graph node, then when updating the graph vector database, is used as the new graph node;

[0085] 2) Similarly, use to calculate the cosine similarity for each vector in the vector database. If is greater than a certain given threshold, then this is considered a matching graph node, and the current is determined as the graph end point vector; if there is no matching graph node, then when updating the graph vector database, is used as the new graph node;

[0086] 3) Match all paths in the graph according to the determined graph start point vector and graph end point vector;

[0087] 4) Sort according to the rotation distance of the passed path and return the top records, which is a preset value.

[0088] Specifically, the initialization and update methods of the graph vector database are as follows:

[0089] There are two ways to initialize:

[0090] 1) Keep the database empty

[0091] 2) Pre-fill the database using the update mechanism

[0092] The update of the database is as follows:

[0093] 1) Record the initial graph starting vector ;

[0094] 2) Wait for the robotic arm to execute completely, and the graph starting point changes;

[0095] 3) The sensor captures new image data;

[0096] 4) Record the subsequent new graph starting vector , that is, the previous graph ending vector;

[0097] 5) According to and match the database vectors, and create a new graph path or insert and merge it into the current known path according to the similarity.

[0098] The embodiment enables the scheme to perform precise end-to-end object manipulation by using the user-defined final state anchor vector and combining the context data in the graph vector database. Continuously update the vector database to improve the model inference accuracy. Achieve precise manipulation of objects while ensuring generalization and model-free end-to-end inference.

Claims

1. An accurate object grasping and manipulation method based on a graph vector database, characterized in that, Including the steps: Color and depth RGB-D sensors at different positions collect multi-view RGB-D images of the scene; perform grasping pose estimation of the target object on the multi-view RGB-D images; the robotic arm executes the grasping instruction on the target object according to the grasping pose estimation and enters the target object placement stage; Target object placement stage: Receive the multi-view RGB-D images and the final state anchor images; the final state anchor images are the images of the target object finally manipulated in place collected in advance; On the one hand, output the frame data obtained by preprocessing and vector encoding the multi-view RGB-D images and the final state anchor features obtained by vector encoding the final state anchor images to the graph vector database; in the graph vector database, the feature vectors of the target object images in different placement states are used as graph nodes, and the three-dimensional transfer relationships between the feature vectors are used as edges; query the relevant states and transfer relationships in the graph vector database with the graph start point determined based on the data sequence and the graph end point determined based on the final state anchor features as the output context information; on the other hand, perform feature extraction and fusion on the data sequence composed of the frame data and the final state anchor features to obtain the fusion features; Perform target placement estimation according to the context information and the fusion features, and the robotic arm executes the placement instruction on the target object according to the target placement estimation and updates the placement completion state, and then updates in the graph vector database until the target object is placed; Among them, the graph nodes in the graph vector database are represented by six degrees of freedom, including the translational degrees of freedom along the three rectangular coordinate axes of x, y, and z and the rotational degrees of freedom around the three coordinate axes of x, y, and z; When determining the graph start point and the graph end point, find the matching graph nodes as the graph start point and the graph end point by calculating the cosine similarity between the data sequence and the final state anchor features and each graph node in the graph vector database.

2. The method according to claim 1, characterized in that The preprocessing of the perspective RGB-D images includes height map transformation, multi-view fusion, and stitching of the historical image sequences in time.

3. The method according to claim 2, wherein The target object placement stage is divided into rotational placement prediction and translational placement prediction. First, perform rotational placement prediction. After the robotic arm completes the rotational placement, then perform translational placement prediction. After the robotic arm completes the translational placement, the target object placement is completed.

4. The method according to claim 3, wherein For rotational placement prediction, the graph nodes in the graph vector database are represented by the rotational degrees of freedom around the three coordinate axes of x, y, and z; For translational placement prediction, the graph nodes in the graph vector database are represented by the translational degrees of freedom along the three coordinate axes of x, y, and z.

5. The method according to claim 4, wherein For rotational placement prediction, after the stitching of the historical image sequences in time in the preprocessing of the perspective RGB-D images, it also includes dynamic image target region ROI pruning; the dynamic ROI pruning is performed around the center point of the gripper of the robotic arm; For translational placement prediction, after the stitching of the historical image sequences in time in the preprocessing of the perspective RGB-D images, it also includes static ROI pruning; the static ROI pruning is performed around the static coordinates.

6. The method according to claim 5, wherein For rotational placement prediction, the multi-view RGB-D images are preprocessed to obtain the current frame data to be rotated and the data sequence to be rotated; On the one hand, the current frame data to be rotated obtained by preprocessing and vector encoding the multi-view RGB-D image and the final state anchor feature obtained by vector encoding the final state anchor image are output to the graph vector database; the relevant states and transition relationships are queried in the graph vector database with the graph start point determined based on the current frame data to be rotated and the graph end point determined based on the final state anchor feature as the context information to be rotated for output; on the other hand, the data sequence to be rotated and the final state anchor feature are subjected to feature extraction and fusion to obtain the fused feature to be rotated. For the prediction of translational placement, the multi-view RGB-D image is preprocessed to obtain the current frame data to be translated and the data sequence to be translated. On the one hand, the current frame data to be translated obtained by preprocessing and vector encoding the multi-view RGB-D image and the final state anchor feature obtained by vector encoding the final state anchor image are output to the graph vector database; the relevant states and transition relationships are queried in the graph vector database with the graph start point determined based on the current frame data to be translated and the graph end point determined based on the final state anchor feature as the context information to be translated for output; on the other hand, the data sequence to be translated and the final state anchor feature are subjected to feature extraction and fusion to obtain the fused feature to be translated.

7. The method according to claim 6, wherein For the prediction of rotational placement, the next rotational action is obtained by estimating the target placement according to the context information to be rotated and the fused feature to be rotated, and the rotational placement state is set. The robotic arm executes the rotational placement instruction on the target object and updates the rotation completion flag, and then updates in the graph vector database until the target object completes the rotational placement in the final state, and then the translational placement prediction is performed. For the prediction of translational placement, the next translational action is obtained by estimating the target placement according to the context information to be translated and the fused feature to be translated, and the translational placement state is set. The robotic arm executes the translational placement instruction on the target object and updates the translation completion flag, and then updates in the graph vector database until the target object completes the translational placement in the final state, and the placement of the target object is completed.

Images