Deformable object double-arm operation method based on large language model

Abstract

The invention discloses a deformable object double-arm operation method based on a large language model. According to the method, a language target instruction of a user is used as input. Compared with visual target information, the language target instruction can contain richer semantic information, more comprehensive context support is provided, and flexible and diversified task expressions can be supported. Besides, the input mode of language instructions is closer to natural interaction habits of human beings, the operation threshold is remarkably reduced, the compactness and efficiency of man-machine cooperation are improved, an innovative three-layer hierarchical planning method is further provided, sub-target decomposition, folding symmetry axis planning and two-arm grabbing and placing action planning are included, and the method is suitable for being popularized and applied. The operation task of a complex deformable object can be effectively solved, and hierarchical planning from high-level task decomposition to specific operation actions is realized. Meanwhile, multi-step operation actions can be planned only based on a final target, and good performance and adaptability are shown in a complex task scene.

Description

Technical Field

[0001] The present invention relates to the field of robot operation, and in particular to a dual-arm operation method for a deformable object based on a large language model. Background Art

[0002] In modern industrial production and daily life, there is an increasing demand for robots to operate deformable objects. Especially in the home environment, auxiliary robots can complete tasks such as grabbing, folding and sorting clothes according to human instructions, providing convenient intelligent services for family life; while in the industrial environment, deformable object operation technology is widely used in complex scenarios such as packaging, rope knotting and flexible material assembly. These applications place high demands on the accuracy of robot perception and motion planning. Especially when dealing with deformable objects, the robot needs to design reasonable perception and planning algorithms to effectively perform dimensionality reduction observations in the infinite-dimensional state space and calculate accurate robot arm operation action sequences.

[0003] Most existing methods for manipulating deformable objects rely on visual goal guidance. Using a set of visual sub-goal sequences as reference, the robot uses deep learning methods to calculate the action sequence required to achieve the target state. However, existing methods have the following problems:

[0004] 1) Lack of multi-step planning capabilities: Existing methods usually rely on precise sub-goals to complete single-step operation calculations and lack the ability to calculate multi-step operation actions from the final goal, resulting in inefficient execution of complex tasks.

[0005] 2) Insufficient generalization ability: Although existing methods perform well in known training tasks, they lack sufficient generalization ability when faced with new tasks and are difficult to adapt to diverse operational requirements;

[0006] 3) Reliance on expert teaching data: Existing methods need to rely on expert teaching data for imitation learning training, and the collection process of expert data is time-consuming and labor-intensive, which limits the scalability and practicality of the method. Summary of the invention

[0007] The present invention aims at the problems existing in the prior art and provides a dual-arm operation method for a deformable object based on a large language model. The specific technical solution adopted by the present invention is as follows:

[0008] The present invention discloses a dual-arm operation method for a deformable object based on a large language model, comprising:

[0009] Obtain the user's language target instruction;

[0010] Parse the language target instructions through a large language model and decompose them into a series of language sub-targets;

[0011] For each language subgoal, the loop iterates and performs the following steps until all subgoals are completed:

[0012] Get the current deformable object image;

[0013] Fitting parameterized polygonal models based on deformable object images;

[0014] Based on the language sub-goals and the parameterized polygonal model, the folding symmetry axis is calculated through the large language model;

[0015] According to the folding symmetry axis and the current deformable object image, a spatial action graph is calculated through a neural network model;

[0016] According to the spatial action graph, the maximum value point is selected as the first grasping point;

[0017] calculating a second grasping point by operability optimization according to the first grasping point, the folding symmetry axis and the current deformable object image;

[0018] According to the first grasping point, the symmetric point about the folding symmetry axis is calculated as the first placement point;

[0019] According to the second grasping point, the symmetric point about the folding symmetry axis is calculated as the second placement point;

[0020] Based on the first grasping point and the placement point, and the second grasping point and the placement point, a dual-arm grasping-placing action is performed to manipulate the deformable object.

[0021] As a further improvement, the calculation of the folding symmetry axis of the present invention specifically includes:

[0022] The parameterized polygonal model is converted into prompt words in text format, which include geometric shape information of the deformable object. The prompt words and the current language sub-goal are input into the large language model, and the prompt words and the language sub-goal are calculated by the large language model to generate an intermediate action representation - the folding symmetry axis, which represents the symmetry axis of the object for folding operation. The folding symmetry axis is used as the input of subsequent steps for the calculation of subsequent grasping points and placement points.

[0023] As a further improvement, the present invention calculates the spatial action graph through a neural network model, specifically:

[0024] The folding symmetry axis and the deformable object image are input into the neural network. The neural network model is a conditional UNet neural network, in which the position encoding of the starting coordinates of the folding symmetry axis is used as the conditional input, and image features are extracted from the deformable object image to generate a spatial action graph. Each pixel in the spatial action graph is a predicted value of the operation result of the position as the grasping point.

[0025] As a further improvement, the present invention can be optimized by maximizing the weighted sum of the area of ​​the quadrilateral and the distance from the point to the folding symmetry axis to determine the second grasping point. The specific steps are as follows:

[0026] Calculate the first grip point , Second grab point and the folding symmetry axis Starting point and end point The area of ​​the enclosed quadrilateral ; Calculate the second grip point To the folding symmetry axis Distance ; The optimization function is the area of ​​the quadrilateral and distance The weighted sum of is as follows:

[0027] ;

[0028] ;

[0029] in is the area of ​​the bounding box of the deformable object, is the weighting coefficient, is an object contour extracted from the deformable object image, and the constraints are that the second grasping point is located on the object contour and the second grasping point is located on the left side of the folding symmetry axis.

[0030] As a further improvement, the present invention parses the language target instruction through a large language model and decomposes it into a series of language sub-targets, specifically including the following steps:

[0031] The user's language target instruction is input into the large language model; the user's language target instruction is semantically analyzed through the large language model to identify multiple subtasks implied in the language target instruction; each subtask is converted into an independent language sub-target instruction, and each language sub-target instruction corresponds to a specific operation step.

[0032] As a further improvement, the present invention fits a parameterized polygonal model based on the current deformable object image, specifically:

[0033] Perform edge detection on deformable object images and extract the contours of objects ; Extract the initial parametric polygonal model outline ; On the object outline and the initial parametric polygonal model outline Uniform sampling is performed on the top to obtain the object contour sampling point set And polygon model contour sampling point set ; Calculate object contour sampling points And polygon model contour sampling points Chamfer distance between , the formula is:

[0034] ;

[0035] ;

[0036] Use black box optimization to adjust the parameters of the polygonal model so that the distance The parameterized polygonal model obtained by fitting is the optimal model.

[0037] As a further improvement, the dual-arm grabbing-placing operation of the present invention is performed by a robotic arm, and the robotic arm performs motion control according to the positions of the grabbing point and the placing point; specifically, the following steps are included:

[0038] According to the first grasping point and the second grasping point, the robot arm is controlled to move to the position of the grasping point and grasp the object; according to the first placement point and the second placement point, the robot arm is controlled to move to the position of the placement point and place the object; the motion control of the robot arm is realized through the motion planning algorithm to ensure the smoothness and accuracy of the grasping-placing process.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1) The present invention does not rely on precise visual sub-target sequences as references, but takes the user's verbal target instructions as input. Compared with visual target information, verbal target instructions can contain richer semantic information, provide more comprehensive contextual support, and support flexible and diverse task expressions. In addition, the input method of verbal instructions is closer to human natural interaction habits, significantly lowering the operating threshold and improving the closeness and efficiency of human-computer collaboration.

[0041] 2) This paper proposes an innovative three-layer hierarchical planning method, which covers sub-goal decomposition, folded symmetry axis planning, and dual-arm grasping and placing action planning. This hierarchical planning method can effectively solve the operation tasks of complex deformable objects and realize hierarchical planning from high-level task decomposition to specific operation actions. At the same time, this method can plan multi-step operation actions based only on the final goal, showing good performance and adaptability in complex task scenarios.

[0042] 3) This invention introduces a large language model to decompose language sub-goals and plan the folded symmetry axis, giving full play to the powerful ability of the large language model in common sense understanding, and significantly improving the robot's generalization ability when facing new or unseen tasks. This enables the robot to perform well in a wider range of unseen application scenarios;

[0043] 4) The present invention does not require the collection of expert data, thus avoiding the time-consuming and labor-intensive problem of expert data collection in existing methods. Through the pre-training knowledge of the large language model, the robot can autonomously generate reasonable operation strategies, significantly reducing the cost of system development and application and improving the practicality of the method.

[0044] 5) The present invention uses the spatial action graph predicted by the neural network model to calculate the first grasping point, giving full play to the powerful prediction ability of the neural network model. The generated spatial action graph can predict the operation result when each pixel point is used as a grasping point. This prediction method based on the neural network model not only improves the grasping accuracy, but also enables the grasping task to adapt to more complex and changing environments, thereby improving the efficiency and reliability of the overall operation.

[0045] 6) The present invention calculates the second grasping point through operability optimization. The proposed optimization function is simple and effective, and can quickly calculate the optimal grasping position in complex operation scenarios. This optimization function can not only effectively avoid the possible action conflicts of the two arms during the operation, but also fully ensure the coordination and stability of the two-arm operation actions, thereby improving the overall operation efficiency and accuracy. In addition, it can also minimize the internal deformation of the variable object during the operation and maintain its structural and morphological integrity. This method has good versatility and robustness, which helps to improve the adaptability and reliability of the system when processing diverse objects.

[0046] 7) The present invention introduces a parametric polygonal model to represent deformable objects, and uses a black box optimization method to update the polygonal model that best matches the current image observation in real time. The representation method based on the parametric polygonal model effectively reduces the dimension of the state observation of the deformable object. As a dimensionality reduction abstract representation method, it only retains the key state information related to the operation task and is easy to convert into text prompts, so that the large language model can understand the current state of the object, thereby better helping the generation of subsequent operation actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is an algorithm flow chart of a method for operating a deformable object with two arms based on a large language model according to the present invention; DETAILED DESCRIPTION

[0048] The technical solution of the present invention is further described below by combining the accompanying drawings of the specification through specific embodiments:

[0049] The specific implementation method of the present invention is as follows:

[0050] Step 1: Get the user's verbal target instruction, which indicates the effect that the user wants the robot to achieve by manipulating the object. It may not contain specific information about each step, but only general information about the final target, such as "fold the T-shirt into a rectangular block". After receiving the instruction, the robot takes it as the target of the task.

[0051] Step 2: parse the user language target instruction through the large language model, identify the implicit multi-step operation tasks, and convert each step of the operation task into the form of language sub-goals. Taking the user language target instruction of "fold the T-shirt into a rectangular square" in step 1 as an example, a possible sequence of language sub-goals after parsing is:

[0052] Sub-goal 1: Fold the left sleeve of the T-shirt inward;

[0053] Sub-goal 2: Fold the right sleeve of the T-shirt inward;

[0054] Sub-Goal 3: Fold the bottom half of the T-shirt upwards.

[0055] For each language sub-goal, execute the following steps repeatedly until all sub-goals are completed. For example, for sub-goal 1 "Fold the left sleeve of the T-shirt inward", execute all the following steps:

[0056] Step 3: Obtain an image of the current deformable object through a camera or other visual sensor. In this embodiment, an Intel Realsense D435i camera is used. The acquired image format is an RGB image with an image size of 400*400. The robot performs subsequent operations based on the image.

[0057] Step 4: Fit the parameterized polygonal model of the current object based on the current deformable object image. Perform edge detection on the current image and extract the outline of the object. ; Extract the initial parametric polygonal model outline ; In the object outline and the initial parameterized polygonal model outline In this embodiment, 10 points are sampled on each edge to obtain the object contour sampling point set And polygon model contour sampling point set ; Calculate the object contour sampling points and the polygonal model contour sampling points Chamfer distance between , the formula is:

[0058] ;

[0059] ;

[0060] The parameters of the polygonal model are adjusted using black box optimization so that the distance The parameterized polygonal model obtained by fitting is the optimal model.

[0061] Step 5. The large language model calculates the folding symmetry axis of the T-shirt based on sub-goal 1 "Fold the left sleeve of the T-shirt inward" and the parameterized polygon model fitted in step 4. The specific process is: first, convert the parameterized polygon model into a text-formatted prompt word. The specific prompt word format is a dictionary in the Python language. The two keys of the dictionary are the nodes and edges of the parameterized polygon model. The value corresponding to the node key includes the coordinates, semantic information and hierarchical information of each node, and the value corresponding to the edge key includes the connection relationship between the nodes at each level. Next, the prompt word in text format and sub-goal 1 "Fold the left sleeve of the T-shirt inward" are input into the large language model to generate the folding symmetry axis. The folding symmetry axis represents the symmetry axis of the T-shirt folding operation. For example, the calculated folding symmetry axis may be a line from the left armpit to the shoulder of the T-shirt.

[0062] Step six, input the images of the folding symmetry axis and the deformable object into the neural network to generate a spatial action graph. The neural network model used in this embodiment is a conditional UNet neural network. The starting coordinates of the folding symmetry axis are all used as conditional inputs through position encoding based on trigonometric functions, as auxiliary inputs of the network. Taking the current image as input, extract the image feature information from it, and combine it with the position encoding to jointly generate a spatial action graph. In this embodiment, each pixel of the spatial action graph represents the predicted folding effect of the T-shirt when the position is used as the grab point. The larger the pixel value, the better the folding prediction result corresponding to the position, reflecting a more ideal folding effect.

[0063] Step 7: Select the pixel with the largest value according to the spatial action graph. This pixel represents the optimal grasping position predicted by the neural network model and has the highest folding effect prediction value, and use it as the first grasping point.

[0064] Step 8. Calculate the second grasping point through operability optimization based on the first grasping point, the folding symmetry axis and the current image. Specifically, the goal of operability optimization is to maximize a weighted objective function, which consists of two parts: one is the area of ​​the quadrilateral formed by the first grasping point, the second grasping point and the folding symmetry axis, and the other is the vertical distance from the second grasping point to the folding symmetry axis. By weighted summing these two parts, the optimization algorithm can determine the most suitable position for the second grasping point to achieve a more stable and efficient folding operation:

[0065] First calculate the first grab point , Second grab point and the folding symmetry axis Starting point and end point The area of ​​the enclosed quadrilateral ; Then calculate the second grab point To the folding symmetry axis Distance ; The optimization function is defined as the area of ​​the quadrilateral and the distance The weighted sum of is as follows:

[0066] ;

[0067] ;

[0068] in is the area of ​​the bounding box of the deformable object, is the weighting coefficient. In this embodiment , is the object contour extracted from the deformable object image. The constraint condition is that the second grasping point is located on the contour of the deformable object, namely the T-shirt, and the second grasping point is located on the left side of the folding symmetry axis.

[0069] Step nine, calculate the placement point of the robot operation. The specific process is as follows: first, calculate the symmetry point of the first grasping point about the folding symmetry axis, and determine it as the first placement point; then, calculate the symmetry point of the second grasping point about the folding symmetry axis, and determine it as the second placement point.

[0070] Step 10: According to the calculated grabbing and placing points, control the dual-arm robot to perform grabbing and placing actions. The specific process is as follows: one robot arm moves to the first grabbing point position to grab the T-shirt; the other robot arm moves to the second grabbing point position to grab the T-shirt at the same time; then, the dual-arm robot moves to the first placement point and the second placement point respectively to put the T-shirt down smoothly. The motion trajectory of the robot arm is generated by the cubic spline interpolation algorithm to ensure the smoothness and accuracy of the grabbing and placing process.

[0071] After that, proceed to the next sub-goal, which is sub-goal 2: "Fold the right sleeve of the T-shirt inward", and loop through the process from step 3 to step 10.

[0072] The above is not a limitation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several changes, modifications, additions or substitutions can be made without departing from the essential scope of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for dual-arm manipulation of a deformable object based on a large language model, characterized in that: include: Obtain the user's language target instruction; Parse the language target instructions through a large language model and decompose them into a series of language sub-targets; For each language subgoal, the loop iterates and performs the following steps until all subgoals are completed: Get the current deformable object image; Fitting parameterized polygonal models based on deformable object images; Based on the language sub-goals and the parameterized polygonal model, the folding symmetry axis is calculated through the large language model; According to the folding symmetry axis and the current deformable object image, a spatial action graph is calculated through a neural network model; According to the spatial action graph, the maximum value point is selected as the first grasping point; calculating a second grasping point by operability optimization according to the first grasping point, the folding symmetry axis and the current deformable object image; According to the first grasping point, the symmetric point about the folding symmetry axis is calculated as the first placement point; According to the second grasping point, the symmetric point about the folding symmetry axis is calculated as the second placement point; Based on the first grasping point and the first placement point, the second grasping point and the second placement point, a dual-arm grasping-placing action is performed to manipulate the deformable object.

2. The method for dual-arm manipulation of a deformable object based on a large language model according to claim 1, characterized in that: The calculation of the folding symmetry axis specifically includes: The parameterized polygonal model is converted into a prompt word in text format, wherein the prompt word includes geometric shape information of the deformable object; the prompt word and the current language sub-goal are input into a large language model, and the prompt word and the language sub-goal are calculated by the large language model to generate an intermediate action representation - a folding symmetry axis, wherein the folding symmetry axis represents the symmetry axis of the object for folding operation; the folding symmetry axis is used as an input for subsequent steps and is used for calculating subsequent grasping points and placement points.

3. The method for dual-arm manipulation of a deformable object based on a large language model according to claim 1, characterized in that: The calculation of the spatial action graph by the neural network model is specifically as follows: The folding symmetry axis and the deformable object image are input into a neural network. The neural network model is a conditional UNet neural network, wherein the position encoding of the starting coordinates of the folding symmetry axis is used as a conditional input, and image features are extracted from the deformable object image to generate a spatial action graph, in which each pixel in the spatial action graph is a predicted value of an operation result with the position as a grasping point.

4. The method for dual-arm manipulation of a deformable object based on a large language model according to claim 1, 2 or 3, characterized in that: The operability optimization determines the second grasping point by maximizing the weighted sum of the quadrilateral area and the distance from the point to the folding symmetry axis. The specific steps are as follows: Calculate the first grasping point , Second grab point and the folding symmetry axis Starting point and end point The area of ​​the enclosed quadrilateral ; Calculate the second grab point To the folding symmetry axis Distance ; The optimization function is the area of ​​the quadrilateral and the distance The weighted sum of is as follows: ； ； in is the area of ​​the bounding box of the deformable object, is the weighting coefficient, is an object contour extracted from the deformable object image, and the constraint condition is that the second grasping point is located on the object contour and the second grasping point is located on the left side of the folding symmetry axis.

5. The method for dual-arm manipulation of a deformable object based on a large language model according to claim 4, characterized in that: The language target instruction is parsed by the large language model and decomposed into a series of language sub-targets, which specifically includes the following steps: The user language target instruction is input into a large language model; semantic analysis is performed on the user language target instruction through the large language model to identify multiple subtasks implied in the language target instruction; each subtask is converted into an independent language sub-target instruction, and each language sub-target instruction corresponds to a specific operation step.

6. The method for dual-arm manipulation of a deformable object based on a large language model according to claim 5, characterized in that: The method of fitting a parameterized polygonal model based on a deformable object image is specifically as follows: Perform edge detection on the deformable object image to extract the outline of the object ; Extract the initial parametric polygonal model outline ; In the object outline and the initial parameterized polygonal model outline Uniform sampling is performed on the top to obtain the object contour sampling point set And polygon model contour sampling point set ; Calculate the object contour sampling points and the polygonal model contour sampling points Chamfer distance between , the formula is: ； ； The parameters of the polygonal model are adjusted using black box optimization so that the distance The parameterized polygonal model obtained by fitting is the optimal model.

7. The method for dual-arm manipulation of a deformable object based on a large language model according to claim 1, 2, 3, 5 or 6, characterized in that: The dual-arm grab-and-place operation is performed by a robotic arm, and the robotic arm performs motion control according to the positions of the grab points and the placement points; specifically, the following steps are included: According to the first grasping point and the second grasping point, the robot arm is controlled to move to the position of the grasping point and grasp the object; according to the first placement point and the second placement point, the robot arm is controlled to move to the position of the placement point and place the object; the motion control of the robot arm is realized through a motion planning algorithm to ensure the smoothness and accuracy of the grasping-placing process.

Images