A dual-arm manipulation method for deformable objects based on a large language model

Abstract

The present invention discloses a dual-arm operation method for deformable objects based on a large language model. The present invention takes the user's language target instructions as input. Compared with visual target information, language target instructions can contain richer semantic information, provide more comprehensive context support, and support flexible and diverse task expressions. In addition, the input method of language instructions is closer to human natural interaction habits, significantly reduces the operation threshold, and improves the closeness and efficiency of human-computer collaboration. An innovative three-layer hierarchical planning method is also proposed, covering sub-goal decomposition, folded symmetry axis planning, and dual-arm grasping and placement action planning, which can effectively solve the operation tasks of complex deformable objects and realize hierarchical planning from high-level task decomposition to specific operation actions. At the same time, multi-step operation actions can be planned based only on the final goal, showing good performance and adaptability in complex task scenarios.

Description

Technical Field

[0001] The present invention relates to the field of robot operation, and in particular to a dual-arm operation method for a deformable object based on a large language model. Background Art

[0002] In modern industrial production and daily life, there is a growing demand for robots to manipulate deformable objects. In particular, in the home, assistive robots can perform tasks such as grasping and folding clothes according to human instructions, providing convenient intelligent services for family life. In industrial environments, deformable object manipulation technology is widely used in complex scenarios such as packaging and sorting, rope tying, and flexible material assembly. These applications place high demands on the accuracy of robot perception and motion planning. Especially when handling deformable objects, robots need to design reasonable perception and planning algorithms to effectively reduce the dimensionality of observations in the infinite-dimensional state space and calculate accurate sequences of robotic arm operation actions.

[0003] Most existing methods for manipulating deformable objects rely on visual goal guidance. Using a set of visual sub-goal sequences as reference, the robot uses deep learning methods to calculate the action sequence required to achieve the target state. However, existing methods have the following problems:

[0004] 1) Lack of multi-step planning capabilities: Existing methods usually rely on precise sub-goals to complete single-step operation calculations and lack the ability to calculate multi-step operation actions from the final goal, resulting in low efficiency in executing complex tasks.

[0005] 2) Insufficient generalization: Although existing methods perform well on known training tasks, they lack sufficient generalization capabilities when faced with new tasks and have difficulty adapting to diverse operational requirements;

[0006] 3) Dependence on expert teaching data: Existing methods need to rely on expert teaching data for imitation learning training, and the collection process of expert data is time-consuming and labor-intensive, which limits the scalability and practicality of the method. Summary of the Invention

[0007] The present invention addresses the problems existing in the prior art and provides a method for manipulating a deformable object with two arms based on a large language model. The specific technical solutions adopted by the present invention are as follows:

[0008] The present invention discloses a dual-arm operation method for a deformable object based on a large language model, comprising:

[0009] Obtain the user's language target instructions;

[0010] Parse the language target instructions through a large language model and decompose them into a series of language sub-goals;

[0011] For each language subgoal, the loop iterates and performs the following steps until all subgoals are completed:

[0012] Get the current deformable object image;

[0013] Fitting parameterized polygonal models based on deformable object images;

[0014] Based on the language sub-goal and the parameterized polygonal model, the folding symmetry axis is calculated through the large language model;

[0015] According to the folding symmetry axis and the current deformable object image, a spatial action graph is calculated through a neural network model;

[0016] According to the spatial action graph, the maximum value point is selected as the first grasping point;

[0017] Calculating a second grasping point through maneuverability optimization according to the first grasping point, the folding symmetry axis, and the current deformable object image;

[0018] According to the first grasping point, the symmetric point about the folding symmetry axis is calculated as the first placement point;

[0019] According to the second grasping point, the symmetric point about the folding symmetry axis is calculated as the second placement point;

[0020] Based on the first grasping point and placement point, and the second grasping point and placement point, a dual-arm grasping-placement action is performed to manipulate the deformable object.

[0021] As a further improvement, the calculation of the folding symmetry axis of the present invention specifically includes:

[0022] The parametric polygonal model is converted into a prompt word in text format, which includes the geometric shape information of the deformable object. The prompt word and the current language sub-goal are input into the large language model, and the prompt word and the language sub-goal are calculated by the large language model to generate an intermediate action representation - the folding symmetry axis. The folding symmetry axis represents the symmetry axis of the object during the folding operation. The folding symmetry axis serves as input for subsequent steps to calculate the subsequent grasping points and placement points.

[0023] As a further improvement, the present invention calculates the spatial action graph through a neural network model, specifically:

[0024] The folding symmetry axis and the deformable object image are input into the neural network. The neural network model is a conditional UNet neural network, in which the position encoding of the starting coordinates of the folding symmetry axis is used as the conditional input, and image features are extracted from the deformable object image to generate a spatial action graph. Each pixel in the spatial action graph is the predicted value of the operation result of the position as the grasping point.

[0025] As a further improvement, the present invention can optimize the operability by maximizing the weighted sum of the area of ​​the quadrilateral and the distance from the point to the folding symmetry axis to determine the second grasping point. The specific steps are as follows:

[0026] Calculate the first grip point , second grip point and the folding symmetry axis Starting point and end point Area of ​​the enclosed quadrilateral ; Calculate the second gripping point To the folding symmetry axis distance ; The optimization function is the area of ​​the quadrilateral and distance The weighted sum of is as follows:

[0027] ;

[0028] ;

[0029] in is the area of ​​the bounding box of the deformable object, is the weighting coefficient, is the object contour extracted from the deformable object image, and the constraints are that the second grasping point is on the object contour and the second grasping point is on the left side of the folding symmetry axis.

[0030] As a further improvement, the present invention parses the language target instruction through a large language model and decomposes it into a series of language sub-targets, specifically including the following steps:

[0031] The user's language target instructions are input into the large language model; the user's language target instructions are semantically analyzed through the large language model to identify multiple subtasks implicit in the language target instructions; each subtask is converted into an independent language sub-target instruction, and each language sub-target instruction corresponds to a specific operation step.

[0032] As a further improvement, the present invention fits a parameterized polygonal model based on the current deformable object image, specifically:

[0033] Perform edge detection on deformable object images and extract the object's contour ; Extract the initial parametric polygonal model outline ; On the object outline and the initial parametric polygonal model outline Uniform sampling is performed on the object to obtain the object contour sampling point set and polygon model contour sampling point set ; Calculate object contour sampling points and polygon model contour sampling points Chamfer distance between , the formula is:

[0034] ;

[0035] ;

[0036] Use black box optimization to adjust the parameters of the polygon model so that the distance The parameterized polygonal model obtained by fitting is the optimal model.

[0037] As a further improvement, the dual-arm grasping-placing operation of the present invention is performed by a robotic arm, and the robotic arm performs motion control according to the positions of the grasping point and the placement point; specifically, the following steps are included:

[0038] Based on the first grasping point and the second grasping point, the robot arm is controlled to move to the position of the grasping point and grasp the object; based on the first placement point and the second placement point, the robot arm is controlled to move to the position of the placement point and place the object; the motion control of the robot arm is realized through the motion planning algorithm to ensure the smoothness and accuracy of the grasping-placing process.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1) This invention does not rely on a precise sequence of visual sub-targets as a reference, but instead uses the user's verbal target instructions as input. Compared to visual target information, verbal target instructions contain richer semantic information, provide more comprehensive contextual support, and support flexible and diverse task expressions. Furthermore, verbal input is more aligned with natural human interaction habits, significantly lowering the operational threshold and improving the closeness and efficiency of human-machine collaboration.

[0041] 2) This paper proposes an innovative three-layer hierarchical planning approach, encompassing sub-goal decomposition, folded symmetry axis planning, and dual-arm grasping and placement action planning. This hierarchical planning approach effectively solves complex, deformable object manipulation tasks, achieving hierarchical planning from high-level task decomposition to specific manipulation actions. Furthermore, this approach can plan multi-step manipulation actions based solely on the final goal, demonstrating excellent performance and adaptability in complex task scenarios.

[0042] 3) This invention leverages the large language model to decompose language sub-goals and plan the folded symmetry axis, leveraging its powerful ability to understand common sense. This significantly improves the robot's generalization ability when faced with new or unseen tasks. This enables the robot to excel in a wider range of unseen application scenarios.

[0043] 4) This method eliminates the need for expert data collection, avoiding the time-consuming and labor-intensive process of collecting expert data in existing methods. By leveraging pre-trained knowledge from a large language model, the robot can autonomously generate reasonable operational strategies, significantly reducing system development and application costs and improving the method's practicality.

[0044] 5) This invention utilizes a spatial action graph predicted by a neural network model to calculate the first grasp point, fully leveraging the powerful predictive capabilities of the neural network model. The generated spatial action graph can predict the operational outcome for each pixel as a grasp point. This neural network-based prediction method not only improves grasping accuracy but also enables grasping tasks to adapt to more complex and changing environments, thereby enhancing overall operational efficiency and reliability.

[0045] 6) This invention calculates the second grasping point through operability optimization. The proposed optimization function is simple and effective, capable of quickly calculating the optimal grasping position in complex operation scenarios. This optimization function not only effectively avoids potential conflicting movements between the two arms during operation, but also fully ensures the coordination and stability of the two-arm operation, thereby improving overall operational efficiency and accuracy. Furthermore, it minimizes the internal deformation of deformable objects during operation, maintaining their structural and morphological integrity. This method is highly versatile and robust, helping to improve the system's adaptability and reliability when handling diverse objects.

[0046] 7) This invention introduces a parametric polygonal model to represent deformable objects and employs a black-box optimization method to update the polygonal model in real time to best match the current image observation. This parametric polygonal model-based representation effectively reduces the dimensionality of the deformable object's state observation. As a dimensionality-reducing abstract representation, it retains only the key state information relevant to the operational task and can be easily converted into text prompts, enabling the large language model to understand the current object state and thus better assist in the generation of subsequent operational actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is an algorithm flow chart of a method for dual-arm manipulation of a deformable object based on a large language model of the present invention; DETAILED DESCRIPTION

[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0049] The specific implementation method of the present invention is as follows:

[0050] Step 1: Receive the user's verbal target instruction. This instruction expresses the desired effect of the robot's manipulation of the object. It does not need to include specific information about each step, but only a general summary of the final goal, such as "fold the T-shirt into a rectangle." After receiving this instruction, the robot uses it as the task goal.

[0051] Step 2: Analyze the user's language target instruction using the large language model, identify the implicit multi-step operation task, and convert each step of the operation task into a language sub-goal. Taking the user's language target instruction "fold the T-shirt into a rectangular square" in Step 1 as an example, a possible sequence of language sub-goals after parsing is:

[0052] Sub-goal 1: Fold the left sleeve of the T-shirt inward;

[0053] Sub-goal 2: Fold the right sleeve of the T-shirt inward;

[0054] Sub-Goal 3: Fold the bottom half of the T-shirt upwards in half.

[0055] For each language sub-goal, loop through the following steps until all sub-goals are completed. For example, for sub-goal 1, "Fold the left sleeve of the T-shirt inward," execute all the following steps:

[0056] Step 3: Obtain an image of the current deformable object through a camera or other visual sensor. In this embodiment, an Intel Realsense D435i camera is used. The acquired image format is an RGB image with an image size of 400*400. The robot performs subsequent operations based on the image.

[0057] Step 4: Based on the current deformable object image, fit the parameterized polygonal model of the current object. Perform edge detection on the current image and extract the outline of the object. ; Extract the initial parametric polygonal model outline ; In the object outline and the initial parameterized polygonal model outline In this embodiment, 10 points are sampled on each edge to obtain the object contour sampling point set and polygon model contour sampling point set ; Calculate the object contour sampling points and the polygonal model contour sampling points Chamfer distance between , the formula is:

[0058] ;

[0059] ;

[0060] The parameters of the polygonal model are adjusted using black box optimization so that the distance The parameterized polygonal model obtained by fitting is the optimal model.

[0061] Step 5: The large language model calculates the T-shirt's folding axis of symmetry based on sub-goal 1 ("Fold the left sleeve of the T-shirt inward") and the parameterized polygonal model fitted in step 4. The specific process is as follows: First, the parameterized polygonal model is converted into a text prompt. The prompt format is a Python dictionary, with the two dictionary keys representing the nodes and edges of the parameterized polygonal model. The values ​​corresponding to the node keys include the coordinates, semantic information, and hierarchical information of each node, while the values ​​corresponding to the edge keys include the connection between nodes at each level. Next, the text prompt and sub-goal 1 ("Fold the left sleeve of the T-shirt inward") are input into the large language model to generate the folding axis of symmetry. The folding axis of symmetry represents the axis of symmetry of the T-shirt folding operation. For example, the calculated folding axis of symmetry might be a line running from the left armpit to the shoulder of the T-shirt.

[0062] Step 6. Input the images of the folding symmetry axis and the deformable object into the neural network to generate a spatial action graph. The neural network model used in this embodiment is a conditional UNet neural network. The starting coordinates of the folding symmetry axis are all used as conditional inputs through position encoding based on trigonometric functions, which serves as auxiliary inputs of the network. Taking the current image as input, the image feature information is extracted from it, and combined with the position encoding to jointly generate a spatial action graph. In this embodiment, each pixel of the spatial action graph represents the predicted folding effect of the T-shirt when the position is used as the grasping point. The larger the pixel value, the better the folding prediction result corresponding to the position, reflecting a more ideal folding effect.

[0063] Step 7: Select the pixel with the largest value according to the spatial action graph. This pixel represents the optimal grasping position predicted by the neural network model and has the highest folding effect prediction value, and use it as the first grasping point.

[0064] Step 8: Based on the first grasping point, the folding symmetry axis, and the current image, calculate the second grasping point through operability optimization. Specifically, the goal of operability optimization is to maximize a weighted objective function, which consists of two parts: one is the area of ​​the quadrilateral formed by the first grasping point, the second grasping point, and the folding symmetry axis, and the other is the vertical distance from the second grasping point to the folding symmetry axis. By weighted summing these two parts, the optimization algorithm can determine the most suitable position for the second grasping point, thereby achieving a more stable and efficient folding operation:

[0065] First calculate the first grip point , second grip point and the folding symmetry axis Starting point and end point Area of ​​the enclosed quadrilateral ; Then calculate the second grip point To the folding symmetry axis distance ; The optimization function is defined as the area of ​​the quadrilateral and the distance The weighted sum of is as follows:

[0066] ;

[0067] ;

[0068] in is the area of ​​the bounding box of the deformable object, is the weighting coefficient. In this embodiment, , is the object contour extracted from the deformable object image. The constraint condition is that the second grasping point is located on the contour of the deformable object, i.e., the T-shirt, and the second grasping point is located to the left of the folding symmetry axis.

[0069] Step nine, calculate the placement point of the robot operation. The specific process is as follows: first, calculate the symmetric point of the first grasping point about the folding symmetry axis, and determine it as the first placement point; then, calculate the symmetric point of the second grasping point about the folding symmetry axis, and determine it as the second placement point.

[0070] Step 10: Based on the calculated grab and place points, the dual-arm robot is controlled to perform the grab and place actions. The specific process is as follows: one robotic arm moves to the first grab point and grabs the T-shirt; the other robotic arm moves to the second grab point and simultaneously grabs the T-shirt. Next, the dual-arm robot moves to the first and second placement points, respectively, to smoothly place the T-shirt. The robotic arm's motion trajectory is generated using a cubic spline interpolation algorithm to ensure smoothness and accuracy during the grab and place processes.

[0071] After that, proceed to the next sub-goal, that is, sub-goal 2: "Fold the right sleeve of the T-shirt inward", and loop through the process of steps 3 to 10.

[0072] The above is not a limitation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several changes, modifications, additions or substitutions can be made without departing from the essential scope of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for dual-arm manipulation of deformable objects based on a large language model, characterized in that: include: Obtain the user's language target instructions; Parse the language target instructions through a large language model and decompose them into a series of language sub-goals; For each language subgoal, the loop iterates and performs the following steps until all subgoals are completed: Get the current deformable object image; Fitting parameterized polygonal models based on deformable object images; Based on the language sub-goal and the parameterized polygonal model, the folding symmetry axis is calculated through the large language model; According to the folding symmetry axis and the current deformable object image, a spatial action graph is calculated through a neural network model; According to the spatial action graph, the maximum value point is selected as the first grasping point; Calculating a second grasping point through maneuverability optimization according to the first grasping point, the folding symmetry axis, and the current deformable object image; According to the first grasping point, the symmetric point about the folding symmetry axis is calculated as the first placement point; According to the second grasping point, the symmetric point about the folding symmetry axis is calculated as the second placement point; performing a dual-arm grasp-and-place action to manipulate the deformable object based on a first grasp point and a first placement point, a second grasp point, and a second placement point; The calculation of the folding symmetry axis specifically includes: Converting the parameterized polygonal model into a prompt word in text format, wherein the prompt word includes geometric shape information of the deformable object; inputting the prompt word and the current language sub-goal into a large language model; and calculating the prompt word and the language sub-goal using the large language model to generate an intermediate action representation—a folding symmetry axis. The folding symmetry axis represents the symmetry axis of the object during the folding operation; the folding symmetry axis serves as input for subsequent steps to calculate subsequent grasping points and placement points; The operability optimization determines the second grasping point by maximizing the weighted sum of the quadrilateral area and the distance from the point to the folding symmetry axis. The specific steps are as follows: Calculate the first grasping point Second grip point The area S of the quadrilateral formed by the starting point f1 and the end point f2 of the folding symmetry axis F; Calculate the second grasping point The distance d to the folding symmetry axis F; the optimization function is the weighted sum of the area S of the quadrilateral and the distance d, and the specific formula is as follows: Among them S max is the area of ​​the bounding box of the deformable object, γ is the weighting coefficient, C img is an object contour extracted from the deformable object image, with the constraints that the second grasping point is located on the object contour and the second grasping point is located to the left of the folding symmetry axis; The method of fitting a parameterized polygonal model based on a deformable object image is as follows: Perform edge detection on the deformable object image and extract the object's contour C img ; Extract the initial parameterized polygonal model outline C poly ; In the object contour C img and the initial parameterized polygonal model outline C poly uniform sampling on the upper surface to obtain the object contour sampling point set P img And the polygon model contour sampling point set P poly ; Calculate the object contour sampling point P img and the polygonal model contour sampling point P poly The Chamfer distance CD(P img , P poly ), the formula is: CD(P img ,P poly )=D(P img ,P poly )+D(P poly ,P img ); The parameters of the polygonal model are adjusted by black box optimization so that the distance CD (P img , P poly ) is the smallest, and the parameterized polygonal model obtained by fitting is the optimal model.

2. The method for dual-arm manipulation of a deformable object based on a large language model according to claim 1, characterized in that: The spatial action graph is calculated by the neural network model, specifically: The folding symmetry axis and the deformable object image are input into a neural network. The neural network model is a conditional UNet neural network, wherein the position encoding of the starting coordinates of the folding symmetry axis is used as a conditional input, and image features are extracted from the deformable object image to generate a spatial action graph. Each pixel in the spatial action graph is a predicted value of a grasping result with the position as a grasping point.

3. The method for dual-arm manipulation of a deformable object based on a large language model according to claim 1, characterized in that: The language target instruction is parsed by the large language model and decomposed into a series of language sub-targets, which specifically includes the following steps: The user's language target instruction is input into a large language model; the user's language target instruction is semantically analyzed by the large language model to identify multiple subtasks implicit in the language target instruction; each subtask is converted into an independent language sub-target instruction, and each language sub-target instruction corresponds to a specific operation step.

4. The method for dual-arm manipulation of a deformable object based on a large language model according to claim 1, 2 or 3, characterized in that: The dual-arm grasping-placing operation is performed by a robotic arm, and the robotic arm is controlled according to the positions of the grasping point and the placement point; specifically, the following steps are included: According to the first grasping point and the second grasping point, the robot arm is controlled to move to the position of the grasping point and grasp the object; according to the first placement point and the second placement point, the robot arm is controlled to move to the position of the placement point and place the object; the motion control of the robot arm is realized through a motion planning algorithm to ensure the smoothness and accuracy of the grasping-placing process.

Images