Deformable object shape control method based on visual language model and historical data learning

Abstract

The invention discloses a deformable object shape control method based on a visual language model and historical data learning, which not only combines the guidance of a microscopic pixel level in a visual target, but also utilizes information in a language target to carry out task planning and logical reasoning of a macroscopic semantic level. And finally, a visual language model is utilized to fuse target information of the two modes, so that a visual target and a language target promote and cooperate with each other, accurate planning of a deformable object shape control task is ensured, a black box optimization algorithm is utilized to carry out online parameter estimation, real-time tracking and updating of an object deformation state are realized, and the real-time tracking and updating of the deformable object shape control task are realized. The geometric representation of the parameterized polygon model compresses high-dimensional visual observation data into a low-dimensional parameter space, so that the dimensionality of state observation of the deformable object is remarkably reduced, and only key state information related to tasks is reserved.

Description

Technical Field

[0001] The present invention relates to the field of robot intelligent operation, and in particular to a shape control method of a deformable object based on a visual language model and historical data learning. Background Art

[0002] At present, shape control of deformable objects is an important task in the field of intelligent robot operation, and has broad application value and prospects in daily life and industrial production. For example, in a home environment, auxiliary robots can help humans complete tasks such as sorting clothes; in industrial assembly lines, shape control technology of deformable objects can be applied to multiple links such as assembly and packaging. This requires effective perception of deformable objects and planning reasonable shape control actions for deformable objects based on different types and modal targets provided by humans.

[0003] Existing deformable object shape control methods can be divided into visual target-based methods and language target-based methods. Visual target-based methods use a set of visual sub-target sequences provided by human experts as reference and guidance to calculate the action sequence required to achieve the target shape of the deformable object. Language target-based methods use the language target instructions provided by human users as guidance, and gradually calculate the action sequence required to achieve the target shape of the deformable object by decomposing the language instructions. However, existing methods have the following problems:

[0004] 1) Lack of the ability to fuse multimodal target information: The visual modality target can provide fine-grained pixel-level guidance in the shape control process of deformable objects, and the language modality target can provide rich semantic information guidance. Existing methods only rely on single-modality target information and do not fully utilize multimodal target information and effectively fuse it.

[0005] 2) Lack of ability to learn from historical data: During the execution of the method, a large amount of historical execution data will accumulate over time. Existing methods do not have the ability to retrieve, evaluate, and learn from historical data, and do not make full use of valuable historical data to help improve the performance of the current task.

[0006] 3) Poor performance on complex tasks: Existing methods can only handle relatively simple deformable object shape control tasks, such as square cloth, but perform poorly when faced with more complex deformable objects and more complex multi-step tasks. Summary of the invention

[0007] The present invention aims at the problems existing in the prior art and provides a shape control method for a deformable object based on a visual language model and historical data learning. The specific technical solution adopted by the present invention is as follows:

[0008] The present invention discloses a shape control method for a deformable object based on a visual language model and historical data learning, comprising:

[0009] Obtain historical execution data, preprocess the data and build a historical database;

[0010] Obtain user language target instructions and visual sub-target sequences;

[0011] The user's language target instructions are processed through the visual language model and decomposed into a sequence of language sub-targets;

[0012] For each language sub-goal and its corresponding visual sub-goal in the language sub-goal sequence, loop through the following steps until all sub-goals are completed;

[0013] Get the image of the current deformable object and fit the corresponding parameterized polygonal model;

[0014] Fit the corresponding parameterized polygonal model according to the current visual teaching sub-goal;

[0015] Calculate the task matching score between each data in the history database and the current task;

[0016] Calculate the object matching score between each data in the historical database and the current object;

[0017] Filtering a specific amount of historical data from the historical database according to the task matching score and the object matching score;

[0018] Compute the folding symmetry axis required for shape control actions of deformable objects through visual language models;

[0019] Calculating a first grasping point, a first placing point, a second grasping point, and a second placing point of the shape control action by an operability optimization algorithm;

[0020] Perform shape-control actions on deformable objects.

[0021] As a further improvement, the preprocessing of data and building of a historical database described in the present invention specifically includes:

[0022] The deformable object shape control historical execution data is divided into language target-based historical data and visual target-based historical data according to the target type;

[0023] For each piece of historical data based on a language target, the preprocessing process includes: using a pre-trained text embedding encoder to map the user language instructions and language sub-targets in the historical data based on the language target into a dimension of The feature vector of is obtained by fitting the corresponding parameterized polygonal model according to the images of the deformable objects before and after deformation in the historical data, and the execution effect of the historical data is scored by the visual language model, which is recorded as , historical data, preprocessed feature vectors, parameterized polygon models and evaluation scores are combined to build a historical database based on language objectives;

[0024] For each piece of historical data based on visual targets, the parameterized polygonal model of the deformable object before and after the shape change in the historical data and the parameterized polygonal model of the visual teaching sub-target sequence in the historical data are fitted. In addition, the evaluation score calculation process of the historical data based on language targets is the same. The visual language model is also used to score the execution effect of the historical data. The historical data and the parameterized polygonal model obtained by preprocessing and the evaluation score are merged to build a historical database based on visual targets.

[0025] The historical database is constructed by integrating the historical database based on the language target and the visual target.

[0026] As a further improvement, the present invention processes the user's language target instruction through a visual language model and decomposes it into a language sub-target sequence, specifically including:

[0027] The user language target instructions and the visual teaching sub-target sequence are input into the visual language model. The user language target instructions are analyzed at the task level through the visual language model, and the user language instructions are decomposed into a series of language sub-targets. Each language sub-target represents an independent single-step sub-task and corresponds one-to-one to the visual teaching sub-target sequence.

[0028] As a further improvement, the calculation of the task matching score between each data in the history database and the current task described in the present invention specifically includes:

[0029] For the historical data based on language targets in the historical database, if the user language target instruction of one set of data is , the language sub-goal is The corresponding eigenvectors are and ; For the user language target command in the current task and the current language subgoal , and the corresponding eigenvector and , calculate the task matching score , the formula is as follows:

[0030] ;

[0031] in represents the cosine similarity between two feature vectors, is the evaluation score calculated by the visual language model, is the weighting coefficient;

[0032] For the historical data based on visual targets in the historical database, if the parameterized polygonal model corresponding to the visual teaching sub-targets of the two frames before and after the current step in a set of data is and , the parameterized polygonal model corresponding to the visual teaching sub-goal of the current step of the current task in the two frames before and after is and ; Normalize the coordinates of all polygonal models to the range [0,1] and calculate the task matching score :

[0033] ;

[0034] in is the evaluation score calculated by the visual language model, is the weighting coefficient, is the Euclidean distance between polygonal models.

[0035] As a further improvement, the object matching score between each data in the history database and the current object is calculated according to the present invention, specifically including:

[0036] If the image of the deformable object before the shape change in a set of data in the historical data is , the corresponding parameterized polygonal model is ; If the parameterized polygonal model of the current deformable object is ; Transform the parametric polygonal model into and the parameterized polygonal model as Rotate to standard orthogonal position and calculate Euclidean distance between polygonal models As the object matching score , which can be expressed by the formula as follows:

[0037] .

[0038] As a further improvement, the method of calculating the folding symmetry axis required for the shape control action of the deformable object through the visual language model of the present invention specifically includes:

[0039] If the current language sub-goal is , the visual teaching sub-goal is , the parameterized polygonal model corresponding to the visual teaching sub-goal is , the current image is , the corresponding parameterized polygonal model is , filtered from the historical database The historical data of the language target is , The historical data of the visual target is , using this as a multimodal input, through the visual language model Perform inference calculations on the current task to generate the folding symmetry axis required for the shape control action of the deformable object , which can be expressed as follows:

[0040] .

[0041] As a further improvement, the operability optimization algorithm described in the present invention specifically includes:

[0042] The operability optimization algorithm determines the first grasping point and the second grasping point by maximizing the weighted sum of the area of ​​the quadrilateral and the distance from the point to the folding symmetry axis. The specific steps are as follows: If the first grasping point at the current step , Second grab point , folding symmetry axis The starting point is , the end point is ;calculate , , and The area of ​​the enclosed quadrilateral ; Calculate the first grip point To the folding symmetry axis Distance ; Calculate the second grip point To the folding symmetry axis Distance , the specific formula is as follows:

[0043]

[0044] ;

[0045] in is the area of ​​the bounding box of the deformable object, is the weighting coefficient, It is an object contour extracted from the deformable object image, and the constraint conditions are that the first grasping point and the second grasping point are located on the object contour, and the first grasping point and the second grasping point are located on the left side of the folding symmetry axis; the first placement point is the symmetrical point of the first grasping point about the folding symmetry axis, and the second placement point is the symmetrical point of the second grasping point about the folding symmetry axis.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1) The present invention has a powerful ability to fuse multimodal target information. It can comprehensively utilize visual targets and language targets to calculate the shape control actions of deformable objects. It combines the guidance at the micro-pixel level in the visual target and utilizes the information in the language target for task planning and logical reasoning at the macro-semantic level. Finally, the visual language model is used to fuse the target information of the two modalities, so that the visual target and the language target promote and cooperate with each other, ensuring the accurate planning of the shape control task of the deformable object.

[0048] 2) This invention proposes a method for representing deformable objects based on a parametric polygonal model, which uses a black box optimization algorithm to perform online parameter estimation to achieve real-time tracking and updating of the object's deformation state. The parametric polygonal model, a geometric representation, compresses high-dimensional visual observation data into a low-dimensional parameter space, significantly reducing the dimension of the deformable object state observation, and only retains key state information related to the task. At the same time, the polygonal model is easy to convert into a text-formatted prompt word input into the visual language model, which facilitates the large language model to understand, analyze and calculate the deformation state of the deformable object.

[0049] 3) The present invention proposes a method for constructing a classified and stored historical database based on historical execution data. The historical execution data is preprocessed, useful information is extracted, and the data quality is evaluated in combination with a visual language model, and finally classified and stored according to the type of target. This method can not only effectively integrate and store a large amount of historical execution data, but also greatly improve the data retrieval efficiency, making the retrieval process more efficient and convenient. In addition, the database also has the ability to expand online, and can dynamically adapt to different types of data and needs, ensuring that the database always remains efficient and flexible during use.

[0050] 4) The present invention proposes a task matching-object matching dual-layer retrieval and matching method, which can efficiently and accurately retrieve the historical data most relevant to the current task from the historical database. The task matching layer ensures the high relevance of the task by screening out the historical data most relevant to the current task level from the historical database; the object matching layer further selects the historical data closest to the shape of the current deformable object from the screened data, thereby ensuring the applicability of the historical data to the current object. This retrieval and matching method is not only simple to calculate and efficient to implement, but also can screen out high-quality historical data with extremely high efficiency from a huge historical data set, providing precise support for practical applications.

[0051] 5) The present invention can learn from historical data. By comparing the execution process of historical data related to the current task, the visual language model can extract valuable information from the context information, thereby accurately calculating the action representation required to complete the current deformable object shape control task. This method based on historical data learning can continuously optimize the model performance as the amount of data accumulates, and improve the accuracy of task execution. In addition, this method can better adapt to different task scenarios and can be dynamically adjusted according to changes in historical data, thereby ensuring continuous and stable performance.

[0052] 6) The present invention introduces a visual language model and utilizes its powerful ability in common sense understanding to significantly improve the performance on unseen tasks or unseen objects, thereby ensuring the efficiency and reliability of the system in a changing environment.

[0053] 7) The present invention proposes an operability optimization algorithm, which can effectively avoid possible conflicts in the shape control process of deformable objects and ensure the coordination and stability of the actions. At the same time, the optimization algorithm can also minimize the internal deformation in the shape control process to ensure that the actual object shape is highly consistent with the target shape. This not only improves the operating accuracy of the system, but also enhances the stability of the execution process, making shape control more accurate and reliable in complex tasks and dynamic environments.

[0054] 8) The present invention performs well in complex deformable object shape control tasks and can handle various shape control tasks for various types of deformable objects. The present invention can flexibly adjust shape control actions according to specific task requirements to ensure the accuracy and efficiency of the shape control process. At the same time, the multi-tasking capability of the present invention enables it to adapt to various scenarios and requirements, providing targeted operation solutions for different types of deformable objects, greatly improving the applicability and flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is an algorithm flow chart of a deformable object shape control method based on visual language model and historical data learning of the present invention. DETAILED DESCRIPTION

[0056] The present invention discloses a shape control method for a deformable object based on a visual language model and historical data learning. Figure 1 The present invention is an algorithm flow chart of a deformable object shape control method based on a visual language model and historical data learning, the method comprising:

[0057] Obtain the historical execution data of the shape control of the deformable object, pre-process the data and build a historical database for classified storage; the historical execution data of the shape control of the deformable object is divided into historical data based on language targets and historical data based on visual targets according to the target type;

[0058] For each piece of historical data based on a language target, the preprocessing process includes: using a pre-trained text embedding encoder to map the user language instructions and language sub-targets in the historical data based on the language target into a dimension of The feature vector of the deformable object in the historical data is used to fit the corresponding parameterized polygonal model. The visual language model is used to score the execution effect of the historical data. The evaluation score ranges from three discrete values: 1 represents a very good execution effect, 0 represents a general execution effect, and -1 represents a poor execution effect. The input of the visual language model is the language sub-goal , images of deformable objects before and after shape change and , parametric polygonal model of a deformable object before and after shape change and , folding symmetry axis , grab-and-place action with both arms , the evaluation score is calculated by the visual language model , historical data, preprocessed feature vectors, parameterized polygon models and evaluation scores are combined to build a historical database based on language objectives;

[0059] For each piece of historical data based on visual targets, the parameterized polygonal model of the deformable object before and after the shape change in the historical data and the parameterized polygonal model of the visual teaching sub-target sequence in the historical data are fitted. In addition, the evaluation score calculation process of the historical data based on language targets is the same. The visual language model is also used to score the execution effect of the historical data. The historical data and the parameterized polygonal model obtained by preprocessing and the evaluation score are merged to build a historical database based on visual targets.

[0060] The historical database based on language objectives and visual objectives is integrated to build a historical database with classified storage.

[0061] Obtain the user's language target instructions and visual teaching sub-target sequence for the current task;

[0062] The user language target instructions of the current task are processed through the visual language model and decomposed into a language sub-target sequence that corresponds one-to-one with the visual teaching sub-target sequence; the user language target instructions and the visual teaching sub-target sequence are input into the visual language model, and the user language target instructions are analyzed at the task level through the visual language model to decompose the user language instructions into a series of language sub-targets, each language sub-target represents an independent single-step sub-task and corresponds one-to-one with the visual teaching sub-target sequence.

[0063] For each language sub-goal and its corresponding visual teaching sub-goal in the language sub-goal sequence, loop through the following steps until all sub-goals in the sequence are completed:

[0064] Get the image of the current deformable object;

[0065] Fitting a parameterized polygonal model of the current deformable object according to the image of the current deformable object;

[0066] Fitting a parameterized polygonal model of a visual teaching sub-goal according to the current visual teaching sub-goal;

[0067] Calculate the task matching score between each data in the history database and the current task;

[0068] For the historical data based on language targets in the historical database, if the user language target instruction of one set of data is , the language sub-goal is The corresponding eigenvectors are and ; For the user language target command in the current task and the current language subgoal , and the corresponding eigenvector and , calculate the task matching score , the formula is as follows:

[0069] ;

[0070] in represents the cosine similarity between two feature vectors, is the evaluation score calculated by the visual language model, is the weighting coefficient;

[0071] For the historical data based on visual targets in the historical database, if the parameterized polygonal model corresponding to the visual teaching sub-targets of the two frames before and after the current step in a set of data is and , the parameterized polygonal model corresponding to the visual teaching sub-goal of the current step of the current task in the two frames before and after is and ; Normalize the coordinates of all polygonal models to the range [0,1] and calculate the task matching score :

[0072] ;

[0073] in is the evaluation score calculated by the visual language model, is the weighting coefficient, is the Euclidean distance between polygonal models. The calculation method is: first determine whether their number of levels is consistent, and verify whether the number of nodes and node labels in each level are the same. If there is any inconsistency, then , indicating that the two polygon models cannot match; for the case where they can match, the Euclidean distance of the polygon models It is defined as the cumulative sum of the Euclidean distances of each layer, where the Euclidean distance of each layer is defined as the cumulative sum of the distances between all nodes with the same label in that layer.

[0074] Calculate the object matching score between each data in the history database and the current deformable object;

[0075] If the image of the deformable object before the shape change in a set of data in the historical data is , the corresponding parameterized polygonal model is ; If the parameterized polygonal model of the current deformable object is ; Transform the parametric polygonal model into and the parameterized polygonal model as Rotate to standard orthogonal position and calculate Euclidean distance between polygonal models As the object matching score , which can be expressed by the formula as follows:

[0076] .

[0077] Filtering a specific amount of historical data from the historical database according to the task matching score and the object matching score;

[0078] Based on the current language sub-goal, the visual teaching sub-goal and the parametric polygonal model of the visual teaching sub-goal, the filtered historical data, the current deformable object image and the parametric polygonal model of the current deformable object, the folding symmetry axis required for the shape control action of the deformable object is calculated through the visual language model; if the current language sub-goal is , the visual teaching sub-goal is , the parameterized polygonal model corresponding to the visual teaching sub-goal is , the current image is , the corresponding parameterized polygonal model is , filtered from the historical database The historical data of the language target is , The historical data of the visual target is , using this as a multimodal input, through the visual language model Perform inference calculations on the current task to generate the folding symmetry axis required for the shape control action of the deformable object , which can be expressed as follows:

[0079] .

[0080] According to the folding symmetry axis and the image of the current deformable object, the first grasping point, the first placement point, the second grasping point and the second placement point of the shape control action are calculated by the operability optimization algorithm; the operability optimization algorithm determines the first grasping point and the second grasping point by maximizing the weighted sum of the area of ​​the quadrilateral and the distance from the point to the folding symmetry axis. The specific steps are as follows: if the first grasping point of the current step , Second grab point , folding symmetry axis The starting point is , the end point is ;calculate , , and The area of ​​the enclosed quadrilateral ; Calculate the first grip point To the folding symmetry axis Distance ; Calculate the second grip point To the folding symmetry axis Distance , the specific formula is as follows:

[0081]

[0082] ;

[0083] in is the area of ​​the bounding box of the deformable object, is the weighting coefficient, It is an object contour extracted from the deformable object image, and the constraint conditions are that the first grasping point and the second grasping point are located on the object contour, and the first grasping point and the second grasping point are located on the left side of the folding symmetry axis; the first placement point is the symmetrical point of the first grasping point about the folding symmetry axis, and the second placement point is the symmetrical point of the second grasping point about the folding symmetry axis.

[0084] A shape control action of the deformable object is performed according to the first grasping point, the first placement point, the second grasping point and the second placement point.

[0085] The specific implementation method of the present invention is as follows:

[0086] Step 1: Obtain the historical execution data of the shape control of deformable objects. The historical data may include the execution data of various shape control tasks of various deformable objects including square cloth, shirt, pants, etc. For example, the data of tasks such as folding the opposite sides of the square cloth twice, folding the shirt in half, and folding the trouser legs upwards. Preprocess the data and build a historical database for classified storage. According to the type of target, it is divided into a historical database based on language targets and a database based on visual targets.

[0087] Step 2: Get the current task user language target instruction and visual teaching sub-target sequence. For example, if the user language target instruction is "fold the pants into a rectangular block", the visual teaching sub-target sequence includes images of pants in a flat state, pants folded from left to right, and pants folded from bottom to top.

[0088] Step 3: Process the current task user language target instruction through the visual language model and decompose it into a language sub-target sequence that corresponds to the visual teaching sub-target sequence. The decomposed language sub-target sequence is:

[0089] Sub-goal 1: Fold the pants in half from left to right;

[0090] Sub-goal 2: Fold the folded pants in half from bottom to top.

[0091] For each language sub-goal and its corresponding visual teaching sub-goal, repeat the following steps until all sub-goals in the sequence are completed. For example, for sub-goal 1 "fold the pants in half from left to right" and its corresponding visual teaching sub-goal, execute all the following steps:

[0092] Step 4: Acquire an image of the current deformable object through a camera. In this embodiment, an Intel RealSense D435i camera is selected to acquire an RGB image with a resolution of 400*400.

[0093] Step 5: Fit the corresponding parameterized polygonal model according to the image of the current deformable object. Perform edge detection on the RGB image obtained in step 5 to extract the contour of the deformable object, the pants. , extract the contour of the initial state parameterized polygonal model ; On the object outline and the initial parametric polygonal model outline Ten points are uniformly sampled on each edge of the deformable object contour sampling point set And polygon model contour sampling point set ; Calculate the contour sampling points of the deformable object And polygon model contour sampling points Between distance , the formula is:

[0094]

[0095]

[0096] Using black box optimization to continuously adjust the parameters of the polygonal model The parameterized polygonal model obtained by fitting is the optimal model that best fits the current RGB image.

[0097] Step 6: Fit the parameterized polygonal model corresponding to the visual teaching sub-target according to the current visual teaching sub-target. The fitting process is consistent with step 5.

[0098] Step 7: Use the pre-trained text embedding encoder to calculate the feature vectors of the user language target instruction "fold the pants into a rectangular block" and the current language sub-goal "fold the pants in half from left to right". The text embedding encoder is the pre-trained OpenAI text-embedding-3-large model, and the generated feature vector dimension is .

[0099] Step 8: Calculate the task matching score in the language target-based historical database based on the feature vector corresponding to the user language target instruction "fold the pants into a rectangular square" and the current language sub-goal "fold the pants in half from left to right". Assume that the user language target instruction of a set of data in the language target-based historical database is , the language sub-goal is , and their corresponding eigenvectors are and For the user language target command in the current task and the current language subgoal , and the corresponding eigenvector and , calculate the task matching score , the formula is as follows:

[0100]

[0101] in Represents the cosine similarity between two feature vectors. A higher value indicates a higher similarity between the two vectors, which in turn indicates a higher semantic similarity. is the evaluation score of the execution effect calculated by the visual language model, is the weighting coefficient.

[0102] Assume that a set of data in the language-based historical database is a user language target instruction For “Fold the shirt into a rectangular square”, the language subgoal is "fold the right sleeve of the shirt inward from right to left", and the execution process of the historical data is successful, then .

[0103] Assume that another set of data based on the language history database user language target instruction For “fold the trousers into a square”, the language subgoal is "fold the trousers from left to right", and the execution process of the historical data is successful, then Therefore, compared with the previous set of historical data, this set of data has a higher task matching score and will be selected first.

[0104] The task matching scores are calculated for all the historical data in the language target-based historical database, and the historical data are sorted in descending order according to the task matching scores.

[0105] Step 9: Filter out 20 pieces of language target-based historical data from the language target-based historical database in descending order of task matching scores.

[0106] Step 10: Based on the parametric polygonal model of the current deformable object, the object matching score with the current deformable object is calculated in the 20 historical data based on language targets that have been screened out. Assume that the image of the deformable object before the shape change in a set of data in the historical data based on language targets that have been screened out is , the corresponding parameterized polygonal model is ; Assume that the parameterized polygonal model of the current deformable object is . The parametric polygonal model is and the parameterized polygonal model as Rotate to standard orthogonal position and calculate Euclidean distance between polygonal models as the object matching score.

[0107] Euclidean distance between polygonal models The calculation method is: for a given polygonal model and First, check whether their number of levels is consistent, and verify whether the number of nodes and node labels in each level are the same. If there is any inconsistency, then , indicating that the two polygon models cannot match; for the case where they can match, the Euclidean distance of the polygon models It is defined as the cumulative sum of the Euclidean distances of each layer, where the Euclidean distance of each layer is defined as the cumulative sum of the distances between all nodes with the same label in that layer, which can be expressed by the formula:

[0108]

[0109] in For polygonal models and The number of levels, For polygonal models The number of nodes in the layer, and Representing polygonal models and No. Layer nodes, is the distance between two nodes. Therefore, the object matching score It can be expressed as follows:

[0110]

[0111] For the parametric polygon model corresponding to the current pants, the number of levels , the number of nodes in the first layer is , so the calculation formula of object matching score can be concretized as: .

[0112] Object matching scores are calculated for the 20 historical data based on language targets that are screened, and the historical data are sorted in descending order according to the object matching scores.

[0113] Step 11: Further select 4 pieces of historical data based on language targets based on the object matching score.

[0114] Step 12: Calculate the task matching score in the history database based on visual targets based on the parameterized polygonal model of the current visual teaching sub-target. Assume that the parameterized polygonal model corresponding to the visual sub-target of the two frames before and after the current step in a set of data in the history database based on visual targets is and , the parameterized polygonal model corresponding to the visual sub-target of the two frames before and after the current step of the current task is and . Normalize the coordinates of all polygonal models to the range [0,1] and calculate the task matching score :

[0115]

[0116] in is the Euclidean distance between polygonal models, is the evaluation score calculated by the visual language model, is the weighting coefficient. The task matching score is calculated for all historical data in the history database based on visual targets, and the historical data are sorted in descending order according to the task matching score.

[0117] Step 13: Filter out 20 pieces of visual target-based historical data from the visual target-based historical database in descending order of task matching scores.

[0118] Step 14: Based on the parametric polygonal model of the current deformable object, the object matching score of the filtered historical data based on the visual target and the current deformable object is calculated, and the object matching scores are sorted from high to low. The calculation process of the object matching score is exactly the same as step 10.

[0119] Step 15: Further filter out 4 historical data based on visual targets based on the object matching score.

[0120] Step 16: Based on the current language sub-goal, the visual teaching sub-goal and the corresponding parametric polygonal model, the selected historical data based on the language goal and the visual goal, the current deformable object image and the corresponding parametric polygonal model, the folding symmetry axis required for the shape control action of the deformable object is calculated through the visual language model. Let the current language sub-goal be , the visual teaching sub-goal is , the parameterized polygonal model corresponding to the visual sub-target is , the current image is , the corresponding parameterized polygonal model is , 4 groups of historical data based on language objectives were screened from the historical database: , the four sets of historical data based on visual targets are , using this as a multimodal input, through the visual language model Perform inference calculations on the current task to generate the folding symmetry axis required for the shape control action of the deformable object , which can be expressed as follows:

[0121]

[0122] The folding symmetry axis calculated at the current step is the vector from the midpoint of the lower left and lower right vertices of the pants to the midpoint of the upper left and upper right vertices.

[0123] Step 17: According to the folding symmetry axis The first grasping point, the first placement point, the second grasping point and the second placement point of the shape control action are calculated by the operability optimization algorithm. The operability optimization algorithm determines the first grasping point and the second grasping point by maximizing the weighted sum of the area of ​​the quadrilateral and the distance from the point to the folding symmetry axis. The specific steps are as follows: Set the first grasping point of the current step , Second grab point , folding symmetry axis The starting point is , the end point is ;calculate , , and The area of ​​the enclosed quadrilateral ; Calculate the first grip point To the folding symmetry axis Distance ; Calculate the second grip point To the folding symmetry axis Distance , the specific formula is as follows:

[0124]

[0125]

[0126] in is the area of ​​the bounding box of the deformable object, is the weighting coefficient, It is the object contour extracted from the deformable object image, and the constraint conditions are that the first grasping point and the second grasping point are located on the object contour, and the first grasping point and the second grasping point are located on the left side of the folding symmetry axis. The first placement point is the symmetric point of the first grasping point about the folding symmetry axis, and the second placement point is the symmetric point of the second grasping point about the folding symmetry axis.

[0127] Step 18: Execute the shape control action of the deformable object according to the first grasping point, the first placement point, the second grasping point and the second placement point. The specific execution process is to grasp the deformable object at the positions of the first grasping point and the second grasping point at the same time, move to the positions of the first placement point and the second placement point respectively and release, and the action trajectory is generated by the cubic spline interpolation algorithm to ensure the smoothness and accuracy of the process.

[0128] Then proceed to the next sub-goal, that is, sub-goal 2 "fold the folded pants in half from bottom to top", and repeat the process from step 4 to step 18.

[0129] The above is not a limitation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several changes, modifications, additions or substitutions can be made without departing from the essential scope of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A deformable object shape control method based on visual language model and historical data learning, characterized in that: include: Obtain historical execution data, preprocess the data and build a historical database; Obtain user language target instructions and visual sub-target sequences; The user's language target instructions are processed through the visual language model and decomposed into a sequence of language sub-targets; For each language sub-goal and its corresponding visual sub-goal in the language sub-goal sequence, loop through the following steps until all sub-goals are completed: Get the image of the current deformable object and fit the corresponding parameterized polygonal model; Fit the corresponding parameterized polygonal model according to the current visual sub-target; Calculate the task matching score between each data in the history database and the current task; Calculate the object matching score between each data in the historical database and the current object; Filtering a specific amount of historical data from the historical database according to the task matching score and the object matching score; Compute the folding symmetry axis required for shape control actions of deformable objects through visual language models; Calculating a first grasping point, a first placing point, a second grasping point, and a second placing point of the shape control action by an operability optimization algorithm; Perform shape-control actions on deformable objects.

2. The method for controlling the shape of a deformable object based on a visual language model and historical data learning according to claim 1, characterized in that: The preprocessing of data and construction of a historical database specifically includes: The deformable object shape control historical execution data is divided into historical data based on language targets and historical data based on visual targets according to the target type; For each piece of historical data based on a language target, the preprocessing process includes: using a pre-trained text embedding encoder to map the user language instructions and language sub-targets in the historical data based on the language target into a dimension of The feature vector of is obtained by fitting the corresponding parameterized polygonal model according to the images of the deformable object before and after deformation in the historical data, and the execution effect of the historical data is scored by the visual language model, which is recorded as , combining the historical data and the preprocessed feature vectors, parameterized polygonal models and evaluation scores to construct a historical database based on language objectives; For each piece of historical data based on visual targets, the parameterized polygonal model of the deformable object before and after the shape change in the historical data and the parameterized polygonal model of the visual teaching sub-target sequence in the historical data are fitted. In addition, the evaluation score calculation process of the historical data based on language targets is the same, and the visual language model is also used to score the execution effect of the historical data. The historical data and the parameterized polygonal model obtained by preprocessing and the evaluation score are merged to construct a historical database based on visual targets; The historical database based on the language target and the visual target is integrated to construct the historical database.

3. The deformable object shape control method based on visual language model and historical data learning according to claim 1, characterized in that: The visual language model is used to process the user's language target instruction and decompose it into a language sub-target sequence, which specifically includes: The user language target instructions and the visual teaching sub-target sequence are input into the visual language model, and the user language target instructions are analyzed at the task level through the visual language model to decompose the user language instructions into a series of language sub-targets, each language sub-target represents an independent single-step sub-task and corresponds one-to-one to the visual teaching sub-target sequence.

4. The method for controlling the shape of a deformable object based on a visual language model and historical data learning according to claim 1, characterized in that: The calculation of the task matching score between each data in the history database and the current task specifically includes: For the historical data based on language targets in the historical database, if the user language target instruction of one set of data is , the language sub-goal is The corresponding eigenvectors are and ; For the user language target instruction in the current task and the current language sub-goal , and the corresponding eigenvector and , calculate the task matching score , the formula is as follows: ； in represents the cosine similarity between two feature vectors, is the evaluation score calculated by the visual language model, is the weighting coefficient; For the historical data based on visual targets in the historical database, if the parameterized polygonal model corresponding to the visual teaching sub-targets of the two frames before and after the current step in a set of data is and , the parameterized polygonal model corresponding to the visual teaching sub-goal of the current step of the current task in the two frames before and after is and ; Normalize the coordinates of all polygonal models to the range [0,1] and calculate the task matching score : ； in is the evaluation score calculated by the visual language model, is the weighting coefficient, is the Euclidean distance between polygonal models.

5. The method for controlling the shape of a deformable object based on a visual language model and historical data learning according to claim 1, characterized in that: The calculation of the object matching score between each data in the history database and the current object specifically includes: If the image of the deformable object before the shape change in a set of data in the historical data is , the corresponding parameterized polygonal model is ; If the parameterized polygonal model of the current deformable object is ; The parameterized polygonal model is and the parameterized polygonal model as Rotate to standard orthogonal position and calculate Euclidean distance between polygonal models As the object matching score , which can be expressed by the formula as follows: 。 6. The method for controlling the shape of a deformable object based on a visual language model and historical data learning according to claim 1, characterized in that: The method of calculating the folding symmetry axis required for the shape control action of the deformable object through the visual language model specifically includes: If the current language sub-goal is , the visual teaching sub-goal is , the parameterized polygonal model corresponding to the visual teaching sub-goal is , the current image is , the corresponding parameterized polygonal model is , the historical database is screened to obtain The historical data of the language target is , The historical data of the visual target is , using this as a multimodal input, through the visual language model Perform inference calculation on the current task to generate the folding symmetry axis required for the shape control action of the deformable object , which can be expressed as follows: 。 7. The method for controlling the shape of a deformable object based on a visual language model and historical data learning according to claim 1, characterized in that: The operability optimization algorithm specifically includes: The operability optimization algorithm determines the first grasping point and the second grasping point by maximizing the weighted sum of the area of ​​the quadrilateral and the distance from the point to the folding symmetry axis. The specific steps are as follows: if the first grasping point , Second grab point , the folding symmetry axis The starting point is , the end point is ;calculate , , and The area of ​​the enclosed quadrilateral ; Calculate the first grasping point To the folding symmetry axis Distance ; Calculate the second grasping point To the folding symmetry axis Distance , the specific formula is as follows: ； ； in is the area of ​​the bounding box of the deformable object, is the weighting coefficient, It is an object contour extracted from the deformable object image, and the constraint conditions are that the first grasping point and the second grasping point are located on the object contour, and the first grasping point and the second grasping point are located on the left side of the folding symmetry axis; the first placement point is a symmetrical point of the first grasping point about the folding symmetry axis, and the second placement point is a symmetrical point of the second grasping point about the folding symmetry axis.

Images