A flexible cloth operation strategy optimization method, device, equipment and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a flexible cloth operation strategy optimization method and device, equipment and a storage medium. The method comprises the following steps: generating a distillation loss of a student model according to an operation sequence of a preset flexible cloth output by the student model at a current time step, an operation sequence of the preset flexible cloth output by a teacher model at the current time step and a second loss model, generating a smoothing loss of the student model through a third loss model; generating a total loss of the student model according to a value loss of the student model, the distillation loss of the student model, the smoothing loss of the student model and a total loss model, updating the student model through an optimizer until the total loss of the student model is smaller than a preset loss value, then stopping updating the student model and saving the updated student model; and generating an operation sequence of a current flexible cloth through the updated student model. The application is beneficial to improving the generation efficiency of the operation sequence of the current flexible cloth.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for optimizing flexible fabric operation strategies. Background Technology

[0002] Flexible fabrics are sheet-like flexible materials made from textile fibers through processes such as spinning and weaving. They are characterized by their softness, flexibility, foldability, and ability to conform to the human body or irregular curved surfaces, making them widely applicable in wearable devices. In practical applications and automated processing, flexible fabrics require specific handling strategies to complete the processing.

[0003] However, traditional flexible fabric manipulation strategies rely on manual operation, requiring the manual setting of operation sequences for each piece of flexible fabric. This manual approach consumes significant human and time resources, increasing the generation time of the operation sequences and making it susceptible to human intervention. Therefore, it hinders the efficiency of generating operation sequences for flexible fabrics. Consequently, how to generate operation sequences for the current flexible fabric is a problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for optimizing flexible fabric operation strategies to solve the aforementioned technical problem of how to generate the current operation sequence of flexible fabric.

[0005] In a first aspect, embodiments of this application provide a method for optimizing the operation strategy of flexible fabric, applied to electronic devices, the method comprising:

[0006] The system acquires the captured image of the preset flexible fabric and the language command of the preset flexible fabric. It then concatenates the feature vector of the captured image of the preset flexible fabric, the feature vector of the language command of the preset flexible fabric, and the random noise vector to generate the concatenated feature vector of the preset flexible fabric. Finally, it inputs the concatenated feature vector of the preset flexible fabric into the student model to obtain the operation sequence of the preset flexible fabric output by the student model at the current time step.

[0007] The predicted operation sequence of the preset flexible fabric output by the student model at the current time step is input into the value evaluator. The value evaluator generates the return estimate of the student model. The VLA model is selected as the teacher model. The feature vector of the captured image of the preset flexible fabric and the feature vector of the language instruction of the preset flexible fabric are input into the teacher model. The operation sequence of the preset flexible fabric output by the teacher model at the current time step is obtained from the output information of the teacher model.

[0008] The value loss of the student model is generated by the return estimate and the first loss model. The distillation loss of the student model is generated by the operation sequence of the preset flexible cloth output by the student model at the current time step, the operation sequence of the preset flexible cloth output by the teacher model at the current time step, and the second loss model. The smoothing loss of the student model is generated by the third loss model.

[0009] Based on the student model's value loss, distillation loss, smoothing loss, and total loss model, the total loss of the student model is generated. With reducing the total loss as the optimization objective, the student model is updated by the optimizer until the total loss of the student model is less than the preset loss value. Only then is the student model updated and the updated student model saved.

[0010] Acquire the captured image of the current flexible fabric and the language command of the current flexible fabric. Concatenate the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate the concatenated feature vector of the current flexible fabric. Input the concatenated feature vector of the current flexible fabric into the updated student model. Through the updated student model, generate the operation sequence of the current flexible fabric.

[0011] In one possible implementation of the first aspect, the step of acquiring a captured image of the preset flexible fabric and the language instructions for the preset flexible fabric, concatenating the feature vector of the captured image of the preset flexible fabric, the feature vector of the language instructions for the preset flexible fabric, and the random noise vector to generate a concatenated feature vector of the preset flexible fabric, and inputting the concatenated feature vector of the preset flexible fabric into the student model to obtain the operation sequence of the preset flexible fabric output by the student model at the current time step, includes:

[0012] Obtain samples from the dataset, extract images of the preset flexible fabric and language instructions of the preset flexible fabric from the samples, use the image feature extraction module to extract features from the images of the preset flexible fabric to generate feature vectors of the images of the preset flexible fabric, and use the semantic encoding module to semantically encode the language instructions of the preset flexible fabric to generate feature vectors of the language instructions of the preset flexible fabric.

[0013] Obtain random noise vectors from the standard normal distribution, and concatenate the feature vectors of the captured image of the preset flexible fabric, the feature vectors of the language instructions of the preset flexible fabric, and the random noise vectors to generate the concatenated feature vector of the preset flexible fabric. Input the concatenated feature vector of the preset flexible fabric into the student model, obtain the output information of the student model, and obtain the operation sequence of the preset flexible fabric output by the student model at the current time step from the output information of the student model.

[0014] In one possible implementation of the first aspect, the step of acquiring a captured image of the current flexible fabric and the language instructions for the current flexible fabric, concatenating the feature vectors of the captured image and the language instructions for the current flexible fabric to generate a concatenated feature vector of the current flexible fabric, inputting the concatenated feature vector of the current flexible fabric into the updated student model, and generating an operation sequence for the current flexible fabric through the updated student model, including:

[0015] The system acquires the captured image of the current flexible fabric and the language command of the current flexible fabric. The image feature extraction module extracts features from the captured image of the current flexible fabric to generate a feature vector of the captured image of the current flexible fabric. The semantic encoding module performs semantic encoding on the language command of the current flexible fabric to generate a feature vector of the language command of the current flexible fabric.

[0016] The feature vectors of the captured image of the current flexible fabric and the feature vectors of the language commands of the current flexible fabric are concatenated to generate the concatenated feature vector of the current flexible fabric. The concatenated feature vector of the current flexible fabric is then input into the updated student model, and the operation sequence of the current flexible fabric is generated through the updated student model.

[0017] In one possible implementation of the first aspect, after acquiring the captured image of the current flexible fabric and the language instructions for the current flexible fabric, concatenating the feature vector of the captured image of the current flexible fabric and the feature vector of the language instructions for the current flexible fabric to generate a concatenated feature vector of the current flexible fabric, and inputting the concatenated feature vector of the current flexible fabric into the updated student model, and generating the operation sequence of the current flexible fabric through the updated student model, the method further includes:

[0018] According to the current operation sequence of the flexible fabric, control the robot's end effector to perform target operations on the current flexible fabric. The target operations include grasping, flattening and folding operations.

[0019] In one possible implementation of the first aspect, the first loss model is defined as follows:

[0020] ;

[0021] This represents the value loss of the student model. The greater the value loss of the student model, the weaker the performance of the preset flexible cloth operation sequence output by the student model at the current time step. The smaller the value loss of the student model, the stronger the performance of the preset flexible cloth operation sequence output by the student model at the current time step.

[0022] Indicates the learning parameters;

[0023] Indicated by A value evaluator for learning parameters;

[0024] This indicates the current state information of the preset flexible fabric, which includes the current position and shape data of the preset flexible fabric.

[0025] This represents the preset flexible cloth operation sequence output by the student model at the current time step;

[0026] Language commands indicating preset flexible fabric;

[0027] This represents the estimated return value of the student model generated by the value evaluator based on the current state of the preset flexible fabric and the operation sequence of the preset flexible fabric output by the student model at the current time step.

[0028] In one possible implementation of the first aspect, the second loss model is defined as follows:

[0029] ;

[0030] This represents the distillation loss of the student model;

[0031] This represents the preset flexible cloth operation sequence output by the student model at the current time step;

[0032] This represents the preset flexible fabric operation sequence output by the teacher model at the current time step;

[0033] express and The square of the L2 norm of the difference between them.

[0034] In one possible implementation of the first aspect, the third loss model is defined as follows:

[0035] ;

[0036] This represents the smoothing loss of the student model;

[0037] This represents the preset flexible cloth operation sequence output by the student model at the current time step;

[0038] This represents the preset flexible cloth operation sequence output by the student model in the previous time step at the current time step;

[0039] express and The square of the L2 norm of the difference between them;

[0040] The total loss model is defined as follows:

[0041] ;

[0042] This represents the total loss of the student model; the higher the total loss of the student model, the weaker the overall performance of the student model in terms of value, distillation, and smoothing; the lower the total loss of the student model, the stronger the overall performance of the student model in terms of value, distillation, and smoothing.

[0043] Weight parameters representing the value loss of the student model;

[0044] This represents the value loss of the student model;

[0045] The weight parameters represent the distillation loss of the student model;

[0046] This represents the distillation loss of the student model;

[0047] The weight parameters represent the smoothing loss of the student model;

[0048] This represents the smoothing loss of the student model.

[0049] Secondly, embodiments of this application provide a flexible fabric handling strategy optimization device, applied to electronic devices, including:

[0050] The first acquisition module is used to acquire the captured image of the preset flexible fabric and the language instructions of the preset flexible fabric. It splices the feature vector of the captured image of the preset flexible fabric, the feature vector of the language instructions of the preset flexible fabric, and the random noise vector to generate the spliced ​​feature vector of the preset flexible fabric. It inputs the spliced ​​feature vector of the preset flexible fabric into the student model to obtain the operation sequence of the preset flexible fabric output by the student model at the current time step.

[0051] The second acquisition module is used to input the predicted operation sequence of the preset flexible fabric output by the student model at the current time step into the value evaluator, generate the return estimate of the student model through the value evaluator, select the VLA model as the teacher model, input the feature vector of the captured image of the preset flexible fabric and the feature vector of the language instruction of the preset flexible fabric into the teacher model, and obtain the operation sequence of the preset flexible fabric output by the teacher model at the current time step from the output information of the teacher model.

[0052] The first generation module is used to generate the value loss of the student model through the return estimate and the first loss model. Based on the operation sequence of the preset flexible cloth output by the student model at the current time step, the operation sequence of the preset flexible cloth output by the teacher model at the current time step, and the second loss model, the distillation loss of the student model is generated. The smoothing loss of the student model is generated through the third loss model.

[0053] The update module is used to generate the total loss of the student model based on the student model's value loss, distillation loss, smoothing loss, and total loss model. With the optimization objective of reducing the total loss, the student model is updated by the optimizer until the total loss of the student model is less than the preset loss value. Then the update of the student model stops and the updated student model is saved.

[0054] The second generation module is used to acquire the captured image of the current flexible fabric and the language command of the current flexible fabric. It concatenates the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate the concatenated feature vector of the current flexible fabric. The concatenated feature vector of the current flexible fabric is input into the updated student model. The updated student model generates the operation sequence of the current flexible fabric.

[0055] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the flexible fabric operation strategy optimization method described in the first aspect above.

[0056] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the flexible fabric operation strategy optimization method described in the first aspect above.

[0057] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the flexible fabric operation strategy optimization method described in the first aspect.

[0058] The beneficial effects of the embodiments of this application are as follows:

[0059] Firstly, the system acquires the captured image of the current flexible fabric and the language command of the current flexible fabric. It then concatenates the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate a concatenated feature vector of the current flexible fabric. This concatenated feature vector of the current flexible fabric is then input into the updated student model. Through the updated student model, the system generates the operation sequence of the current flexible fabric. This process eliminates the need for manual operation, thus reducing the generation time of the operation sequence of the current flexible fabric and improving its generation efficiency.

[0060] Secondly, since the operation sequence of the current flexible fabric is automatically generated, it is not affected by human intervention, which helps to improve the reliability of the operation sequence of the current flexible fabric. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is an application scenario diagram of the flexible fabric operation strategy optimization method provided in the embodiments of this application;

[0063] Figure 2 This is a flowchart illustrating the flexible fabric operation strategy optimization method provided in the embodiments of this application;

[0064] Figure 3 A flowchart illustrating the implementation of S205 provided in this application embodiment;

[0065] Figure 4 A schematic block diagram of a flexible fabric operation strategy optimization device provided in an embodiment of this application;

[0066] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0068] The flexible fabric operation strategy optimization method provided in this application embodiment can be applied to electronic devices, including but not limited to servers, mobile phones, tablets, wearable devices, vehicle-mounted devices, and laptops. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0069] Please see Figure 1 , Figure 1 The application scenario diagram of the flexible fabric operation strategy optimization method provided in the embodiments of this application is described in detail below:

[0070] The application scenario diagram includes electronic devices and robot end effectors, and the electronic devices and robot end effectors establish data transmission channels;

[0071] The electronic device acquires the captured image of the preset flexible fabric and the language command of the preset flexible fabric. It splices the feature vector of the captured image of the preset flexible fabric, the feature vector of the language command of the preset flexible fabric, and the random noise vector to generate the spliced ​​feature vector of the preset flexible fabric. The spliced ​​feature vector of the preset flexible fabric is input into the student model to obtain the operation sequence of the preset flexible fabric output by the student model at the current time step.

[0072] The electronic device inputs the predicted operation sequence of the preset flexible fabric output by the student model at the current time step into the value evaluator. The value evaluator generates the return estimate of the student model. The VLA model is selected as the teacher model. The feature vector of the captured image of the preset flexible fabric and the feature vector of the language instruction of the preset flexible fabric are input into the teacher model. The operation sequence of the preset flexible fabric output by the teacher model at the current time step is obtained from the output information of the teacher model.

[0073] The electronic device generates the value loss of the student model through the return estimate and the first loss model. It generates the distillation loss of the student model based on the operation sequence of the preset flexible cloth output by the student model at the current time step, the operation sequence of the preset flexible cloth output by the teacher model at the current time step, and the second loss model. Finally, it generates the smoothing loss of the student model through the third loss model.

[0074] The electronic device generates the total loss of the student model based on the student model's value loss, distillation loss, smoothing loss, and total loss model. With reducing the total loss as the optimization objective, the device updates the student model through the optimizer until the total loss of the student model is less than the preset loss value. Only then does it stop updating the student model and save the updated student model.

[0075] The electronic device acquires the captured image of the current flexible fabric and the language command of the current flexible fabric. It concatenates the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate the concatenated feature vector of the current flexible fabric. The concatenated feature vector of the current flexible fabric is input into the updated student model. The updated student model generates the operation sequence of the current flexible fabric.

[0076] The electronic device sends the current operation sequence of the flexible fabric to the robot's end effector via multiple data transmission channels;

[0077] The robot end effector performs target operations on the current flexible fabric according to the current operation sequence. The target operations include grasping, flattening and folding operations.

[0078] In this embodiment, the electronic device sends the operation sequence of the current flexible fabric to the robot end effector through multiple data transmission channels. The robot end effector performs the target operation on the current flexible fabric according to the operation sequence, which can effectively reduce the operation time of the current flexible fabric and improve the operation efficiency of the current flexible fabric.

[0079] Please see Figure 2 , Figure 2 This is a flowchart illustrating the flexible fabric operation strategy optimization method provided in this application embodiment, which can be applied to electronic devices.

[0080] like Figure 2 As shown in the embodiments of this application, the method for optimizing the operation strategy of flexible fabric includes the following steps, which are detailed below:

[0081] S201, acquire the captured image of the preset flexible fabric and the language command of the preset flexible fabric, concatenate the feature vector of the captured image of the preset flexible fabric, the feature vector of the language command of the preset flexible fabric and the random noise vector to generate the concatenated feature vector of the preset flexible fabric, input the concatenated feature vector of the preset flexible fabric into the student model, and acquire the operation sequence of the preset flexible fabric output by the student model at the current time step.

[0082] The process involves acquiring a captured image of the preset flexible fabric and its associated language commands. The feature vectors of the captured image, the language commands, and random noise vectors are then concatenated to generate a concatenated feature vector of the preset flexible fabric. This concatenated feature vector is then input into a student model to obtain the operation sequence of the preset flexible fabric output by the student model at the current time step, including:

[0083] Obtain samples from the dataset, extract images of the preset flexible fabric and language instructions of the preset flexible fabric from the samples, use the image feature extraction module to extract features from the images of the preset flexible fabric to generate feature vectors of the images of the preset flexible fabric, and use the semantic encoding module to semantically encode the language instructions of the preset flexible fabric to generate feature vectors of the language instructions of the preset flexible fabric.

[0084] Obtain random noise vectors from the standard normal distribution, and concatenate the feature vectors of the captured image of the preset flexible fabric, the feature vectors of the language instructions of the preset flexible fabric, and the random noise vectors to generate the concatenated feature vector of the preset flexible fabric. Input the concatenated feature vector of the preset flexible fabric into the student model, obtain the output information of the student model, and obtain the operation sequence of the preset flexible fabric output by the student model at the current time step from the output information of the student model.

[0085] S202, input the predicted operation sequence of the preset flexible fabric output by the student model at the current time step into the value evaluator, generate the return estimate of the student model through the value evaluator, select the VLA model as the teacher model, input the feature vector of the captured image of the preset flexible fabric and the feature vector of the language instruction of the preset flexible fabric into the teacher model, and obtain the operation sequence of the preset flexible fabric output by the teacher model at the current time step from the output information of the teacher model.

[0086] The full Chinese name of the VLA model is: Vision-Language-Action Model, and the full English name of the VLA model is Vision-Language-Action Mode.

[0087] Among them, the VLA model, or Visual-Language-Action model, is a multimodal model that integrates visual perception, language understanding, and action decision-making.

[0088] S203, generate the value loss of the student model through the return estimate and the first loss model, generate the distillation loss of the student model based on the operation sequence of the preset flexible cloth output by the student model at the current time step, the operation sequence of the preset flexible cloth output by the teacher model at the current time step and the second loss model, and generate the smoothing loss of the student model through the third loss model.

[0089] The first loss model is defined as follows:

[0090] ;

[0091] This represents the value loss of the student model. The greater the value loss of the student model, the weaker the performance of the preset flexible cloth operation sequence output by the student model at the current time step. The smaller the value loss of the student model, the stronger the performance of the preset flexible cloth operation sequence output by the student model at the current time step.

[0092] Indicates the learning parameters;

[0093] Indicates A value evaluator for learning parameters;

[0094] This indicates the current state information of the preset flexible fabric, which includes the current position and shape data of the preset flexible fabric.

[0095] This represents the preset flexible cloth operation sequence output by the student model at the current time step;

[0096] Language commands indicating preset flexible fabric;

[0097] This represents the estimated return value of the student model generated by the value evaluator based on the current state of the preset flexible fabric and the operation sequence of the preset flexible fabric output by the student model at the current time step.

[0098] Based on the return estimates of the student model, the rationality, reliability, and quality of the student model can be quantified. The higher the return estimate, the smaller the value loss of the student model; the lower the return estimate, the greater the value loss of the student model.

[0099] The second loss model is defined as follows:

[0100] ;

[0101] This represents the distillation loss of the student model;

[0102] This represents the preset flexible cloth operation sequence output by the student model at the current time step;

[0103] This represents the preset flexible fabric operation sequence output by the teacher model at the current time step;

[0104] express and The square of the L2 norm of the difference between them.

[0105] The third loss model is defined as follows:

[0106] ;

[0107] This represents the smoothing loss of the student model;

[0108] This represents the preset flexible cloth operation sequence output by the student model at the current time step;

[0109] This represents the preset flexible cloth operation sequence output by the student model in the previous time step at the current time step;

[0110] express and The square of the L2 norm of the difference between them.

[0111] Using the VLA model as the teacher model for knowledge transfer to the student model allows for the efficient transfer of the visual-language-action joint representation capabilities, complex scene understanding, and end-to-end decision-making knowledge integrated in the teacher model to the lightweight student model. During this knowledge transfer process, the student model inherits prior knowledge learned from the teacher model, such as multimodal alignment, semantic understanding, and action planning. While maintaining its small size and fast inference speed, it significantly improves perception accuracy, decision reliability, and generalization ability, thereby achieving efficient deployment and stable operation for practical tasks such as flexible cloth manipulation without significantly increasing computational overhead.

[0112] S204. Based on the student model's value loss, student model's distillation loss, student model's smoothing loss, and total loss model, generate the student model's total loss. With reducing the total loss as the optimization objective, update the student model through the optimizer until the student model's total loss is less than the preset loss value, then stop updating the student model and save the updated student model.

[0113] The total loss model is defined as follows:

[0114] ;

[0115] This represents the total loss of the student model; the higher the total loss of the student model, the weaker the overall performance of the student model in terms of value, distillation, and smoothing; the lower the total loss of the student model, the stronger the overall performance of the student model in terms of value, distillation, and smoothing.

[0116] Weight parameters representing the value loss of the student model;

[0117] This represents the value loss of the student model;

[0118] The weight parameters represent the distillation loss of the student model;

[0119] This represents the distillation loss of the student model;

[0120] The weight parameters represent the smoothing loss of the student model;

[0121] This represents the smoothing loss of the student model.

[0122] S205: Obtain the captured image of the current flexible fabric and the language command of the current flexible fabric; concatenate the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate the concatenated feature vector of the current flexible fabric; input the concatenated feature vector of the current flexible fabric into the updated student model; and generate the operation sequence of the current flexible fabric through the updated student model.

[0123] Among them, the preset flexible fabric is a pre-set flexible fabric.

[0124] Among them, the current flexible fabric is the flexible fabric to be processed.

[0125] Among them, by generating the operation sequence of the current flexible fabric through the updated student model, the computational load and memory usage of the model can be greatly reduced while ensuring the accuracy of operation and the effectiveness of decision-making, thus achieving lightweight and low-latency inference.

[0126] For ease of explanation, the following example is provided:

[0127] For example, student models can be efficiently deployed on hardware platforms with limited computing power, such as embedded devices and edge computing modules, to quickly complete the grasping, flattening, and folding operations of the current flexible fabric, effectively improving the real-time response speed and operating efficiency of the system, while reducing equipment costs and energy consumption, making them more suitable for long-term stable operation in actual automated production scenarios.

[0128] The method further includes, after acquiring the captured image of the current flexible fabric and the language command of the current flexible fabric, concatenating the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate a concatenated feature vector of the current flexible fabric, and inputting the concatenated feature vector of the current flexible fabric into the updated student model, and generating the operation sequence of the current flexible fabric through the updated student model:

[0129] According to the current operation sequence of the flexible fabric, control the robot's end effector to perform target operations on the current flexible fabric. The target operations include grasping, flattening and folding operations.

[0130] Among them, the robot end effector is the robot's execution device.

[0131] The gripping operation is used to locate, hold, and move the current flexible fabric, and is a prerequisite for all subsequent operations.

[0132] The purpose of the flattening operation is to eliminate fabric wrinkles, keep the current flexible fabric flat, and ensure the accuracy of subsequent processing.

[0133] The function of the folding operation is to straighten the current flexible fabric according to a preset shape, so as to achieve the shaping, storage or assembly of the current flexible fabric.

[0134] Controlling the robot's end effector to perform target operations on the current flexible fabric enables precise and controllable fabric operations, avoiding the uncertainties of manual operation.

[0135] For ease of explanation, the following example is provided:

[0136] In the current flexible fabric grasping operation, by controlling the robot's end effector, it can accurately fit the surface of the flexible fabric and grasp it smoothly. This avoids problems such as slippage, tearing and deformation of the flexible fabric caused by improper force control during manual grasping, ensuring that the flexible fabric remains intact during the grasping process and laying a good foundation for subsequent operations such as flattening and folding.

[0137] The beneficial effects of the embodiments of this application are as follows:

[0138] Firstly, the system acquires the captured image of the current flexible fabric and the language command of the current flexible fabric. It then concatenates the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate a concatenated feature vector of the current flexible fabric. This concatenated feature vector of the current flexible fabric is then input into the updated student model. Through the updated student model, the system generates the operation sequence of the current flexible fabric. This process eliminates the need for manual operation, thus reducing the generation time of the operation sequence of the current flexible fabric and improving its generation efficiency.

[0139] Secondly, since the operation sequence of the current flexible fabric is automatically generated, it is not affected by human intervention, which helps to improve the reliability of the operation sequence of the current flexible fabric.

[0140] Please see Figure 3 , Figure 3The implementation flowchart of S205 provided in the embodiments of this application is described in detail below:

[0141] S301, acquire the captured image of the current flexible fabric and the language command of the current flexible fabric, use the image feature extraction module to extract features from the captured image of the current flexible fabric to generate the feature vector of the captured image of the current flexible fabric, and use the semantic encoding module to semantically encode the language command of the current flexible fabric to generate the feature vector of the language command of the current flexible fabric.

[0142] S302, the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric are spliced ​​together to generate the spliced ​​feature vector of the current flexible fabric. The spliced ​​feature vector of the current flexible fabric is input into the updated student model. The updated student model is used to generate the operation sequence of the current flexible fabric.

[0143] In this embodiment, the feature vector of the captured image of the current flexible fabric and the feature vector of the language instruction of the current flexible fabric are spliced ​​together to generate the spliced ​​feature vector of the current flexible fabric. The spliced ​​feature vector of the current flexible fabric retains the spatial structure and visual features of the image, and also contains the semantic information and abstract description of the language text, which can significantly improve the updated student model's ability to understand the current flexible fabric.

[0144] For the flexible fabric operation strategy optimization method described in the above embodiments, please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic block diagram of the flexible fabric operation strategy optimization device provided in the embodiments of this application. Figure 4 The flexible fabric handling strategy optimization device 400 shown can be applied to, for example... Figure 1 The application scenario diagram shows electronic devices. The following section uses electronic devices as an example to illustrate this. Figure 4 The flexible fabric operation strategy optimization device 400 shown will be described in detail. The flexible fabric operation strategy optimization device 400 may include a first acquisition module 401, a second acquisition module 402, a first generation module 403, an update module 404, and a second generation module 405.

[0145] The first acquisition module 401 is used to acquire the captured image of the preset flexible fabric and the language instructions of the preset flexible fabric, and to splice the feature vector of the captured image of the preset flexible fabric, the feature vector of the language instructions of the preset flexible fabric, and the random noise vector to generate the spliced ​​feature vector of the preset flexible fabric. The spliced ​​feature vector of the preset flexible fabric is input into the student model to obtain the operation sequence of the preset flexible fabric output by the student model at the current time step.

[0146] The second acquisition module 402 is used to input the predicted operation sequence of the preset flexible fabric output by the student model at the current time step into the value evaluator, generate the return estimate of the student model through the value evaluator, select the VLA model as the teacher model, input the feature vector of the captured image of the preset flexible fabric and the feature vector of the language instruction of the preset flexible fabric into the teacher model, and obtain the operation sequence of the preset flexible fabric output by the teacher model at the current time step from the output information of the teacher model.

[0147] The first generation module 403 is used to generate the value loss of the student model through the return estimate and the first loss model, generate the distillation loss of the student model based on the operation sequence of the preset flexible cloth output by the student model at the current time step, the operation sequence of the preset flexible cloth output by the teacher model at the current time step and the second loss model, and generate the smoothing loss of the student model through the third loss model.

[0148] The update module 404 is used to generate the total loss of the student model based on the student model's value loss, distillation loss, smoothing loss, and total loss model. With reducing the total loss as the optimization objective, the student model is updated by the optimizer until the total loss of the student model is less than the preset loss value. Then the update of the student model is stopped and the updated student model is saved.

[0149] The second generation module 405 is used to acquire the captured image of the current flexible fabric and the language command of the current flexible fabric, and to concatenate the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate the concatenated feature vector of the current flexible fabric. The concatenated feature vector of the current flexible fabric is input into the updated student model, and the operation sequence of the current flexible fabric is generated through the updated student model.

[0150] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0151] The beneficial effects of the embodiments of this application are as follows:

[0152] Firstly, the system acquires the captured image of the current flexible fabric and the language command of the current flexible fabric. It then concatenates the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate a concatenated feature vector of the current flexible fabric. This concatenated feature vector of the current flexible fabric is then input into the updated student model. Through the updated student model, the system generates the operation sequence of the current flexible fabric. This process eliminates the need for manual operation, thus reducing the generation time of the operation sequence of the current flexible fabric and improving its generation efficiency.

[0153] Secondly, since the operation sequence of the current flexible fabric is automatically generated, it is not affected by human intervention, which helps to improve the reliability of the operation sequence of the current flexible fabric.

[0154] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0155] like Figure 5 As shown, Figure 5 The electronic device includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 executes the computer program 22 to implement the steps in any of the above method embodiments.

[0156] The electronic device may include, but is not limited to, processor 20 and memory 21. Those skilled in the art will understand that... Figure 5 This is merely an example of an electronic device and does not constitute a limitation on electronic devices. It may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0157] The processor 20 is used to run a computer program 22 stored in the memory 21, and performs the following steps when executing the computer program 22:

[0158] The system acquires the captured image of the preset flexible fabric and the language command of the preset flexible fabric. It then concatenates the feature vector of the captured image of the preset flexible fabric, the feature vector of the language command of the preset flexible fabric, and the random noise vector to generate the concatenated feature vector of the preset flexible fabric. Finally, it inputs the concatenated feature vector of the preset flexible fabric into the student model to obtain the operation sequence of the preset flexible fabric output by the student model at the current time step.

[0159] The predicted operation sequence of the preset flexible fabric output by the student model at the current time step is input into the value evaluator. The value evaluator generates the return estimate of the student model. The VLA model is selected as the teacher model. The feature vector of the captured image of the preset flexible fabric and the feature vector of the language instruction of the preset flexible fabric are input into the teacher model. The operation sequence of the preset flexible fabric output by the teacher model at the current time step is obtained from the output information of the teacher model.

[0160] The value loss of the student model is generated by the return estimate and the first loss model. The distillation loss of the student model is generated by the operation sequence of the preset flexible cloth output by the student model at the current time step, the operation sequence of the preset flexible cloth output by the teacher model at the current time step, and the second loss model. The smoothing loss of the student model is generated by the third loss model.

[0161] Based on the student model's value loss, distillation loss, smoothing loss, and total loss model, the total loss of the student model is generated. With reducing the total loss as the optimization objective, the student model is updated by the optimizer until the total loss of the student model is less than the preset loss value. Only then is the student model updated and the updated student model saved.

[0162] Acquire the captured image of the current flexible fabric and the language command of the current flexible fabric. Concatenate the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate the concatenated feature vector of the current flexible fabric. Input the concatenated feature vector of the current flexible fabric into the updated student model. Through the updated student model, generate the operation sequence of the current flexible fabric.

[0163] The processor 20 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors, field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0164] In some embodiments, the memory 21 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory 21 may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the electronic device.

[0165] Furthermore, the memory 21 may include both internal storage units and external storage devices of the electronic device. The memory 21 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 21 can also be used to temporarily store data that has been output or will be output.

[0166] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0167] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0168] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for optimizing the handling strategy of flexible fabric, characterized in that, The flexible fabric operation strategy optimization method, applied to electronic devices, includes: The system acquires the captured image of the preset flexible fabric and the language command of the preset flexible fabric. It then concatenates the feature vector of the captured image of the preset flexible fabric, the feature vector of the language command of the preset flexible fabric, and the random noise vector to generate the concatenated feature vector of the preset flexible fabric. Finally, it inputs the concatenated feature vector of the preset flexible fabric into the student model to obtain the operation sequence of the preset flexible fabric output by the student model at the current time step. The predicted operation sequence of the preset flexible fabric output by the student model at the current time step is input into the value evaluator. The value evaluator generates the return estimate of the student model. The VLA model is selected as the teacher model. The feature vector of the captured image of the preset flexible fabric and the feature vector of the language instruction of the preset flexible fabric are input into the teacher model. The operation sequence of the preset flexible fabric output by the teacher model at the current time step is obtained from the output information of the teacher model. The value loss of the student model is generated by the return estimate and the first loss model. The distillation loss of the student model is generated by the operation sequence of the preset flexible cloth output by the student model at the current time step, the operation sequence of the preset flexible cloth output by the teacher model at the current time step, and the second loss model. The smoothing loss of the student model is generated by the third loss model. Based on the student model's value loss, distillation loss, smoothing loss, and total loss model, the total loss of the student model is generated. With reducing the total loss as the optimization objective, the student model is updated by the optimizer until the total loss of the student model is less than the preset loss value. Only then is the student model updated and the updated student model saved. Acquire the captured image of the current flexible fabric and the language command of the current flexible fabric. Concatenate the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate the concatenated feature vector of the current flexible fabric. Input the concatenated feature vector of the current flexible fabric into the updated student model. Through the updated student model, generate the operation sequence of the current flexible fabric.

2. The method for optimizing the operation strategy of flexible fabric according to claim 1, characterized in that, The process involves acquiring a captured image of the preset flexible fabric and the language instructions for the preset flexible fabric. The feature vectors of the captured image, the language instructions, and random noise vectors are then concatenated to generate a concatenated feature vector of the preset flexible fabric. This concatenated feature vector is then input into the student model to obtain the operation sequence of the preset flexible fabric output by the student model at the current time step, including: Obtain samples from the dataset, extract images of the preset flexible fabric and language instructions of the preset flexible fabric from the samples, use the image feature extraction module to extract features from the images of the preset flexible fabric to generate feature vectors of the images of the preset flexible fabric, and use the semantic encoding module to semantically encode the language instructions of the preset flexible fabric to generate feature vectors of the language instructions of the preset flexible fabric. Obtain random noise vectors from the standard normal distribution, and concatenate the feature vectors of the captured image of the preset flexible fabric, the feature vectors of the language instructions of the preset flexible fabric, and the random noise vectors to generate the concatenated feature vector of the preset flexible fabric. Input the concatenated feature vector of the preset flexible fabric into the student model, obtain the output information of the student model, and obtain the operation sequence of the preset flexible fabric output by the student model at the current time step from the output information of the student model.

3. The method for optimizing the operation strategy of flexible fabric according to claim 1, characterized in that, The process involves acquiring a captured image of the current flexible fabric and the language commands associated with it. The feature vectors of the captured image and the language commands are then concatenated to generate a concatenated feature vector for the current flexible fabric. This concatenated feature vector is then input into an updated student model. The updated student model then generates an operation sequence for the current flexible fabric, including: The system acquires the captured image of the current flexible fabric and the language command of the current flexible fabric. The image feature extraction module extracts features from the captured image of the current flexible fabric to generate a feature vector of the captured image of the current flexible fabric. The semantic encoding module performs semantic encoding on the language command of the current flexible fabric to generate a feature vector of the language command of the current flexible fabric. The feature vectors of the captured image of the current flexible fabric and the feature vectors of the language commands of the current flexible fabric are concatenated to generate the concatenated feature vector of the current flexible fabric. The concatenated feature vector of the current flexible fabric is then input into the updated student model, and the operation sequence of the current flexible fabric is generated through the updated student model.

4. The method for optimizing the operation strategy of flexible fabric according to claim 1, characterized in that, After acquiring the captured image of the current flexible fabric and the language command of the current flexible fabric, concatenating the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate the concatenated feature vector of the current flexible fabric, and inputting the concatenated feature vector of the current flexible fabric into the updated student model, and generating the operation sequence of the current flexible fabric through the updated student model, the method further includes: According to the current operation sequence of the flexible fabric, control the robot's end effector to perform target operations on the current flexible fabric. The target operations include grasping, flattening and folding operations.

5. The method for optimizing the operation strategy of flexible fabric according to claim 1, characterized in that, The first loss model is defined as follows: ； This represents the value loss of the student model. The greater the value loss of the student model, the weaker the performance of the preset flexible cloth operation sequence output by the student model at the current time step. The smaller the value loss of the student model, the stronger the performance of the preset flexible cloth operation sequence output by the student model at the current time step. Indicates the learning parameters; Indicates A value evaluator for learning parameters; This indicates the current state information of the preset flexible fabric, which includes the current position and shape data of the preset flexible fabric. This represents the preset flexible cloth operation sequence output by the student model at the current time step; Language commands indicating preset flexible fabric; This represents the estimated return value of the student model generated by the value evaluator based on the current state of the preset flexible fabric and the operation sequence of the preset flexible fabric output by the student model at the current time step.

6. The method for optimizing the operation strategy of flexible fabric according to claim 1, characterized in that, The second loss model is defined as follows: ； This represents the distillation loss of the student model; This represents the preset flexible cloth operation sequence output by the student model at the current time step; This represents the preset flexible fabric operation sequence output by the teacher model at the current time step; express and The square of the L2 norm of the difference between them.

7. The method for optimizing the operation strategy of flexible fabric according to claim 1, characterized in that, The third loss model is defined as follows: ； This represents the smoothing loss of the student model; This represents the preset flexible cloth operation sequence output by the student model at the current time step; This represents the preset flexible cloth operation sequence output by the student model in the previous time step at the current time step; express and The square of the L2 norm of the difference between them; The total loss model is defined as follows: ； This represents the total loss of the student model; the higher the total loss of the student model, the weaker the overall performance of the student model in terms of value, distillation, and smoothing; the lower the total loss of the student model, the stronger the overall performance of the student model in terms of value, distillation, and smoothing. Weight parameters representing the value loss of the student model; This represents the value loss of the student model; The weight parameters represent the distillation loss of the student model; This represents the distillation loss of the student model; The weight parameters represent the smoothing loss of the student model; This represents the smoothing loss of the student model.

8. A flexible fabric handling strategy optimization device, characterized in that, Applied to electronic devices, including: The first acquisition module is used to acquire the captured image of the preset flexible fabric and the language instructions of the preset flexible fabric. It splices the feature vector of the captured image of the preset flexible fabric, the feature vector of the language instructions of the preset flexible fabric, and the random noise vector to generate the spliced ​​feature vector of the preset flexible fabric. It inputs the spliced ​​feature vector of the preset flexible fabric into the student model to obtain the operation sequence of the preset flexible fabric output by the student model at the current time step. The second acquisition module is used to input the predicted operation sequence of the preset flexible fabric output by the student model at the current time step into the value evaluator, generate the return estimate of the student model through the value evaluator, select the VLA model as the teacher model, input the feature vector of the captured image of the preset flexible fabric and the feature vector of the language instruction of the preset flexible fabric into the teacher model, and obtain the operation sequence of the preset flexible fabric output by the teacher model at the current time step from the output information of the teacher model. The first generation module is used to generate the value loss of the student model through the return estimate and the first loss model. Based on the operation sequence of the preset flexible cloth output by the student model at the current time step, the operation sequence of the preset flexible cloth output by the teacher model at the current time step, and the second loss model, the distillation loss of the student model is generated. The smoothing loss of the student model is generated through the third loss model. The update module is used to generate the total loss of the student model based on the student model's value loss, distillation loss, smoothing loss, and total loss model. With the optimization objective of reducing the total loss, the student model is updated by the optimizer until the total loss of the student model is less than the preset loss value. Then the update of the student model stops and the updated student model is saved. The second generation module is used to acquire the captured image of the current flexible fabric and the language command of the current flexible fabric. It concatenates the feature vector of the captured image of the current flexible fabric and the feature vector of the language command of the current flexible fabric to generate the concatenated feature vector of the current flexible fabric. The concatenated feature vector of the current flexible fabric is input into the updated student model. The updated student model generates the operation sequence of the current flexible fabric.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the flexible fabric operation strategy optimization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the flexible fabric operation strategy optimization method as described in any one of claims 1 to 7.

Images