Deformable object manipulation method and device based on vision and language model

Abstract

The invention relates to a deformable object manipulation method and device based on a vision and language model, and the method comprises the steps: building an operation environment for a deformable object, and the operation environment comprises a camera used for collecting the visual data of the deformable object and a mechanical arm used for manipulating the deformable object; performing visual information capture on the deformable object in the operation environment to obtain image data, and recording a corresponding language instruction; extracting a key action and a target object through a language model according to the language instruction, and generating an operation instruction; extracting spatial features according to the image data through a visual processing model; the spatial features and the operation instructions are fused, and a final operation strategy is generated through a machine learning algorithm; the operation strategy is converted into an execution instruction of the mechanical arm so as to control the deformable object; compared with the prior art, the precision and efficiency of the robot for processing the flexible object are remarkably improved, and effective technical support is provided for application in the fields of intelligent manufacturing, home furnishing, service and the like.

Description

Technical Field

[0001] The present invention relates to the field of deformable object manipulation technology, and in particular to a deformable object manipulation method and device based on vision and language models. Background Art

[0002] In the field of intelligent robotics and automation technology, the demand for handling cloth, ropes and other flexible materials continues to grow, especially in industries such as clothing manufacturing, medical, home furnishing and service industries. These materials, due to their softness and easy deformation, pose great challenges to traditional automation and robotics. Conventional robotic systems are usually designed to handle hard, fixed-shape objects, and they rely on precise manipulators and predetermined programs to perform tasks. However, the uncertain shape and changeable physical properties of flexible objects require robotic systems to have more advanced perception and adaptation capabilities to achieve high-precision operations.

[0003] Traditional methods usually involve using machine vision systems to identify the position and orientation of objects, but these systems often have difficulty accurately identifying and processing the complexity of visual information caused by irregular morphology. For example, when automatically folding clothes on a clothing production line or flattening gauze during surgery, the robot needs to be able to understand the specific state and needs of the object in order to adjust its operation strategy. In addition, robot operations in the prior art mostly rely on pre-programmed actions and lack the ability to flexibly respond to real-time changes. In addition, although natural language processing technology has been introduced into robot control systems to provide a more intuitive way to input commands, most existing systems have failed to effectively integrate language commands with machine vision data. This results in robots still showing limitations in understanding and execution when performing tasks that contain complex language descriptions. For example, when the operation instruction involves "precisely folding a corner on an irregularly shaped fabric", it is often difficult to complete precise operations relying solely on language or visual information.

[0004] Therefore, there is an urgent need to develop a new robotic manipulation technology that can effectively combine advanced visual processing capabilities and deep language understanding to improve the accuracy and efficiency of flexible object handling. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method and device for manipulating a deformable object based on a vision and language model, so as to improve the accuracy and efficiency of handling flexible objects.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for manipulating a deformable object based on a vision and language model comprises the following steps:

[0008] Building an operating environment for the deformable object to be manipulated, the operating environment comprising a camera for collecting visual data of the deformable object and a robotic arm for manipulating the deformable object;

[0009] Capture visual information of deformable objects in an operating environment, obtain image data, and record corresponding language instructions;

[0010] Extract key actions and target objects according to the language instructions through a language model to generate operation instructions; extract spatial features according to the image data through a visual processing model; fuse the spatial features and the operation instructions to generate a final operation strategy through a machine learning algorithm;

[0011] Convert the operation strategy into execution instructions of the robot arm to manipulate the deformable object;

[0012] The training process of the language model, visual processing model and machine learning algorithm includes:

[0013] Simulate the deformable object to simulate the movement and deformation of the deformable object and generate simulation training data; capture visual information of the deformable object in the real environment and record the corresponding language instructions to generate real training data; combine the simulation training data with the real training data to form a sample set including image data and language instructions; train the language model, visual processing model and machine learning algorithm according to the sample set.

[0014] Furthermore, the operating environment further comprises an operating platform, the deformable object to be manipulated is located on the operating platform, and the number of the cameras is multiple, which are respectively located around the operating platform and are used to capture visual information of the deformable object at different angles;

[0015] The working space of the robotic arm covers the entire operating platform.

[0016] Furthermore, the language model is a CLIP language model, the input of which is a language instruction based on a natural language, and the output of which is the operation goal and steps of the task, so as to extract key actions and target objects.

[0017] Furthermore, the visual processing model is a Transformer network;

[0018] The camera is an RGB-D camera, the extracted image data includes an RGB image and a corresponding depth image, the input of the visual processing model is the RGB image and the corresponding depth image, and the spatial features of the object are output.

[0019] Furthermore, the machine learning algorithm is a deep reinforcement learning algorithm, and the generation expression of the operation strategy is:

[0020] S=f(F i , Task)

[0021] In the formula, S is the operation strategy, F i is the spatial feature, Task is the operation instruction, and f is the strategy generation function, which is optimized and adjusted based on the deep reinforcement learning algorithm.

[0022] Furthermore, the motion planning of the robot arm follows the following model:

[0023] q t+1 =q t +Δq

[0024] In the formula, q t is the joint angle vector of the robot at time t, Δq is the change in joint angle under the current operation strategy, and q t+1 is the joint angle vector of the robot at time t+2.

[0025] Furthermore, when the robotic arm is manipulating the deformable object, the execution process of the robotic arm is monitored in real time through a camera, and the operation strategy is dynamically adjusted according to the feedback.

[0026] Furthermore, the deformable object includes cloth and rope.

[0027] Further, the robotic arm performs picking, moving and / or placing operations based on an operation strategy.

[0028] The present invention also provides a deformable object manipulation device based on vision and language models, comprising:

[0029] A camera for collecting visual data of deformable objects;

[0030] A robotic arm, used to manipulate deformable objects;

[0031] The processor is connected to the camera and the robotic arm respectively, and is used to execute the steps of the method as described above.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] (1) The present invention integrates advanced visual processing technology and CLIP language model, uses visual processing technology to capture visual data of the operating environment, and analyzes the data through deep learning to identify the spatial position and morphological changes of objects; uses CLIP language model to parse natural language instructions of operating tasks and extract specific action requirements and operating goals; deeply integrates the recognized visual information with the parsed results of language instructions to generate accurate and detailed operating strategies, thereby significantly improving the robot's ability to perform picking, moving and placing tasks in complex environments, and improving the accuracy and efficiency of the robot in handling flexible objects, providing effective technical support for applications in the fields of smart manufacturing, home furnishings and services.

[0034] (2) The present invention simultaneously collects and processes data in the simulation environment and the real environment. By combining the simulation data with the data collected in the real environment, it can narrow the gap between the simulation and the real environment and improve the optimization performance of the visual processing and natural language processing models.

[0035] (3) The CLIP language model of the present invention can accurately understand and transform complex language expressions through multimodal deep learning capabilities; the Transformer used in the operation strategy generation r The network combines the local perception ability of the convolutional layer and the Transformer r The global context capture capability can effectively extract detailed spatial features of deformable objects (such as cloth) BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic flow chart of a method for manipulating a deformable object based on a vision and language model provided in an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of offline training and real prediction of a fabric provided in an embodiment of the present invention;

[0038] Figure 3 The present invention is a schematic diagram of an operating environment provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0040] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0042] Example 1

[0043] like Figure 1 As shown, this embodiment provides a method for manipulating a deformable object based on a vision and language model, comprising the following steps:

[0044] S1: Building an operating environment for the deformable object to be manipulated, the operating environment comprising a camera for collecting visual data of the deformable object and a robotic arm for manipulating the deformable object;

[0045] S2: Figure 2 As shown, a deformable object is simulated to simulate the movement and deformation of the deformable object to generate simulation training data; visual information of the deformable object in the real environment is captured, and corresponding language instructions are recorded to generate real training data; the simulation training data is combined with the real training data to form a sample set including image data and language instructions;

[0046] S3: Extract key actions and target objects based on language instructions through the language model and generate operation instructions;

[0047] S4: Extract spatial features from image data through visual processing models; fuse spatial features with operation instructions, and generate the final operation strategy through machine learning algorithms;

[0048] S5: Convert the operation strategy into execution instructions for the robot arm to manipulate the deformable object.

[0049] Specifically, in step S1, Figure 3 As shown, the operating environment also includes an operating platform, the deformable object to be manipulated is located on the operating platform, and there are multiple cameras, which are respectively located around the operating platform and are used to capture visual information of the deformable object at different angles;

[0050] The working space of the robot arm covers the entire operating platform.

[0051] In this embodiment, the construction of the operating environment includes installing two RGB-D cameras, one on each side of the operating platform, to capture visual data of the operating scene from different angles and improve the accuracy and reliability of visual recognition. At least one precision robotic arm is installed to complete complex object manipulation tasks; and a 1m x 1m operating platform is used to provide sufficient working space to ensure the flexibility and efficiency of the robotic arm operation. The hardware configuration of the operating environment ensures that the system can capture high-quality visual data and provide precise robotic arm control. The image data captured by the RGB-D camera can be expressed as:

[0052] (I i , D i ), i=1,2,...,N

[0053] Among them, I i is an RGB image, D i is the corresponding depth image.

[0054] The software configuration includes installing an operating system that supports the basic operation of the robot system, deploying a visual processing program to process image data collected from the RGB-D camera to identify the shape and position of objects; deploying the CLIP language model to parse natural language instructions and convert them into specific action instructions that the robotic arm can execute; and tools for collecting and analyzing operation process data to support real-time feedback and subsequent optimization of the system.

[0055] In step S2, in a simulation environment, PyFlex software is used to simulate the physical properties of cloth and other deformable objects, collect language instructions and visual data, and generate high-quality training data. These data are combined with data collected in the real environment to narrow the gap between simulation and real environment (sim-to-real) and are used to optimize the performance of visual processing and natural language processing models. In the real environment, the RGB-D camera system captures the visual information of the operation scene and records the corresponding language instructions. The collected data set is processed to form a sample set The sample set contains image data (I i , D i ) and the corresponding language instructions The corresponding expression is:

[0056]

[0057] in, is the corresponding natural language instruction.

[0058] In step S3, the language model is a CLIP language model, the input of the CLIP language model is a language instruction based on a natural language, and the output is the operation goal and steps of the task to extract key actions and target objects.

[0059] That is, the CLIP language model uses multimodal understanding capabilities to convert natural language instructions Convert it into specific operational goals and steps, extract key actions and target objects, and ensure that the system can accurately parse complex natural language expressions and convert them into operational instructions that can be executed by the robot.

[0060] The CLIP model uses multimodal deep learning capabilities to accurately understand and transform complex language expressions, and can identify key actions and target objects in instructions (such as "grab", "move", "place", etc.), as well as their corresponding positions and attribute descriptions. Through the CLIP language model, the system can generate accurate operation strategies and optimize the accuracy and efficiency of task execution. In addition, the CLIP language model enhances the system's responsiveness to complex and ambiguous instructions, thereby significantly improving the flexibility and adaptability of robot operations. The application of this language processing technology has significantly improved the robot's performance in processing deformable object tasks, providing advanced language understanding support for intelligent robot technology.

[0061] That is, the input of the CLIP language model is natural language instructions The output is the specific operation goals and steps of the task:

[0062]

[0063] In step S4, the visual processing model uses a lightweight Transformer network (such as MobileViT) to process the RGB-D image and extract the spatial features F of the object from it. i :

[0064] F i =MobileViT(I i , D i )

[0065] These features are fused with the instruction results parsed by the CLIP language model to generate the final operation strategy S:

[0066] S=f(F i , Task)

[0067] Among them, f is the policy generation function, which is optimized and adjusted by the deep reinforcement learning algorithm.

[0068] The strategy generation function f is optimized and adjusted in combination with machine learning algorithms (such as deep reinforcement learning) to ensure the accuracy and effectiveness of the operation.

[0069] The Transformer network combines the local perception ability of the convolutional layer with the global context capture ability of the Transformer, which can effectively extract detailed spatial features of deformable objects (such as cloth), including the object's position, shape, size, and possible deformation state. At the same time, the method also uses machine learning algorithms, especially deep reinforcement learning, to analyze and optimize the fused visual and language parsing results. These algorithms learn from historical operation data, automatically adjust and optimize operation parameters and strategies to perform complex operation tasks more accurately, and dynamically adjust operation strategies based on real-time feedback to ensure the efficiency and accuracy of operations.

[0070] In step S5, the robot arm is instructed to perform actions including picking, moving and placing according to the generated operation strategy. The operation strategy is converted into execution instructions of the robot arm through the controller module, and the motion planning of the robot arm follows the following model:

[0071] q t+1 =q t +Δq

[0072] Where qt is the joint angle vector of the robot at time t, Δq is the change in joint angle under the current operation strategy, and q t+1 is the joint angle vector of the robot at time t+2.

[0073] Preferably, the system monitors the execution process in real time and dynamically adjusts the operation strategy based on feedback to ensure efficient operation results.

[0074] This step ensures that the robot arm can accurately follow the strategy generated based on the analysis of visual data and the analysis results of the CLIP language model to perform tasks involving complex spatial positions and object shape adjustments. The system monitors the execution process in real time and adjusts the robot operation in time according to the feedback to adapt to the real-time state changes of the object, thereby achieving high-precision operation results.

[0075] The above is an introduction to a method embodiment. The following is a further explanation of the present invention through an apparatus embodiment.

[0076] The present invention also provides a deformable object manipulation device based on vision and language models, comprising:

[0077] A camera, used to collect visual data of the deformable object, preferably an RGB-D camera;

[0078] A robotic arm, used to manipulate deformable objects;

[0079] The processor is connected to the camera and the robotic arm respectively, and is used to execute the steps of the deformable object manipulation method based on vision and language models as described above.

[0080] It should be noted that the specific content and beneficial effects of the device of the present application can be found in the above-mentioned method embodiment, which will not be repeated here.

[0081] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for manipulating deformable objects based on vision and language models, characterized in that: The following steps are involved: Building an operating environment for the deformable object to be manipulated, the operating environment comprising a camera for collecting visual data of the deformable object and a robotic arm for manipulating the deformable object; Capture visual information of deformable objects in an operating environment, obtain image data, and record corresponding language instructions; Extracting key actions and target objects according to the language instructions through a language model and generating operation instructions; Extracting spatial features from the image data using a visual processing model; fusing the spatial features with the operation instructions, and generating a final operation strategy using a machine learning algorithm; Convert the operation strategy into execution instructions of the robot arm to manipulate the deformable object; The training process of the language model, visual processing model and machine learning algorithm includes: Simulate the deformable object, simulate the movement and deformation of the deformable object, and generate simulation training data; Capture visual information of deformable objects in a real environment, and record corresponding language instructions to generate real training data; combine the simulated training data with the real training data to form a sample set including image data and language instructions; train the language model, visual processing model and machine learning algorithm based on the sample set.

2. The method for manipulating a deformable object based on a vision and language model according to claim 1, characterized in that: The operating environment further includes an operating platform, the deformable object to be manipulated is located on the operating platform, and there are multiple cameras, which are respectively located around the operating platform and are used to capture visual information of the deformable object at different angles; The working space of the robotic arm covers the entire operating platform.

3. The method for manipulating a deformable object based on a vision and language model according to claim 1, characterized in that: The language model is a CLIP language model, the input of which is a language instruction based on a natural language, and the output of which is the operation goal and steps of the task, so as to extract key actions and target objects.

4. The method for manipulating a deformable object based on a vision and language model according to claim 1, characterized in that: The visual processing model is a Transformer network; The camera is an RGB-D camera, the extracted image data includes an RGB image and a corresponding depth image, the input of the visual processing model is the RGB image and the corresponding depth image, and the spatial features of the object are output.

5. The method for manipulating a deformable object based on a vision and language model according to claim 1, characterized in that: The machine learning algorithm is a deep reinforcement learning algorithm, and the generation expression of the operation strategy is: S=f(F i ,Task) In the formula, S is the operation strategy, F i is the spatial feature, Task is the operation instruction, and f is the strategy generation function, which is optimized and adjusted based on the deep reinforcement learning algorithm.

6. The method for manipulating a deformable object based on a vision and language model according to claim 1, characterized in that: The motion planning of the robot arm follows the following model: q t+1 =q t +Δq In the formula, q t is the joint angle vector of the robot at time t, Δq is the change in joint angle under the current operation strategy, and q t+1 is the joint angle vector of the robot at time t+2.

7. The method for manipulating a deformable object based on a vision and language model according to claim 6, characterized in that: When the robot arm is manipulating the deformable object, the camera monitors the execution process of the robot arm in real time, and dynamically adjusts the operation strategy according to the feedback.

8. The method for manipulating a deformable object based on a vision and language model according to claim 1, characterized in that: The deformable objects include cloth and ropes.

9. The method for manipulating a deformable object based on a vision and language model according to claim 1, characterized in that: The robotic arm performs pick, move and / or place operations based on an operation strategy.

10. A deformable object manipulation device based on vision and language model, characterized in that: include: A camera for collecting visual data of deformable objects; A robotic arm, used to manipulate deformable objects; The processor is connected to the camera and the robotic arm respectively, and is used to execute the steps of any method described in claims 1 to 9.

Images