Flexible object control strategy generation method based on multi-modal fusion

Abstract

The invention relates to a flexible object control strategy generation method based on multi-modal fusion, and the method comprises the steps: obtaining a grabbing point two-dimensional Gaussian distribution diagram, three-dimensional point cloud data and an RGB image of a target flexible object based on a human demonstration video; based on the three-dimensional point cloud data, obtaining a visibility connection diagram of the target flexible object; fusing the two-dimensional Gaussian distribution map and the RGB map of the grabbed point to obtain a fused map; respectively extracting a first feature and a second feature from the visibility connection graph and the fusion graph; fusing the first feature and the second feature to obtain a fused feature; obtaining an optimal grabbing point and an optimal placement point of the target flexible object based on the fusion features; and generating a control strategy of the target flexible object based on the optimal grabbing point and the optimal placement point. Compared with the prior art, the method has the advantages that the operation precision of the flexible object is improved, the adaptability to different states of the flexible object is high, and the difference from simulation to reality is reduced.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a method for generating a flexible object manipulation strategy based on multi-modal fusion. Background Art

[0002] In the field of intelligent robotics and computer vision, the manipulation of flexible objects has always been a challenging topic. Flexible objects, such as cloth, paper, and rope, have complex physical properties and dynamic behaviors, making them much more difficult to manipulate than rigid objects. Traditional robotic manipulation methods mainly rely on pre-set paths and force feedback mechanisms, which are difficult to deal with the uncertainty and deformation of flexible objects during manipulation.

[0003] Existing methods for manipulating flexible objects usually rely on model-based control and visual servo control. However, model-based control methods have limited practical application effects due to the difficulty in accurately modeling the complex behaviors of flexible objects. Although visual servo control methods can monitor the state of flexible objects in real time through visual sensors, data from a single modality cannot fully capture all the characteristics of flexible objects, especially when dealing with fabrics with arbitrary wrinkles.

[0004] In recent years, learning-based methods have shown great potential in the field of flexible object manipulation by learning complex manipulation tasks from large datasets. However, existing learning-based methods usually rely on large datasets and simulated environments to achieve good performance. In particular, flexible object manipulation is particularly sensitive to the sim-to-real gap, because accurately simulating flexible dynamics is a major challenge, and the implementation schemes in related technologies cannot achieve accurate simulation of flexible objects. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a flexible object manipulation strategy generation method based on multimodal fusion, which improves the accuracy of flexible object operation, has strong adaptability to different states of flexible objects, and reduces the gap between simulation and reality.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for generating a flexible object manipulation strategy based on multimodal fusion, the method comprising:

[0008] Based on human demonstration videos, obtain the 2D Gaussian distribution map, 3D point cloud data and RGB map of the target flexible object’s grasping points;

[0009] Based on the three-dimensional point cloud data, obtaining a visibility connection graph of the target flexible object;

[0010] Fusing the two-dimensional Gaussian distribution map of the grasping points and the RGB map to obtain a fused map;

[0011] Extracting a first feature and a second feature from the visibility connection graph and the fusion graph respectively;

[0012] fusing the first feature and the second feature to obtain a fused feature;

[0013] Based on the fusion features, obtaining the best grasping point and the best placement point of the target flexible object;

[0014] Based on the optimal grasping point and the optimal placement point, a manipulation strategy of the target flexible object is generated.

[0015] As a preferred technical solution, obtaining the two-dimensional Gaussian distribution map of the grab points includes:

[0016] Using a deep learning model to perform hand tracking and key point detection on the operator's hands in the human demonstration video to obtain the grasping points of the operator when operating the flexible object;

[0017] Based on the grasping points, a two-dimensional Gaussian distribution of the grasping points of the target flexible object is generated.

[0018] As a preferred technical solution, obtaining the three-dimensional point cloud data includes:

[0019] Based on the infrared image and the depth image obtained when collecting the human demonstration video, three-dimensional point cloud data of the target flexible object is constructed.

[0020] As a preferred technical solution, the method further includes:

[0021] Perform image preprocessing on the infrared image and the depth image; wherein the image preprocessing includes at least one of noise filtering, edge enhancement and dynamic contrast adjustment.

[0022] As a preferred technical solution, the step of obtaining a visibility connection graph of the target flexible object based on the three-dimensional point cloud data includes:

[0023] Based on the three-dimensional point cloud data, construct a graphical model of the target flexible object;

[0024] Based on the graph model, a visibility connection graph of the target flexible object is obtained.

[0025] As a preferred technical solution, the step of constructing the graphical model of the target flexible object includes:

[0026] Using the three-dimensional point cloud data as model input, a graphical model for representing the state of the target flexible object is constructed<V,E> ;

[0027] Among them, V is a node; E is an edge;

[0028] Node V = {v i} represents the particles constituting the target flexible object; i = 1, 2, ..., N, N is the number of particles; particle node x i represents the spatial position of the particle, represents the particle velocity vector;

[0029] Edge E includes mesh edge E M and adjacent edge E C , wherein the grid edge E M Represents the inherent connection relationship between particles in the flexible object grid structure, adjacent edge E C Represents the dynamic collision relationship between particles that are close in space.

[0030] As a preferred technical solution, the adjacent edge E C Based on the dynamic definition of the real-time Euclidean distance between particles, that is, at each time step t:

[0031]

[0032] in, represents the set of adjacent edges formed between particles at time step t; e ij Indicates that at time step t, there is a collision relationship between particle i and particle j; x i,t and x j,t They represent the spatial positions of particle i and particle j at time step t respectively; ||·||2 is the operation for calculating the Euclidean distance; and R is the preset distance threshold.

[0033] As a preferred technical solution, the step of constructing the graphical model of the target flexible object further includes:

[0034] Construct the mesh edge E M With the adjacent edge E C Constraints:

[0035] As a preferred technical solution, the step of obtaining a visibility connection graph of the target flexible object based on the three-dimensional point cloud data includes:

[0036] Performing voxel processing on the three-dimensional point cloud data to convert the three-dimensional point cloud data into a regular voxel grid array;

[0037] The visibility connection graph of the target flexible object is obtained by using the voxel grid array.

[0038] As a preferred technical solution, the fusing the first feature and the second feature to obtain a fused feature includes:

[0039] The first feature and the second feature are concatenated according to channels to obtain a fusion feature.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. Improve the accuracy of flexible object operation: The flexible object manipulation strategy generation method in the present invention synchronously captures and fuses multimodal data such as depth extraction, RGB images, infrared images and two-dimensional Gaussian distribution maps of grasping points. The obtained multimodal fusion features can more comprehensively characterize the state of the flexible object. Based on the fusion features, the optimal grasping defect and the optimal placement point of the flexible object can be more accurately predicted, which is conducive to improving the robot's operation accuracy on flexible objects in model learning tasks.

[0042] 2. Strong adaptability to different states of flexible objects: The flexible object manipulation strategy generation method in the present invention learns the complex gestures and operation details in human demonstration videos, so that the robot can realize the conversion of flexible objects from any wrinkled state to the target folded state under different flexible object states, and has strong adaptability.

[0043] 3. Reduce the gap between simulation and reality (sim-to-real): The flexible object manipulation strategy generation method in the present invention extracts multimodal data through human demonstration videos, allowing the robot to learn directly from the real environment, avoiding the gap between the simulated environment and the actual operation in the traditional method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a method for generating a flexible object manipulation strategy based on multimodal fusion in an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a process for constructing a graph model in an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0048] "One embodiment" or "embodiment" referred to herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present application. In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Moreover, the terms "first", "second", etc. are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0049] like Figure 1 As shown, an embodiment of the present application provides a method for generating a flexible object manipulation strategy based on multimodal fusion, the method comprising:

[0050] Step S110: based on the human demonstration video, obtaining a two-dimensional Gaussian distribution map, three-dimensional point cloud data and RGB map of the grasping points of the target flexible object;

[0051] Step S120: acquiring a visibility connection graph of the target flexible object based on the three-dimensional point cloud data;

[0052] Step S130: Fusing the two-dimensional Gaussian distribution map of the grasped points and the RGB map to obtain a fused map;

[0053] Step S140: extracting the first feature and the second feature from the visibility connection graph and the fusion graph respectively;

[0054] Step S150: fusing the first feature and the second feature to obtain a fused feature;

[0055] Step S160: based on the fusion features, obtaining the best grasping point and the best placement point of the target flexible object;

[0056] Step S170: Generate a manipulation strategy for the target flexible object based on the optimal grasping point and the optimal placement point.

[0057] The above steps S110 to S170 are described in detail below:

[0058] The human demonstration video in the above step S110 can be collected by a Realsense L515 camera. The Realsense L515 camera can also synchronously capture the depth map, RGB map and infrared map of the target flexible object when collecting the human demonstration video.

[0059] Optionally, the above step S110 obtains a two-dimensional Gaussian distribution map of the grasping points, including: using a deep learning model to track the hands of the operator in the human demonstration video and detect key points to obtain the grasping points of the operator when operating the flexible object; based on the grasping points, generating a two-dimensional Gaussian distribution of the grasping points of the target flexible object. This implementation example: using a deep learning model based on Yolov10 to track the hands of the operator in the human demonstration video and detect key points to accurately identify the dynamic gesture data of humans when operating flexible objects and record the grasping points and placement points in human operations.

[0060] It is understandable that the above Yolov10-based deep learning model can process video stream data in real time, identify and record dynamic gesture data when humans operate flexible objects, including the dynamic changes of finger positions, grasping actions and placing actions, thereby providing high-precision operator hand motion trajectory data. The acquisition of these data lays the foundation for subsequent multimodal data collection and processing.

[0061] In addition, the deep learning model based on Yolov10 has specially designed an adaptive algorithm, which can make real-time adjustments according to the hand characteristics and movement differences of different operators, thereby improving the recognition accuracy and stability in changing environments (such as different lighting conditions and background changes).

[0062] In addition, the Realsense L515 camera used in collecting human operation videos has high-precision depth perception capabilities and can obtain the three-dimensional depth information of the scene as well as infrared images and RGB images in real time. These images provide comprehensive information support for the state description of the target flexible object.

[0063] Optionally, the above step S110 of acquiring three-dimensional point cloud data includes: constructing three-dimensional point cloud data of the target flexible object based on the infrared image and depth image acquired when collecting the human demonstration video. This implementation example is:

[0064] The infrared image, depth image and the camera intrinsic parameters of the Realsense L515 camera are used to accurately reconstruct the three-dimensional point cloud of the target flexible object to represent the complex spatial structure of the target flexible object.

[0065] It can be understood that when constructing a three-dimensional point cloud, the above-mentioned infrared image can be used to accurately identify and segment the boundaries of flexible objects, and then reconstruct the three-dimensional point cloud based on the boundary information, depth map and camera intrinsic parameters to reconstruct a higher-precision three-dimensional point cloud.

[0066] Optionally, the above step S120 includes: performing voxel processing on the three-dimensional point cloud data to convert the three-dimensional point cloud data into a regular voxel grid array; and obtaining a visibility connection graph of the target flexible object using the voxel grid array.

[0067] It can be understood that the point cloud data is converted into a regular voxel grid array after voxelization, which simplifies the data structure and is conducive to improving the efficiency of subsequent calculations.

[0068] Optionally, the above-mentioned method for generating a flexible object manipulation strategy based on multimodal fusion also includes: performing image preprocessing on the infrared image and the depth image; wherein the image preprocessing includes at least one of noise filtering, edge enhancement and dynamic contrast adjustment.

[0069] The above-mentioned preprocessing methods such as noise filtering, edge enhancement and dynamic contrast adjustment are aimed at optimizing images, especially in low-light or high dynamic range environments, to ensure the highest accuracy in the segmentation and recognition of flexible object boundaries, thereby providing the clearest and most accurate image input for point cloud generation and flexible object modeling.

[0070] Optionally, the above step S120 includes: constructing a graph model of the target flexible object based on the three-dimensional point cloud data; and acquiring a visibility connection graph of the target flexible object based on the graph model.

[0071] Optionally, the above-mentioned graph model for constructing the target flexible object includes:

[0072] Using 3D point cloud data as model input, a graphical model is constructed to represent the state of the target flexible object.<V,E> ;

[0073] Among them, V is a node; E is an edge;

[0074] Node V = {v i} represents the particles that make up the target flexible object; i = 1, 2, ..., N, N is the number of particles; particle node x i represents the spatial position of the particle, represents the particle velocity vector;

[0075] Edge E includes mesh edge E M and adjacent edge E C , where the grid edge E M Represents the inherent connection relationship between particles in the grid structure of the flexible object. These connection relationships are determined by the structure of the flexible object and remain invariant over time. C Represents the dynamic collision relationships between particles that are close in space, which can change due to the folding or movement of flexible objects.

[0076] Optionally, the adjacent edge E C Based on the dynamic definition of the real-time Euclidean distance between particles, that is, at each time step t:

[0077]

[0078] in, represents the set of adjacent edges formed between particles at time step t; e ij Indicates that at time step t, there is a collision relationship between particle i and particle j; x i,t and x j,t They represent the spatial positions of particle i and particle j at time step t respectively; ||·||2 is the operation for calculating the Euclidean distance; and R is the preset distance threshold.

[0079] Optionally, the above-mentioned construction of the target flexible object graph model further includes: constructing a mesh edge E N With adjacent edge E C Constraints:

[0080] It is understandable that due to the mesh edge E M The nodes to be connected are physically close to each other, so set the constraint relationship To maintain the consistency of the model and satisfy the actual physical properties of flexible objects.

[0081] The construction process of the graph model is as follows Figure 2 As shown, the construction process may include the following steps:

[0082] (a) Obtaining 3D point cloud data, i.e., original point cloud;

[0083] (b) Convert the original point cloud into voxelized point cloud;

[0084] (c) Build a graph model based on voxelized point cloud to obtain graph data.

[0085] In the above scheme, the use of graph models can not only capture the structure and dynamic changes of flexible objects, but also help perceive the current state of self-occluded flexible objects and predict the interaction relationship between particles of flexible objects. The application of this technology significantly improves the understanding of the dynamic characteristics of flexible objects and the accuracy of operations, providing a powerful tool for intelligent robots to handle complex deformable objects.

[0086] The following is an introduction to the scheme of fusing the two-dimensional Gaussian distribution map of the grasped points and the RGB map in step S130:

[0087] The image fusion algorithm in the related technology can be used to fuse the two-dimensional Gaussian distribution map and the RGB map of the grasped points, such as the weighted fusion method, the interpolation fusion method, etc. Since the image fusion algorithm is a relatively mature technology in this field, it will not be repeated in the embodiments of this application.

[0088] The following is an introduction to the scheme of extracting the first feature and the second feature in the visibility connection graph and the fusion graph in step S140:

[0089] In an embodiment of the present application, step S140 can use UNet to extract the first feature and the second feature in the visibility connection graph and the fusion graph respectively. The above-mentioned UNet is a deep learning model, and its network architecture includes a feature extraction part located in the first half and an upsampling part located in the second half. In the feature extraction stage, UNet extracts the features of the image layer by layer through convolution and pooling operations; in the upsampling stage, UNet gradually restores the image size through deconvolution or up-pooling operations, and fuses the feature map extracted in the feature extraction stage with the feature map restored in the upsampling stage through skip connection.

[0090] Optionally, the above step S150 includes: splicing the first feature and the second feature by channel to obtain a fusion feature.

[0091] It can be understood that since the RGB image provides the texture and color information of the flexible object, the two-dimensional Gaussian distribution map of the grasping point provides the possible grasping point position information, and the three-dimensional point cloud data provides the three-dimensional collective morphological information of the flexible object, the fusion feature integrates the above information and can achieve a comprehensive perception and understanding of the state of the flexible object.

[0092] It is understandable that step S160 may use the above-mentioned UNet decoding module to obtain the best grasping point and the best placement point of the target flexible object based on the fusion features.

[0093] In addition, it is understood that after obtaining the best grasping point and the best placement point, step S170 can combine robot motion planning technology, path planning technology, and force control technology to jointly generate a manipulation strategy for the target flexible object, thereby achieving efficient conversion of the flexible object from any wrinkled state to the target folded state. The above-mentioned robot motion planning technology, path planning technology, and force control technology are all relatively mature technologies in the field, and therefore, the embodiments of this application will not be repeated.

[0094] Figure 3 Schematic block diagram of an electronic device that can be used to implement an embodiment of the present disclosure is shown. Figure 3 As shown, the electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0095] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0096] The processing unit performs the various methods and processes described above, such as steps S110 to S170 of the method of the present invention. For example, in some embodiments, steps S110 to S170 of the method of the present invention may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of steps S110 to S170 of the method of the present invention described above may be executed. Alternatively, in other embodiments, the CPU may be configured to execute steps S110 to S170 of the method of the present invention by any other appropriate means (e.g., by means of firmware).

[0097] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0098] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0099] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for generating a flexible object manipulation strategy based on multimodal fusion, characterized in that: The method comprises: Based on human demonstration videos, obtain the two-dimensional Gaussian distribution map, three-dimensional point cloud data and RGB map of the grasping point of the target flexible object; Based on the three-dimensional point cloud data, obtaining a visibility connection graph of the target flexible object; Fusing the two-dimensional Gaussian distribution map of the grasping points and the RGB map to obtain a fused map; Extracting a first feature and a second feature from the visibility connection graph and the fusion graph respectively; fusing the first feature and the second feature to obtain a fused feature; Based on the fusion features, obtaining the best grasping point and the best placement point of the target flexible object; Based on the optimal grasping point and the optimal placement point, a manipulation strategy of the target flexible object is generated.

2. The method for generating a flexible object manipulation strategy based on multimodal fusion according to claim 1, characterized in that: Obtaining a two-dimensional Gaussian distribution diagram of the grab points includes: Using a deep learning model to perform hand tracking and key point detection on the operator's hands in the human demonstration video to obtain the grasping points of the operator when operating the flexible object; Based on the grasping points, a two-dimensional Gaussian distribution of the grasping points of the target flexible object is generated.

3. The method for generating a flexible object manipulation strategy based on multimodal fusion according to claim 1, characterized in that: Acquiring the three-dimensional point cloud data includes: Based on the infrared image and the depth image obtained when collecting the human demonstration video, three-dimensional point cloud data of the target flexible object is constructed.

4. The method for generating a flexible object manipulation strategy based on multimodal fusion according to claim 3 is characterized in that: The method further comprises: Perform image preprocessing on the infrared image and the depth image; wherein the image preprocessing includes at least one of noise filtering, edge enhancement and dynamic contrast adjustment.

5. The method for generating a flexible object manipulation strategy based on multimodal fusion according to claim 1, characterized in that: The step of acquiring a visibility connection graph of the target flexible object based on the three-dimensional point cloud data includes: Based on the three-dimensional point cloud data, construct a graphical model of the target flexible object; Based on the graph model, a visibility connection graph of the target flexible object is obtained.

6. The method for generating a flexible object manipulation strategy based on multimodal fusion according to claim 5, characterized in that: The step of constructing a graphical model of the target flexible object includes: Using the three-dimensional point cloud data as model input, a graphical model for representing the state of the target flexible object is constructed<V,E> ; Among them, V is a node; E is an edge; Node V = {v i } represents the particles constituting the target flexible object; i = 1, 2, ..., N, N is the number of particles; particle node x i represents the spatial position of the particle, represents the particle velocity vector; Edge E includes mesh edge E M and adjacent edge E C , wherein the grid edge E M Represents the inherent connection relationship between particles in the flexible object grid structure, adjacent edge E C Represents the dynamic collision relationship between particles that are close in space.

7. The method for generating a flexible object manipulation strategy based on multimodal fusion according to claim 6, characterized in that: The adjacent edge E C Based on the dynamic definition of the real-time Euclidean distance between particles, that is, at each time step t: in, represents the set of adjacent edges formed between particles at time step t; e ij Indicates that at time step t, there is a collision relationship between particle i and particle j; x i,t and x j,t They represent the spatial positions of particle i and particle j at time step t respectively; ‖·‖2 is the operation for calculating the Euclidean distance; and R is the preset distance threshold.

8. The method for generating a flexible object manipulation strategy based on multimodal fusion according to claim 6, characterized in that: The step of constructing the graphical model of the target flexible object further includes: Construct the mesh edge E M With the adjacent edge E C Constraints:

9. The method for generating a flexible object manipulation strategy based on multimodal fusion according to any one of claims 1 to 8, characterized in that: The step of acquiring a visibility connection graph of the target flexible object based on the three-dimensional point cloud data includes: Performing voxel processing on the three-dimensional point cloud data to convert the three-dimensional point cloud data into a regular voxel grid array; The visibility connection graph of the target flexible object is obtained by using the voxel grid array.

10. The method for generating a flexible object manipulation strategy based on multimodal fusion according to any one of claims 1 to 8, characterized in that: The fusing the first feature and the second feature to obtain a fused feature includes: The first feature and the second feature are concatenated according to channels to obtain a fusion feature.

Images