A flexible object manipulation strategy generation method based on multi-modal fusion

Abstract

The present application relates to a kind of flexible object control strategy generation method based on multimodal fusion, comprising: based on human demonstration video, the acquisition target flexible object's grasping point two-dimensional Gaussian distribution diagram, three-dimensional point cloud data and RGB chart;Based on three-dimensional point cloud data, the acquisition target flexible object's visibility connection diagram;Fusion grasping point two-dimensional Gaussian distribution diagram and RGB chart, obtain fusion chart;In visibility connection diagram and fusion chart respectively extract first feature and second feature;Fusion first feature and second feature, obtain fusion feature;Based on fusion feature, the acquisition target flexible object's best grasping point and best placement point;Based on best grasping point and best placement point, generate the control strategy of target flexible object. Compared with prior art, the present application has the advantages of improving the precision of flexible object operation, strong adaptability to different states of flexible object and reducing the gap between simulation and reality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to a flexible object manipulation strategy generation method based on multi-modal fusion. BACKGROUND

[0002] In the field of intelligent robot technology and computer vision, the manipulation of flexible objects has always been a challenging topic. Flexible objects, such as cloth, paper, and ropes, have complex physical properties and dynamic behaviors, making their manipulation much more difficult than that of rigid objects. Traditional robot manipulation methods mainly rely on pre-set paths and force feedback mechanisms, which are difficult to cope with the uncertainty and deformation of flexible objects during operation.

[0003] Existing flexible object manipulation methods usually rely on model-based control and visual servoing control. However, model-based control methods have limited practical application due to the difficulty in accurately modeling the complex behavior of flexible objects. Visual servoing control methods can monitor the state of flexible objects in real time through visual sensors, but single-modal data cannot fully capture all the characteristics of flexible objects, especially when dealing with cloth with arbitrary wrinkle states.

[0004] In recent years, learning-based methods have shown great potential in the field of flexible object manipulation, which learn complex manipulation tasks from large data sets. However, existing learning-based methods usually rely on large data sets and simulation environments to achieve good performance. In particular, flexible object manipulation is particularly sensitive to the simulation-to-real gap, as accurately simulating flexible dynamics is a major challenge, and related technologies cannot achieve accurate simulation of flexible objects. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a flexible object manipulation strategy generation method based on multi-modal fusion, which improves the accuracy of flexible object manipulation, has strong adaptability to different states of flexible objects, and reduces the simulation-to-real gap.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] A flexible object manipulation strategy generation method based on multi-modal fusion, the method comprising:

[0008] Based on human demonstration videos, obtain the target flexible object's grasp point two-dimensional Gaussian distribution map, three-dimensional point cloud data, and RGB image;

[0009] Based on the three-dimensional point cloud data, obtain the target flexible object's visibility connection graph;

[0010] fuse the grasp point two-dimensional Gaussian distribution map and the RGB image to obtain a fusion image;

[0011] extract first features and second features from the visibility connection graph and the fusion image respectively;

[0012] fuse the first features and the second features to obtain fusion features;

[0013] based on the fusion features, obtain the optimal grasp point and the optimal placement point of the target flexible object;

[0014] based on the optimal grasp point and the optimal placement point, generate a control strategy for the target flexible object.

[0015] As a preferred technical solution, obtaining the grasp point two-dimensional Gaussian distribution map comprises:

[0016] using a deep learning model to perform hand tracking and key point detection on the operating hand in the human demonstration video to obtain the grasp point of the operating person when operating the flexible object;

[0017] based on the grasp point, generate a grasp point two-dimensional Gaussian distribution of the target flexible object.

[0018] As a preferred technical solution, obtaining the three-dimensional point cloud data comprises:

[0019] based on the infrared image and the depth image obtained when the human demonstration video is collected, construct the three-dimensional point cloud data of the target flexible object.

[0020] As a preferred technical solution, the method further comprises:

[0021] perform image preprocessing on the infrared image and the depth image; wherein the image preprocessing comprises at least one of noise filtering, edge enhancement and dynamic contrast adjustment.

[0022] As a preferred technical solution, based on the three-dimensional point cloud data, obtaining the visibility connection graph of the target flexible object comprises:

[0023] based on the three-dimensional point cloud data, construct a graph model of the target flexible object;

[0024] based on the graph model, obtain the visibility connection graph of the target flexible object.

[0025] As a preferred technical solution, the method further comprises:

[0026] using the three-dimensional point cloud data as model input, construct a graph model <V, E> for representing the state of the target flexible object.

[0027] wherein, V represents a node; E represents an edge;

[0028] nodes V = {v i} represent particles constituting the target flexible object; i = 1, 2, …, N, N represents the number of particles; particle nodes x i represents the spatial position of a particle, represents the velocity vector of a particle;

[0029] edges E include grid edges E M and adjacent edges E C , wherein the grid edges E M represent the connection relationship between particles inherent in the grid structure of the flexible object, and the adjacent edges E C represent the dynamic collision relationship between particles that are close in space.

[0030] As a preferred technical solution, the adjacent edges E C are dynamically defined based on the real-time Euclidean distance between particles, that is, at each time step t:

[0031]

[0032] wherein, represents the set of adjacent edges formed between particles at time step t; e ij represents that there is a collision relationship between particle i and particle j at time step t; x i,t and x j,t respectively represent the spatial positions of particle i and particle j at time step t; ||·||2 is the operation of calculating the Euclidean distance; R is a preset distance threshold.

[0033] As a preferred technical solution, the method further comprises:

[0034] constructing the constraint relationship of the grid edges E M and the adjacent edges E C .

[0035] As a preferred technical solution, the method further comprises:

[0036] performing voxelization processing on the three-dimensional point cloud data to convert the three-dimensional point cloud data into a regular voxel grid array;

[0037] acquiring the visibility connection graph of the target flexible object by using the voxel grid array.

[0038] As a preferred technical solution, the first feature and the second feature are fused to obtain a fusion feature, which comprises:

[0039] The first feature and the second feature are spliced by channel to obtain a fusion feature.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] I. Improve the precision of flexible object operation: the flexible object manipulation strategy generation method in the present application can more comprehensively represent the state of the flexible object by synchronously capturing and fusing multi-modal data such as depth, RGB image, infrared image and two-dimensional Gaussian distribution image of the grasping point, and can more accurately predict the best grasping point and the best placing point of the flexible object based on the fusion feature, which is beneficial to improve the operation precision of the robot on the flexible object in the model learning task.

[0042] II. Strong adaptability to different states of flexible objects: the flexible object manipulation strategy generation method in the present application can make the robot realize the conversion of the flexible object from any wrinkle state to the target folding state under different flexible object states by learning the complex gestures and operation details in the human demonstration video, and has strong adaptability.

[0043] III. Reduce the gap between simulation and reality (sim-to-real): the flexible object manipulation strategy generation method in the present application extracts multi-modal data from human demonstration video, so that the robot can learn directly from the real environment, avoiding the gap between the simulation environment and the actual operation in the traditional method. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a flowchart of the flexible object manipulation strategy generation method based on multi-modal fusion in the embodiment of the present application.

[0045] Figure 2 It is a flowchart of constructing a graph model in the embodiment of the present application.

[0046] Figure 3 It is a structural diagram of an electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0048] "One embodiment" or "embodiment" referred to herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present application. In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Moreover, the terms "first", "second", etc. are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0049] like Figure 1 As shown, an embodiment of the present application provides a method for generating a flexible object manipulation strategy based on multimodal fusion, the method comprising:

[0050] Step S110: Based on the human demonstration video, a two-dimensional Gaussian distribution map, three-dimensional point cloud data, and RGB map of the grasping points of the target flexible object are obtained;

[0051] Step S120: obtaining a visibility connection graph of the target flexible object based on the three-dimensional point cloud data;

[0052] Step S130: Fusing the two-dimensional Gaussian distribution map of the grasped points and the RGB map to obtain a fused map;

[0053] Step S140: extracting the first feature and the second feature from the visibility connection graph and the fusion graph respectively;

[0054] Step S150: fusing the first feature and the second feature to obtain a fused feature;

[0055] Step S160: obtaining the optimal grasping point and the optimal placement point of the target flexible object based on the fusion features;

[0056] Step S170: generating a manipulation strategy for the target flexible object based on the optimal grasping point and the optimal placement point.

[0057] The above steps S110 to S170 are described in detail below:

[0058] The human demonstration video in the above step S110 can be captured by a Realsense L515 camera. The Realsense L515 camera can also synchronously capture the depth map, RGB map and infrared map of the target flexible object when capturing the human demonstration video.

[0059] Optionally, the step S110 of acquiring the two-dimensional Gaussian distribution diagram of the grabbing point comprises: performing hand tracking and key point detection on the hand of the operator in the human demonstration video by using a deep learning model to acquire the grabbing point of the operator when operating the flexible object; and generating a two-dimensional Gaussian distribution of the grabbing point of the target flexible object based on the grabbing point. For example, a deep learning model based on Yolov10 is used to perform hand tracking and key point detection on the hand of the operator in the human demonstration video to accurately identify dynamic gesture data of the human when operating the flexible object and record the grabbing point and the placing point of the human operation.

[0060] It can be understood that the deep learning model based on Yolov10 can process video stream data in real time, identify and record dynamic gesture data of the human when operating the flexible object, including the dynamic change process of the finger position, the grabbing action and the placing action, thereby providing high-precision hand movement trajectory data of the operator. The acquisition of these data lays a foundation for subsequent multi-modal data acquisition and processing.

[0061] In addition, the deep learning model based on Yolov10 is specially designed with an adaptive algorithm, which can be adjusted in real time according to the hand features and action differences of different operators, thereby improving the recognition accuracy and stability in changing environments (such as different lighting conditions and background changes).

[0062] In addition, the Realsense L515 camera used when collecting the human operation video has high-precision depth perception capability and can acquire three-dimensional depth information, infrared images and RGB images of the scene in real time. These images provide comprehensive information support for the state description of the target flexible object.

[0063] Optionally, the step S110 of acquiring the three-dimensional point cloud data comprises: constructing three-dimensional point cloud data of the target flexible object based on the infrared image and the depth image acquired when collecting the human demonstration video. For example, the implementation manner is as follows:

[0064] The infrared image, the depth image and the camera intrinsic parameter of the Realsense L515 camera are used to accurately reconstruct the three-dimensional point cloud of the target flexible object to represent the complex spatial structure of the target flexible object.

[0065] It can be understood that when constructing the three-dimensional point cloud, the infrared image can be used to accurately identify and segment the boundary of the flexible object, and then the three-dimensional point cloud is reconstructed based on the boundary information, the depth image and the camera intrinsic parameter to reconstruct a three-dimensional point cloud with higher accuracy.

[0066] Optionally, the step S120 comprises: performing voxelization processing on the three-dimensional point cloud data to convert the three-dimensional point cloud data into a regular voxel grid array; and acquiring a visibility connection graph of the target flexible object by using the voxel grid array.

[0067] It can be understood that the point cloud data is converted into a regular voxel grid array after voxelization, which simplifies the data structure and thus facilitates the improvement of the efficiency of subsequent calculation.

[0068] Optionally, the flexible object manipulation strategy generation method based on multi-modal fusion further comprises: image preprocessing of the infrared image and the depth image; wherein the image preprocessing comprises at least one of noise filtering, edge enhancement and dynamic contrast adjustment.

[0069] The preprocessing methods such as noise filtering, edge enhancement and dynamic contrast adjustment are aimed at optimizing the image, especially in low-light or high-dynamic-range environments, to ensure the highest accuracy of segmentation and identification of the flexible object boundary, thereby providing the clearest and most accurate image input for point cloud generation and flexible object modeling.

[0070] Optionally, the step S120 comprises: constructing a graph model of the target flexible object based on the three-dimensional point cloud data; and obtaining a visibility connection graph of the target flexible object based on the graph model.

[0071] Optionally, the construction of the graph model of the target flexible object comprises:

[0072] The three-dimensional point cloud data is used as the model input to construct a graph model <V, E> for representing the state of the target flexible object.

[0073] Wherein, V is a node; E is an edge;

[0074] The node V = {v i} represents the particles constituting the target flexible object; i = 1, 2, …, N, N is the number of particles; the particle node x i represents the spatial position of the particle, represents the particle velocity vector;

[0075] The edge E includes a grid edge E M and a neighboring edge E C , wherein the grid edge E M represents the connection relationship between the particles inherent in the grid structure of the flexible object, which is determined by the structure of the flexible object and remains unchanged in time; the neighboring edge E C represents the dynamic collision relationship between the particles that are close in space, which can change due to the folding or movement of the flexible object.

[0076] Optionally, the neighboring edge E C is dynamically defined based on the real-time Euclidean distance between the particles, i.e. at each time step t:

[0077]

[0078] wherein, represents a set of adjacent edges formed between particles at time step t; e ij represents a collision relationship between particle i and particle j at time step t; x i,t and x j,t respectively represent the spatial positions of particle i and particle j at time step t; ||·||2 is a Euclidean distance operation; R is a preset distance threshold.

[0079] Optionally, the method for constructing the graph model of the target flexible object further comprises: constructing a constraint relationship of the grid edge E N and the adjacent edge E C .

[0080] It can be understood that, since the grid edge E M and the connected nodes are physically close to each other, the constraint relationship is set to maintain the consistency of the model and meet the actual physical characteristics of the flexible object.

[0081] The construction process of the graph model is shown in Figure 2 , and the construction process can include the following steps:

[0082] (a) obtaining three-dimensional point cloud data, that is, an original point cloud;

[0083] (b) converting the original point cloud into a voxelized point cloud;

[0084] (c) constructing a graph model based on the voxelized point cloud to obtain graph data.

[0085] In the above scheme, the graph model can not only capture the structure and dynamic changes of the flexible object, but also help to perceive the current state of the self-occluded flexible object and predict the interaction relationship between the particles of the flexible object. The application of this technology significantly improves the accuracy of understanding and operating the dynamic characteristics of the flexible object, and provides a powerful tool for intelligent robots to handle complex deformable objects.

[0086] The scheme of step S130 of fusing the two-dimensional Gaussian distribution graph of the grasping point and the RGB graph will be introduced as follows:

[0087] The image fusion algorithm in the related art can be used to fuse the two-dimensional Gaussian distribution graph of the grasping point and the RGB graph, such as the weighted fusion method, the interpolation fusion method, etc. Since the image fusion algorithm is a relatively mature technology in the field, the present application will not be described in detail.

[0088] The scheme of step S140 of extracting the first feature and the second feature in the visibility connection graph and the fused graph will be introduced as follows:

[0089] In an embodiment of the present application, step S140 can use UNet to extract the first feature and the second feature in the visibility connection graph and the fusion graph respectively. The above-mentioned UNet is a deep learning model, and its network architecture includes a feature extraction part in the first half and an upsampling part in the second half. In the feature extraction stage, UNet extracts the features of the image layer by layer through convolution and pooling operations; in the upsampling stage, UNet gradually restores the image size through deconvolution or up-pooling operations, and fuses the feature map extracted in the feature extraction stage with the feature map restored in the upsampling stage through skip connection.

[0090] Optionally, the above step S150 includes: splicing the first feature and the second feature by channel to obtain a fusion feature.

[0091] It can be understood that since the RGB image provides the texture and color information of the flexible object, the two-dimensional Gaussian distribution map of the grasping point provides the possible grasping point position information, and the three-dimensional point cloud data provides the three-dimensional collective morphological information of the flexible object, the fusion feature integrates the above information and can achieve a comprehensive perception and understanding of the state of the flexible object.

[0092] It is understandable that step S160 may use the decoding module of the UNet described above to obtain the optimal grasping point and the optimal placement point of the target flexible object based on the fusion features.

[0093] Furthermore, it is understood that after obtaining the optimal grasping point and the optimal placement point, step S170 can integrate robotic motion planning technology, path planning technology, and force control technology to jointly generate a manipulation strategy for the target flexible object, thereby achieving efficient conversion of the flexible object from any wrinkled state to the target folded state. The aforementioned robotic motion planning technology, path planning technology, and force control technology are all relatively mature technologies in the field, and therefore, are not further described in detail in the present embodiment.

[0094] Figure 3 1 shows a schematic block diagram of an electronic device that can be used to implement an embodiment of the present disclosure. Figure 3 As shown, the electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0095] A number of components in the device are connected to the I / O interface, including: input units, such as a keyboard, a mouse, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as a magnetic disk, an optical disk, etc.; and communication units, such as a network card, a modem, a wireless communication transceiver, etc. The communication units allow the device to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0096] The processing unit performs the various methods and processes described above, such as the steps S110-S170 of the method of the present application. For example, in some embodiments, the steps S110-S170 of the method of the present application can be implemented as a computer software program tangibly embodied in a machine readable medium, such as the storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device via the ROM and / or the communication unit. When the computer program is loaded onto the RAM and executed by the CPU, one or more of the steps S110-S170 of the method of the present application described above can be performed. Alternatively, in other embodiments, the CPU can be configured, by way of firmware or software, to perform the steps S110-S170 of the method of the present application.

[0097] The functionality described above above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0098] Program code for carrying out the methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partly on a machine, partly on a machine as a stand-alone software package, partly on a machine and partly on a remote machine or entirely on a remote machine or server.

[0099] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage media can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include one or more lines of electrical wire, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0100] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for generating flexible object manipulation strategies based on multimodal fusion, characterized in that: The method comprises: Based on human demonstration videos, we obtain the 2D Gaussian distribution map, 3D point cloud data, and RGB image of the grasping points of the target flexible object. Acquire a visibility connection graph of the target flexible object based on the three-dimensional point cloud data; Fusing the two-dimensional Gaussian distribution map of the grasping points and the RGB map to obtain a fused map; Extracting a first feature and a second feature from the visibility connection graph and the fusion graph respectively; fusing the first feature and the second feature to obtain a fused feature; Based on the fusion features, obtaining an optimal grasping point and an optimal placement point of the target flexible object; generating a manipulation strategy for the target flexible object based on the optimal grasping point and the optimal placement point; The obtaining of a visibility connection graph of the target flexible object based on the three-dimensional point cloud data includes: constructing a graphical model of the target flexible object based on the three-dimensional point cloud data; Based on the graph model, obtaining a visibility connection graph of the target flexible object; The step of constructing a graphical model of the target flexible object includes: Using the three-dimensional point cloud data as model input, a graphical model is constructed to represent the state of the target flexible object. ; in, is a node; For the edge; node representing particles constituting the target flexible object; , is the number of particles; particle nodes , represents the spatial position of the particle, represents the particle velocity vector; side Include Mesh Edges and adjacent edges , wherein the grid edge Indicates the inherent connection between particles in the flexible object grid structure, adjacent edges Represents the dynamic collision relationship between particles that are close in space; The adjacent edge Based on the dynamic definition of the real-time Euclidean distance between particles, that is, at each time step : in, Indicates that at time step When , the set of adjacent edges formed between particles; Indicates that at time step When the particle With particles There is a collision relationship between them; and Represents the time step When the particle With particles spatial location; To calculate the Euclidean distance operation; is the preset distance threshold; The step of constructing the graphical model of the target flexible object further includes: Construct the mesh edge With the adjacent edge Constraints: .

2. The method for generating a flexible object manipulation strategy based on multimodal fusion according to claim 1, characterized in that: Obtaining a two-dimensional Gaussian distribution map of the grasping points includes: Using a deep learning model to perform hand tracking and key point detection on the operator's hands in the human demonstration video to obtain the operator's grasping points when operating the flexible object; Based on the grasping points, a two-dimensional Gaussian distribution of the grasping points of the target flexible object is generated.

3. The method for generating a flexible object manipulation strategy based on multimodal fusion according to claim 1, characterized in that: Acquiring the three-dimensional point cloud data includes: Based on the infrared image and the depth image obtained when collecting the human demonstration video, three-dimensional point cloud data of the target flexible object is constructed.

4. The method for generating a flexible object manipulation strategy based on multimodal fusion according to claim 3, characterized in that: The method further comprises: Perform image preprocessing on the infrared image and the depth image; wherein the image preprocessing includes at least one of noise filtering, edge enhancement, and dynamic contrast adjustment.

5. The method for generating a flexible object manipulation strategy based on multimodal fusion according to any one of claims 1 to 4, characterized in that: The obtaining of a visibility connection graph of the target flexible object based on the three-dimensional point cloud data includes: Performing voxel processing on the three-dimensional point cloud data to convert the three-dimensional point cloud data into a regular voxel grid array; A visibility connection graph of the target flexible object is obtained using the voxel grid array.

6. The method for generating a flexible object manipulation strategy based on multimodal fusion according to any one of claims 1 to 4, characterized in that: The fusing the first feature and the second feature to obtain a fused feature includes: The first feature and the second feature are spliced ​​by channel to obtain a fusion feature.

Images