A method and system for dynamic grasping of robots based on multimodal data fusion

Abstract

This invention provides a robot dynamic grasping method and system based on multimodal data fusion, comprising: performing multimodal fusion of visual and tactile data of a dynamic object to be grasped by the robot to obtain multimodal fused data; based on the multimodal fused data, performing target recognition on the dynamic object to be grasped using a region proposal-based target detection algorithm to obtain position and pose information; based on the position and pose information, performing state prediction on the dynamic object to be grasped to obtain state prediction information; and based on the state prediction information, generating a grasping path for the robot using a reinforcement learning algorithm. This application, by performing multimodal fusion of the collected visual and tactile data, enables the robot to obtain comprehensive environmental information, increasing the robot's adaptability to environmental changes; and by predicting the state of the object to be grasped, the robot can predict the object's trajectory, which helps improve the robot's grasping success rate.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a robot dynamic grasping method and system based on multimodal data fusion. Background Technology

[0002] Currently, with the rapid development of robotics technology, robots are being used more and more widely in many fields, such as logistics, manufacturing, and healthcare. Robotic grasping technology, as a crucial component, enables automated operation and task execution, improving production efficiency. However, most existing grasping methods are designed based on static environments, relying primarily on the static features of objects for learning and recognition. These methods perform well in fixed environments, but their ability to recognize and plan grasping objects in dynamic environments is clearly insufficient.

[0003] In many practical applications, objects may move or deform due to external forces, or be obstructed by other objects, increasing the difficulty of recognition. Traditional grasping methods often cannot handle these complex changes, leading to grasping failures or low efficiency. Furthermore, existing methods often focus only on data from a single modality, failing to fully utilize information from multiple sensory signals, thus limiting the accuracy and efficiency of recognition. Summary of the Invention

[0004] To address the issue that existing grasping methods can only learn to grasp objects in static environments due to their movement, deformation, or occlusion, and lack the ability to recognize and plan grasping objects in dynamic environments, this invention proposes a robot dynamic grasping method based on multimodal data fusion, comprising:

[0005] The robot collects visual and tactile data of the dynamic object to be grasped, and performs multimodal fusion on the visual and tactile data to obtain multimodal fused data of the dynamic object to be grasped.

[0006] Based on the multimodal fusion data, a target detection algorithm based on region proposal is used to identify the dynamic object to be grasped, thereby obtaining the position and pose information of the dynamic object to be grasped.

[0007] Based on the position and pose information, the state of the dynamic object to be grasped is predicted to obtain the state prediction information of the dynamic object to be grasped.

[0008] Based on the state prediction information of the dynamic object to be grasped, a reinforcement learning algorithm is used to generate the robot's grasping path.

[0009] Optionally, the step of multimodal fusion of the visual data and the tactile data to obtain multimodal fused data of the dynamic object to be grasped includes:

[0010] Visual feature data is obtained by extracting features from the visual data using a visual feature selection method.

[0011] Tactile feature data is obtained by extracting features from the tactile data using a one-dimensional convolutional neural network.

[0012] An adversarial training mechanism is used to perform cross-modal mapping on the visual feature data and the tactile feature data to obtain the mapped fused feature vector;

[0013] Based on the mapped fusion feature vector, output the multimodal fusion data of the dynamic object to be captured;

[0014] The visual data includes one or more of the following: color image data, depth image data, optical flow data, and ambient lighting data;

[0015] The tactile data includes one or more of the following: robot force feedback data, robot torque data, grasping pressure data, object surface friction data, and object hardness data.

[0016] Optionally, the expression for the adversarial training mechanism is as follows:

[0017] F fused =G(F visual ;θ g )+βH(F haptic ;θ h );

[0018] In the formula,

[0019]

[0020] Among them, F fused G represents the fused feature vector after mapping; F represents the generator model; visual Represents visual feature data; θ g The parameters of the generator model are represented by β; the dynamic weighting factor is represented by H; the nonlinear transformation model is represented by F. haptic Represents tactile feature data; θ h The parameters represent the nonlinear transformation model; Similarity represents the similarity score.

[0021] Optionally, the step of performing target recognition on the dynamic object to be grasped using a region-proposal-based target detection algorithm based on the multimodal fusion data to obtain the position and pose information of the dynamic object to be grasped includes:

[0022] Based on the multimodal fusion data, a candidate region proposal is generated using a selective search algorithm;

[0023] Based on the proposed candidate regions, a target detection algorithm is used to identify the dynamic object to be grasped, thereby obtaining the classification result and bounding box coordinate information of the dynamic object to be grasped.

[0024] Based on the classification results and bounding box coordinate information of the dynamic object to be grasped, the attitude of the dynamic object to be grasped is estimated to obtain the position and attitude information of the dynamic object to be grasped.

[0025] Optionally, the step of predicting the state of the dynamic object to be grasped based on the position and pose information to obtain the state prediction information of the dynamic object to be grasped includes:

[0026] An autoencoder is used to extract state features from the position and pose information to obtain the fused feature representation of the dynamic object to be grasped.

[0027] Based on the fused feature representation of the dynamic object to be grasped, the trajectory of the dynamic object to be grasped is predicted by using a long short-term memory network combined with a conditional random field, thereby obtaining the motion trajectory prediction information of the dynamic object to be grasped.

[0028] Based on the motion trajectory prediction information of the dynamic object to be grasped, a deep reinforcement learning algorithm is used to predict the state of the dynamic object to be grasped, thereby obtaining the state prediction information of the dynamic object to be grasped.

[0029] Optionally, the step of predicting the trajectory of the dynamic object to be grasped by using a long short-term memory network combined with a conditional random field based on the fused feature representation of the dynamic object to be grasped, to obtain the motion trajectory prediction information of the dynamic object to be grasped, includes:

[0030] The feature representation of the fused dynamic object to be captured is transformed into contextual features to obtain feature sequence data.

[0031] The feature sequence data is implicitly predicted using a long short-term memory network, and the hidden state sequence of the feature sequence data is output.

[0032] By post-processing the hidden state sequence using a conditional random field, the motion trajectory prediction information of the dynamic object to be captured is obtained.

[0033] Optionally, generating the robot's grasping path using a reinforcement learning algorithm based on the state prediction information of the dynamic object to be grasped includes:

[0034] Based on the state prediction information of the dynamic object to be grasped and the environmental information of the dynamic object to be grasped collected in advance, scene perception is performed on the dynamic object to be grasped to obtain the scene state description information of the dynamic object to be grasped.

[0035] Based on the scene state description information, a dynamic crawling strategy network is constructed using the deep deterministic strategy gradient algorithm to obtain the dynamic crawling strategy.

[0036] According to the dynamic grasping strategy, a path generation algorithm based on imitation learning is used to generate a path to obtain the grasping path of the dynamic object to be grasped.

[0037] Based on the same inventive concept, this invention also provides a robot dynamic grasping system based on multimodal data fusion, comprising:

[0038] The multimodal fusion module is used to collect visual and tactile data of a dynamic object to be grasped by the robot, and to perform multimodal fusion on the visual and tactile data to obtain multimodal fused data of the dynamic object to be grasped.

[0039] The target recognition module is used to identify the dynamic object to be grasped based on the multimodal fusion data and a target detection algorithm based on region proposal, so as to obtain the position and pose information of the dynamic object to be grasped.

[0040] The state prediction module is used to predict the state of the dynamic object to be grasped based on the position and posture information, and obtain the state prediction information of the dynamic object to be grasped.

[0041] The path generation module is used to generate the robot's grasping path based on the state prediction information of the dynamic object to be grasped, using a reinforcement learning algorithm.

[0042] Optionally, the multimodal fusion module includes:

[0043] The visual feature extraction submodule is used to extract features from the visual data using a visual feature selection method to obtain visual feature data.

[0044] The tactile feature extraction submodule is used to extract features from the tactile data using a one-dimensional convolutional neural network to obtain tactile feature data;

[0045] The cross-modal mapping submodule is used to perform cross-modal mapping on the visual feature data and the tactile feature data using an adversarial training mechanism to obtain the mapped fused feature vector;

[0046] The data fusion submodule is used to output the multimodal fusion data of the dynamic object to be captured based on the mapped fusion feature vector.

[0047] The visual data includes one or more of the following: color image data, depth image data, optical flow data, and ambient lighting data;

[0048] The tactile data includes one or more of the following: robot force feedback data, robot torque data, grasping pressure data, object surface friction data, and object hardness data.

[0049] Optionally, the expression for the adversarial training mechanism is as follows:

[0050] F fused =G(F visual ;θ g )+βH(F haptic ;θ h );

[0051] In the formula,

[0052]

[0053] Among them, F fused G represents the fused feature vector after mapping; F represents the generator model; visual Represents visual feature data; θ g The parameters of the generator model are represented by β; the dynamic weighting factor is represented by H; the nonlinear transformation model is represented by F. haptic Represents tactile feature data; θ h The parameters represent the nonlinear transformation model; Similarity represents the similarity score.

[0054] Optionally, the target recognition module includes:

[0055] The candidate region generation submodule is used to generate candidate region proposals based on the multimodal fusion data using a selective search algorithm.

[0056] The target detection submodule is used to identify the target of the dynamic object to be grasped by using a target detection algorithm based on the candidate region proposal, and to obtain the classification result and bounding box coordinate information of the dynamic object to be grasped.

[0057] The pose estimation submodule is used to estimate the pose of the dynamic object to be grasped based on the classification result and bounding box coordinate information of the dynamic object to be grasped, so as to obtain the position and pose information of the dynamic object to be grasped.

[0058] Optionally, the state prediction module includes:

[0059] The feature representation submodule is used to extract state features from the position and pose information using an autoencoder to obtain the fused feature representation of the dynamic object to be grasped.

[0060] The trajectory prediction submodule is used to predict the trajectory of the dynamic object to be grasped by using a long short-term memory network combined with a conditional random field based on the fused feature representation of the dynamic object to be grasped, so as to obtain the motion trajectory prediction information of the dynamic object to be grasped.

[0061] The state estimation submodule is used to predict the state of the dynamic object to be grasped by using a deep reinforcement learning algorithm based on the motion trajectory prediction information of the dynamic object to be grasped, so as to obtain the state prediction information of the dynamic object to be grasped.

[0062] Optionally, the trajectory prediction submodule includes:

[0063] The feature conversion unit is used to perform context feature conversion on the feature representation of the fused dynamic object to be captured to obtain feature sequence data.

[0064] The hidden prediction unit is used to perform hidden prediction on the feature sequence data using a long short-term memory network and output the hidden state sequence of the feature sequence data.

[0065] The post-processing unit is used to post-process the hidden state sequence through a conditional random field to obtain the motion trajectory prediction information of the dynamic object to be captured.

[0066] Optionally, the path generation module includes:

[0067] Based on the state prediction information of the dynamic object to be grasped and the environmental information of the dynamic object to be grasped collected in advance, scene perception is performed on the dynamic object to be grasped to obtain the scene state description information of the dynamic object to be grasped.

[0068] Based on the scene state description information, a dynamic crawling strategy network is constructed using the deep deterministic strategy gradient algorithm to obtain the dynamic crawling strategy.

[0069] According to the dynamic grasping strategy, a path generation algorithm based on imitation learning is used to generate a path to obtain the grasping path of the dynamic object to be grasped.

[0070] In another aspect, the present invention also provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0071] The memory is used to store one or more programs;

[0072] When the one or more programs are executed by the at least one processor, a robot dynamic grasping method based on multimodal data fusion as described above is implemented.

[0073] In another aspect, the present invention also provides a computer device readable storage medium having an executable program stored thereon, wherein when the executable program is executed, it implements the robot dynamic grasping method based on multimodal data fusion as described above.

[0074] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0075] This invention provides a robot dynamic grasping method and system based on multimodal data fusion, comprising: collecting visual and tactile data of a dynamic object to be grasped by a robot, and performing multimodal fusion on the visual and tactile data to obtain multimodal fused data of the dynamic object to be grasped; performing target recognition on the dynamic object to be grasped using a region proposal-based target detection algorithm based on the multimodal fused data to obtain the position and posture information of the dynamic object to be grasped; performing state prediction on the dynamic object to be grasped based on the position and posture information to obtain state prediction information of the dynamic object to be grasped; and generating a grasping path for the robot using a reinforcement learning algorithm based on the state prediction information of the dynamic object to be grasped. This application, by performing multimodal fusion on the collected visual and tactile data, enables the robot to obtain comprehensive environmental information, increasing the robot's adaptability to environmental changes; by predicting the state of the object to be grasped, the robot can predict the object's trajectory, which helps improve the robot's grasping success rate; therefore, the method of this invention can significantly improve the grasping effect and efficiency of the robot in dynamic environments. Attached Figure Description

[0076] Figure 1 A flowchart illustrating a robot dynamic grasping method based on multimodal data fusion provided by the present invention;

[0077] Figure 2 This is a schematic diagram illustrating the process of multimodal data fusion in a robot dynamic grasping method based on multimodal data fusion provided by the present invention.

[0078] Figure 3 This is a schematic diagram of the robot target recognition process in a robot dynamic grasping method based on multimodal data fusion provided by the present invention.

[0079] Figure 4 This is a schematic diagram of the robot state prediction process in a robot dynamic grasping method based on multimodal data fusion provided by the present invention.

[0080] Figure 5 This is a schematic diagram illustrating the robot path generation process in a robot dynamic grasping method based on multimodal data fusion provided by the present invention.

[0081] Figure 6 A schematic diagram illustrating the structural composition of a robot dynamic grasping system based on multimodal data fusion provided by the present invention;

[0082] Figure 7 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0083] This invention proposes a robot dynamic grasping method, system, device, and medium based on multimodal data fusion. The specific embodiments of this invention will be further described in detail below with reference to the accompanying drawings.

[0084] Example 1:

[0085] This invention provides a robot dynamic grasping method based on multimodal data fusion, the flowchart of which is shown below. Figure 1 As shown, it includes:

[0086] Step 1: The robot collects visual and tactile data of the dynamic object to be grasped, and performs multimodal fusion on the visual and tactile data to obtain multimodal fused data of the dynamic object to be grasped.

[0087] Step 2: Based on the multimodal fusion data, use a target detection algorithm based on region proposal to identify the dynamic object to be grasped, and obtain the position and pose information of the dynamic object to be grasped.

[0088] Step 3: Based on the position and pose information, perform state prediction on the dynamic object to be grasped to obtain the state prediction information of the dynamic object to be grasped.

[0089] Step 4: Based on the state prediction information of the dynamic object to be grasped, use a reinforcement learning algorithm to generate the robot's grasping path.

[0090] In general, object grasping is a key aspect of robotics research and application. However, most existing grasping methods focus on recognizing and manipulating objects in static environments, and typically rely on single-modality data (such as visual information). For example, traditional visual recognition systems mainly acquire image data through cameras and then extract object features for target recognition. This single visual modality often faces challenges in handling dynamic scenes, including object occlusion, changes in lighting, and state changes caused by object movement. These factors can lead to grasping failures or misidentifications, severely impacting the robot's efficiency and success rate in practical applications. To address these issues, this application considers acquiring multimodal data, i.e., simultaneously acquiring tactile and visual data of dynamic objects. This multimodal data acquisition helps the robot to more comprehensively and accurately understand the characteristics and current state of objects, thereby improving its grasping performance in dynamic environments. For example, the visual data acquired by the robot may include one or more of the following: color image data, depth image data, optical flow data, and ambient lighting data. In this example, visual data can be captured by a high-resolution camera or depth sensor (such as an RGB-D camera). These devices can provide image and depth information of objects, enabling the robot to recognize features such as the shape, color, and size of objects.

[0091] For example, the tactile data collected by the robot may include one or more of the following: robot force feedback data, robot torque data, grasping pressure data, object surface friction data, and object hardness data. In this example, the tactile data can be acquired by tactile sensors installed on the robot arm. These sensors can sense information such as the object's force feedback, surface texture, and temperature, helping the robot to assess the object's state and texture when actually contacting it.

[0092] In addition to tactile and visual modal data, it is also possible to collect auditory data (through additional microphones or acoustic sensors, the robot can collect audio information in the environment, which is helpful in identifying the collision sounds of objects or the sounds of mechanical movement, thus providing auxiliary information for grasping decisions) and inertial data (with the help of sensors such as accelerometers and gyroscopes, the robot can obtain its own position and posture changes) to enhance the robot's perception capabilities and grasping effect.

[0093] The multimodal data collected through the above steps can enhance the robot's perception and all-around information acquisition capabilities. To further achieve efficient utilization of this multimodal data, the following multimodal data fusion method can be considered:

[0094] In one implementation, such as Figure 2As shown, the process of multimodal fusion of visual and tactile data in step 1 above to obtain multimodal fused data of the dynamic object to be grasped may include:

[0095] Visual feature data is obtained by extracting features from visual data using the visual feature selection method.

[0096] Tactile feature data is obtained by extracting features from tactile data using a one-dimensional convolutional neural network.

[0097] An adversarial training mechanism is used to perform cross-modal mapping on visual feature data and tactile feature data to obtain the mapped fused feature vector;

[0098] Based on the mapped fusion feature vector, output the multimodal fusion data of the dynamic object to be captured;

[0099] In this implementation, the visual feature selection method used for visual data processing filters out low-quality light deviation data by targeting different types of visual inputs (such as RGB images, depth images, or optical flow data), focusing attention on edge information and shape contours. This significantly reduces redundant features and selects the most representative feature data. This not only improves the efficiency of data processing but also enhances the ability of key features to express dynamic objects. The result helps the robot to more accurately perceive the shape, trajectory, and environmental changes of objects. Furthermore, this feature extraction reduces noise interference, ultimately improving the accuracy of the target detection algorithm. Especially in dynamic object grasping scenarios, the visual feature selection method eliminates redundant visual information and retains only key feature data. This selection not only optimizes computational resources but also ensures the real-time performance of the grasping operation. When processing tactile data, the one-dimensional convolutional neural network (CNN) can extract key time-series features from tactile data with compact computational resources. When processing feedback data (such as force feedback and pressure data), CNN refines the raw tactile data into sequence features of surface friction, hardness, and dynamic contact of objects by identifying the patterns of change of tactile sensors over time. This feature extraction method can not only handle highly dynamic unstructured data, but also support information synchronization between tactile and visual senses, establishing deep semantic associations for subsequent fusion operations. Furthermore, by using a one-dimensional CNN, this implementation can focus on feature extraction of time-series signals, capturing subtle changes in tactile sensors at the moment of object contact, such as a sudden increase in pressure, changes in friction, or differences in object hardness. In object grasping scenarios, the advantage of using one-dimensional convolution is that it can directly process time-series inputs, has high computational efficiency, and is easy to parallelize. Through this design, tactile data is not only transformed into high-dimensional semantic information, but also expresses the dynamic contact state of objects in real-time scenes, which is of great significance for solving the time-series synchronization problem in multimodal data integration.This implementation introduces an adversarial training mechanism for cross-modal mapping. Through the game between the generator and the discriminator, visual and tactile features are mapped to a shared latent semantic space. Visual and tactile data are inherently heterogeneous, possessing natural intermodal differences. Through the adversarial network, the shared feature space of the two modalities can be gradually found within a game framework between the generator and the discriminator. This allows visual and tactile information to complement and align with each other in high-dimensional semantics. For example, features about object contours in visual data can correspond to hardness or friction features in tactile data. The generator establishes this correspondence during the mapping process, while the adversarial design of the discriminator further enhances this effect. This approach ensures the accuracy of feature mapping and avoids potential information loss or ambiguity during multimodal data fusion. It significantly enhances the expressive power of the fused feature vectors, enabling the fused data to possess both the global perception of vision and the fine-grained discrimination of touch. This not only addresses the differences in characteristics between different modalities but also maximizes the preservation of implicit correlations between modalities. For example, by extracting the mapping weights between vision and touch through a generator model, fine-grained target recognition and cross-modal collaboration can be simultaneously guaranteed. This design avoids information loss and improves the expressiveness of multimodal data fusion, allowing robots to efficiently extract global features of dynamic objects in complex environments. When outputting multimodal fusion data based on the mapped fused feature vectors, a semantic alignment mechanism can be established to ensure that the final integrated data accurately reflects the complex dynamics of the target object. For example, visually, it can more accurately reflect the target object's trajectory or deformation; tactilely, it can reflect subtle changes such as real-time contact force and dynamic friction, thereby improving the overall grasping success rate.

[0100] For example, the expression for the adversarial training mechanism described above can be as follows:

[0101] F fused =G(F visual ;θ g )+βH(F haptic ;θ h );

[0102] In the formula,

[0103]

[0104] Among them, F fused G represents the fused feature vector after mapping; F represents the generator model; visual Represents visual feature data; θ g The parameters of the generator model are represented by β; the dynamic weighting factor is represented by H; the nonlinear transformation model is represented by F. haptic Represents tactile feature data; θ hThe parameters represent the nonlinear transformation model; Similarity represents the similarity score. In this example, by fusing the generator model and the nonlinear transformation model, the feature vector is adjusted and optimized during the fusion of multimodal data, thereby achieving more accurate and robust dynamic environmental target perception. By introducing a dynamic weight factor β (corrected by the similarity score between visual and tactile features), when visual and tactile features have high similarity, the weight factor β tends to a higher value, indicating that the fused features after mapping combine the common characteristics of the two to a greater extent, enhancing information compatibility. When the similarity is low, β reduces the fusion dependency between the two, and the important information unique to either modality is not ignored. This dynamic adjustment enables the generator to more accurately map data from two different modalities into a shared semantic space, thereby preserving the unique features of each modality that are crucial for understanding the overall dynamic behavior.

[0105] The above steps of multimodal data fusion processing can extract the visual and tactile features of the dynamic object to be grasped. An adversarial training mechanism can be considered to achieve effective mapping between visual and tactile feature data, forming a fused feature vector that reflects the object's state. Specifically:

[0106] In one implementation, such as Figure 3 As shown, step 2 above, which uses a region-proposal-based target detection algorithm to identify the dynamic object to be grasped based on multimodal fusion data, and obtains the position and pose information of the dynamic object to be grasped, may include:

[0107] Based on multimodal fusion data, a selective search algorithm is used to generate candidate region proposals;

[0108] Based on the candidate region proposal, the target detection algorithm is used to identify the target of the dynamic object to be grasped, and the classification result and bounding box coordinate information of the dynamic object to be grasped are obtained.

[0109] Based on the classification results and bounding box coordinate information of the dynamic object to be grasped, the pose of the dynamic object to be grasped is estimated to obtain the position and pose information of the dynamic object to be grasped.

[0110] In this implementation, the selective search algorithm generates candidate region proposals on multimodal fusion data, which reduces the computational burden in complex scenes and prioritizes regions that may contain target objects, thereby optimizing the subsequent target detection process. This algorithm adopts an image segmentation-based method, which can significantly reduce the number of candidate regions while maintaining high proposal quality, allowing the target detection algorithm to focus on the most relevant parts. This focusing strategy helps improve computational efficiency, reduces the risk of false recognition, and speeds up the overall processing speed. Target detection algorithms (such as Fast R-CNN) used in target recognition utilize these candidate regions for accurate object identification and classification, generating classification results and bounding box coordinates for each potential object. The core of this process lies in accurately identifying the type of object, thereby quickly providing the necessary data for the grasping strategy. Especially in dynamic environments, objects may move with different postures and speeds, so target detection algorithms need to have high accuracy and real-time response capabilities. Once the target is correctly identified, it will further provide a reliable basis for subsequent grasping operations, laying the foundation for robots to achieve more automated grasping tasks. When a robot grasps dynamic objects, it needs to ensure that it can not only identify the object but also grasp it in the correct way. Therefore, it is necessary to perform posture estimation on the dynamic object (consider 3D keypoint detection, global minimization methods, or deep learning-based posture regression, etc.). Because objects often change position and shape in actual operation, accurate posture information can help the robot adjust its grasping strategy, thereby minimizing the grasping error rate and improving the grasping success rate. Therefore, this implementation method does not stop at the theoretical or single-modal data analysis level, but breaks through the limitations of traditional technology in dealing with dynamic environments. It makes full use of various types of information to enhance the judgment and execution capabilities, demonstrating the improvement of new multimodal data fusion and real-time response capabilities, and ensuring that the robot can achieve efficient and accurate grasping in unpredictable environments.

[0111] The above steps enable target recognition based on multimodal fusion data, obtaining the position and pose information of the dynamic object to be grasped. For further research on the object's dynamic characteristics and behavior prediction, it is possible to combine Long Short-Term Memory networks and Conditional Random Fields to predict the trajectory of the dynamic object, enabling the robot to accurately identify the target to be grasped in complex environments. Specifically:

[0112] In one implementation, such as Figure 4 As shown, step 3 above, which involves predicting the state of the dynamic object to be grasped based on its position and pose information, can include the following steps:

[0113] An autoencoder is used to extract state features from the position and pose information to obtain a fused feature representation of the dynamic object to be grasped.

[0114] Based on the fused feature representation of the dynamic object to be grasped, the trajectory prediction of the dynamic object to be grasped is performed by using a long short-term memory network combined with a conditional random field, thus obtaining the motion trajectory prediction information of the dynamic object to be grasped.

[0115] Based on the motion trajectory prediction information of the dynamic object to be grasped, a deep reinforcement learning algorithm is used to predict the state of the dynamic object to be grasped, and the state prediction information of the dynamic object to be grasped is obtained.

[0116] This implementation achieves precise dynamic behavior understanding and control by predicting the state of the dynamic object to be grasped. The key lies in the effective combination of autoencoders, Long Short-Term Memory (LSTM) networks, Conditional Random Fields (CRFs), and deep reinforcement learning. In this implementation, the autoencoder is used to extract state features from position and pose information. It transforms high-dimensional input data into low-dimensional feature representations, extracting the most crucial information and eliminating redundancy. This approach not only effectively reduces the complexity of the input data but also enhances the focus and effectiveness of the subsequent model's feature representation, making the state features more representative and practical, thus laying the foundation for subsequent data analysis and processing. After extracting the fused features, trajectory prediction is performed using LSTM combined with CRFs. LSTM networks focus on processing time-series data and can capture long- and short-term dependencies in dynamic changes, allowing the model to remember previous states and predict future trajectories. Simultaneously, the CRF provides contextual information enhancement, resulting in higher coherence and accuracy in trajectory prediction. This method improves the predictive ability for complex dynamic behaviors, better reflecting the laws of object motion and providing a practical trajectory reference for dynamic grasping. After trajectory prediction, deep reinforcement learning is introduced for state prediction. Utilizing interaction with the environment and a feedback mechanism to optimize the decision-making process, it can evaluate the impact and results of different strategies on the grasped object. This makes state prediction not merely a static result, but a dynamic and adaptive process. Specifically, the deep reinforcement learning model continuously learns and adjusts its strategy, updating the grasping strategy in real time based on the current state and predicted trajectory, thus more effectively responding to dynamic changes. This adaptive capability significantly improves the robot's operational ability in complex environments, making state prediction not only reflect existing patterns but also possess predictability and flexibility, significantly enhancing the success rate and efficiency of grasping. In real-world environments, the states of objects are highly variable, and single technical means cannot meet practical needs. This approach, through a multi-layered and multi-dimensional processing approach, provides an effective solution for dynamic grasping tasks, highlighting the necessity of synergistic effects of different algorithms and achieving accurate capture and classification of dynamic object behavior.

[0117] Specifically, the process of predicting the motion trajectory of the dynamic object to be grasped by using a long short-term memory network combined with a conditional random field based on the fused feature representation of the dynamic object to be grasped, as described above, may include:

[0118] The feature representation of the fused dynamic object to be captured is transformed into contextual features to obtain feature sequence data;

[0119] The Long Short-Term Memory network is used to perform implicit prediction on the feature sequence data, and the hidden state sequence of the feature sequence data is output.

[0120] By post-processing the hidden state sequence using a conditional random field, we can obtain the motion trajectory prediction information of the dynamic object to be captured.

[0121] In this implementation, during context feature transformation, the contextual information of the time-series data is extracted by transforming the fused feature representation. This process effectively captures the correlation between features, transforming fixed features into a temporally sequential feature sequence. This allows subsequent models to better understand the changing trends of dynamic objects over time. This transformation not only enhances the timeliness and importance of feature data but also provides more valuable input information for deep learning models, improving the overall performance of the model. After obtaining the feature sequence, a Long Short-Term Memory (LSTM) network is used for implicit prediction. Due to its forget gate, input gate, and output gate design, LSTM can effectively learn and remember long-term dependencies. Compared to traditional recurrent neural networks, it is better able to handle longer time-series data. Therefore, when processing the motion of the object to be grasped, LSTM can effectively predict future motion trends based on previous states and output a hidden state sequence. This provides a reliable foundation for trajectory prediction. This stage provides in-depth insights into the behavioral patterns of dynamic objects, enabling the model to better adapt to changing motion situations. This implementation utilizes Conditional Random Fields (CRF). The introduction of Contingent Fields (CRF) allows for post-processing based on the hidden state sequence of LSTM to obtain more accurate trajectory predictions. The power of CRF lies in its ability to consider the contextual relationships between feature sequences. By modeling the preceding and following states, it can effectively correct potential prediction biases in LSTM, enhancing the coherence and accuracy of trajectory prediction. Through feature transfer and correlation analysis of the output sequence, the predicted trajectory is not only affected by a single output but also reflects the overall integrity and consistency of the object's motion state. This post-processing method significantly improves the stability and parsability of motion trajectory prediction, ensuring the reliability of the model in various scenarios. Therefore, this implementation, through the combined use of LSTM and CRF, makes the entire prediction process smoother in dynamic scenes, enabling real-time adjustment and optimization of prediction results, demonstrating high adaptability and practicality, especially in dynamic environments, where it achieves accurate prediction of the motion trajectory of dynamically grasped objects.

[0122] The above steps achieve trajectory prediction for dynamic objects. To further optimize the grasping strategy, scene perception can be achieved by combining environmental information of the dynamic object, and a suitable grasping path for the robot can be generated through reinforcement learning algorithms. Specifically:

[0123] In one implementation, such as Figure 5 As shown, step 4 above, which involves generating the robot's grasping path using a reinforcement learning algorithm based on the predicted state information of the dynamic object to be grasped, may include:

[0124] Based on the state prediction information of the dynamic object to be grasped and the environmental information of the dynamic object to be grasped collected in advance, scene perception is performed on the dynamic object to be grasped to obtain the scene state description information of the dynamic object to be grasped.

[0125] Based on the scene state description information, a dynamic crawling strategy network is constructed using the deep deterministic policy gradient algorithm to obtain the dynamic crawling strategy.

[0126] Based on the dynamic grasping strategy, a path generation algorithm based on imitation learning is used to generate the grasping path of the dynamic object to be grasped.

[0127] In this implementation, during scene perception, the robot compares and analyzes the predicted state information of the dynamic object to be grasped with pre-collected environmental information. This process can employ computer vision technologies, such as object detection and image segmentation, to identify and extract the feature information of objects in the environment and generate a detailed description of the scene state. This perception capability enables the robot to obtain rich environmental information, ensuring that the grasping decision is based on the real situation of the current environment, thereby improving the reliability and success rate of grasping. Through a comprehensive understanding of the environment, it can better adapt to dynamically changing conditions and reduce the grasping risks caused by environmental changes. After obtaining the scene state description information, the model uses the Deep Deterministic Policy Gradient (DDPG) algorithm to construct a dynamic grasping policy network. The advantage of the DDPG algorithm lies in its ability to handle continuous action spaces and optimize policies through deep learning. Specifically, the algorithm gains experience by exploring the environment and selects and evaluates actions through the policy network and value network, effectively improving the efficiency of policy optimization. This process enables the grasping policy to flexibly cope with different grasping environments and object states, raising the quality of grasping decisions to a new level and ensuring that the robot can better perform complex grasping tasks in changing environments. Furthermore, by introducing imitation learning in the path generation process, the core of imitation learning lies in improving learning efficiency by imitating the decision-making process of experts. Specifically, the path generation algorithm learns expert-level path selection from the training set as a reference for generating grasping paths. This design not only shortens the process of the robot learning the optimal path but also greatly reduces unnecessary time and resource waste caused by erroneous exploration. This strategy allows the robot to not only learn how to grasp in ideal conditions but also master effective grasping strategies in complex environments by combining with expert experience. Although some of the technical means in this implementation method are already widely used in the industry, the comprehensive application of these technologies to the specific scenario of dynamic object grasping, by combining scene perception and deep learning, combines the timeliness and accuracy of scene perception with the flexibility of dynamic grasping strategies, forming a holistic dynamic grasping solution. The application of DDPG ensures the continuity and real-time response of strategy optimization, thereby being able to cope with the ever-changing dynamic environment, while imitation learning provides efficient reference for path generation, reduces learning time, and ensures an efficient grasping process.

[0128] In summary, this invention addresses the problem that existing grasping methods can only learn to grasp objects in static environments, and lack the ability to recognize and plan objects in dynamic environments due to the movement, deformation, or occlusion of objects in the environment. To solve this problem, this invention proposes a robot dynamic grasping method based on multimodal data fusion. By fusing visual and tactile data in a multimodal manner, the robot can acquire richer and more comprehensive information, not limited to static features. This fusion enables the robot to more accurately identify and locate the dynamic object to be grasped, thus solving the problem that static grasping methods cannot be effectively applied under dynamic conditions. After acquiring multimodal fusion data, the target detection algorithm based on region proposal further improves the accuracy of target recognition, enabling the robot to clearly extract the position and pose information of objects even in complex and dynamic environments. Due to the complexity of dynamic environments, this step is crucial to ensuring successful grasping. This application predicts the state of the acquired pose information, which can evaluate the motion trajectory and changing trend of objects in real time, providing dynamic support for the grasping strategy. In addition, by introducing reinforcement learning algorithms, the robot can adaptively generate the optimal grasping path in dynamic environments. This mechanism not only makes up for the shortcomings of traditional static methods, but also provides flexibility and real-time performance for the robot's decision-making, ensuring that it can react quickly when facing constantly changing objects.

[0129] Example 2:

[0130] Based on the same inventive concept, this invention also provides a robot dynamic grasping system based on multimodal data fusion, the structural composition of which is shown in the schematic diagram below. Figure 6 As shown, it includes:

[0131] The multimodal fusion module is used to collect visual and tactile data of the dynamic object to be grasped by the robot, and to perform multimodal fusion on the visual and tactile data to obtain multimodal fused data of the dynamic object to be grasped.

[0132] The target recognition module is used to identify the target of the dynamic object to be grasped based on the multimodal fusion data and the target detection algorithm based on region proposal, so as to obtain the position and pose information of the dynamic object to be grasped.

[0133] The state prediction module is used to predict the state of the dynamic object to be grasped based on its position and orientation information, and to obtain the state prediction information of the dynamic object to be grasped.

[0134] The path generation module is used to generate the robot's grasping path based on the state prediction information of the dynamic object to be grasped, using a reinforcement learning algorithm.

[0135] In one implementation, the aforementioned multimodal fusion module may include:

[0136] The visual feature extraction submodule is used to extract features from visual data using the visual feature selection method to obtain visual feature data.

[0137] The tactile feature extraction submodule is used to extract features from tactile data using a one-dimensional convolutional neural network to obtain tactile feature data;

[0138] The cross-modal mapping submodule is used to perform cross-modal mapping on visual feature data and tactile feature data using an adversarial training mechanism to obtain the mapped fused feature vector;

[0139] The data fusion submodule is used to output multimodal fusion data of the dynamic object to be captured based on the mapped fusion feature vector;

[0140] The visual data includes one or more of the following: color image data, depth image data, optical flow data, and ambient lighting data.

[0141] Tactile data includes one or more of the following: robot force feedback data, robot torque data, grasping pressure data, object surface friction data, and object hardness data.

[0142] For example, the expression for the adversarial training mechanism described above can be as follows:

[0143] F fused =G(F visual ;θ g )+βH(F haptic ;θ h );

[0144] In the formula,

[0145]

[0146] Among them, F fused G represents the fused feature vector after mapping; F represents the generator model; visual Represents visual feature data; θ g The parameters of the generator model are represented by β; the dynamic weighting factor is represented by H; the nonlinear transformation model is represented by F. haptic Represents tactile feature data; θ h The parameters represent the nonlinear transformation model; Similarity represents the similarity score.

[0147] In one implementation, the target recognition module described above may include:

[0148] The candidate region generation submodule is used to generate candidate region proposals based on multimodal fusion data using a selective search algorithm.

[0149] The object detection submodule is used to identify the dynamic object to be grasped based on the candidate region proposal and the object detection algorithm, so as to obtain the classification result and bounding box coordinate information of the dynamic object to be grasped.

[0150] The pose estimation submodule is used to estimate the pose of the dynamic object to be grasped based on the classification results and bounding box coordinate information, and obtain the position and pose information of the dynamic object to be grasped.

[0151] In one implementation, the aforementioned state prediction module may include:

[0152] The feature representation submodule is used to extract state features from position and pose information using an autoencoder to obtain the fused feature representation of the dynamic object to be grasped.

[0153] The trajectory prediction submodule is used to predict the trajectory of the dynamic object to be grasped by combining the feature representation of the fused dynamic object with a long short-term memory network and a conditional random field, based on the feature representation of the dynamic object to be grasped, and to obtain the motion trajectory prediction information of the dynamic object to be grasped.

[0154] The state estimation submodule is used to predict the state of the dynamic object to be grasped by using a deep reinforcement learning algorithm based on the motion trajectory prediction information of the dynamic object to be grasped, and obtain the state prediction information of the dynamic object to be grasped.

[0155] In this implementation, the trajectory prediction submodule described above may include:

[0156] The feature transformation unit is used to perform context feature transformation on the feature representation of the fused dynamic object to be captured, so as to obtain feature sequence data.

[0157] The hidden prediction unit is used to perform hidden prediction on feature sequence data using a long short-term memory network and output the hidden state sequence of the feature sequence data.

[0158] The post-processing unit is used to post-process the hidden state sequence through a conditional random field to obtain the motion trajectory prediction information of the dynamic object to be captured.

[0159] In one implementation, the path generation module described above may include:

[0160] Based on the state prediction information of the dynamic object to be grasped and the environmental information of the dynamic object to be grasped collected in advance, scene perception is performed on the dynamic object to be grasped to obtain the scene state description information of the dynamic object to be grasped.

[0161] Based on the scene state description information, a dynamic crawling strategy network is constructed using the deep deterministic policy gradient algorithm to obtain the dynamic crawling strategy.

[0162] Based on the dynamic grasping strategy, a path generation algorithm based on imitation learning is used to generate the grasping path of the dynamic object to be grasped.

[0163] Example 3:

[0164] like Figure 7 As shown, the present invention also provides an electronic device, which may be a computer device, a microcontroller device, a smart mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, processor, and transceiver component are connected via a bus; the memory can be used to store executable programs, and an exemplary executable program may include instructions; the processor is used to execute the instructions stored in the memory. The memory can also be used to store data, which can be accessed and / or modified when instructions are executed.

[0165] The processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, and it is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the storage medium to realize the corresponding method flow or corresponding function, so as to realize the steps of the robot dynamic grasping method based on multimodal data fusion in the above embodiments.

[0166] Example 4:

[0167] Based on the same inventive concept, this invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory). An electronic device readable storage medium is a memory device within an electronic device used to store programs and data. It is understood that the storage medium here can include both built-in storage media within the electronic device and extended storage media supported by the electronic device. The storage medium provides storage space, which stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more executable programs (including program code). It should be noted that the storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. Loading and executing one or more instructions stored in the storage medium by the processor can implement the steps of a robot dynamic grasping method based on multimodal data fusion in the above embodiments.

[0168] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0169] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending approval.

Claims

1. A robot dynamic grasping method based on multi-modal data fusion, characterized in that, The method comprises the following steps: Collecting visual data and tactile data of a dynamic object to be grabbed by a robot, and performing multi-modal fusion on the visual data and the tactile data to obtain multi-modal fusion data of the dynamic object to be grabbed; According to the multi-modal fusion data, a candidate region proposal is generated by a selective search algorithm; According to the candidate region proposal, a target detection algorithm is used to identify the target of the dynamic object to be grabbed, to obtain the classification result and the bounding box coordinate information of the dynamic object to be grabbed; According to the classification result and the bounding box coordinate information of the dynamic object to be grabbed, the pose of the dynamic object to be grabbed is estimated to obtain the position and pose information of the dynamic object to be grabbed; Using a self-encoder to extract state features from the position and pose information to obtain a feature representation of the dynamic object to be grabbed after fusion; According to the feature representation of the dynamic object to be grabbed after fusion, a long short-term memory network combined with a conditional random field is used to predict the trajectory of the dynamic object to be grabbed to obtain motion trajectory prediction information of the dynamic object to be grabbed; According to the motion trajectory prediction information of the dynamic object to be grabbed, a deep reinforcement learning algorithm is used to predict the state of the dynamic object to be grabbed to obtain state prediction information of the dynamic object to be grabbed; According to the state prediction information of the dynamic object to be grabbed and the pre-collected environmental information of the dynamic object to be grabbed, the scene of the dynamic object to be grabbed is perceived to obtain scene state description information of the dynamic object to be grabbed; According to the scene state description information, a deep deterministic policy gradient algorithm is used to construct a dynamic grabbing strategy network to obtain a dynamic grabbing strategy; According to the dynamic grabbing strategy, a path generation algorithm based on imitation learning is used to generate a path to obtain a grabbing path of the dynamic object to be grabbed.

2. The method of claim 1, wherein, The multi-modal fusion of the visual data and the tactile data to obtain the multi-modal fusion data of the dynamic object to be grabbed comprises: Extracting visual features from the visual data by a visual feature selection method to obtain visual feature data; Extracting tactile features from the tactile data by a one-dimensional convolutional neural network to obtain tactile feature data; Using an adversarial training mechanism to perform cross-modal mapping on the visual feature data and the tactile feature data to obtain a mapped fusion feature vector; Outputting the multi-modal fusion data of the dynamic object to be grabbed according to the mapped fusion feature vector; The visual data comprises one or more of the following: color image data, depth image data, optical flow data, and ambient light data; The tactile data comprises one or more of the following: robot force feedback data, robot torque data, grabbing pressure data, object surface friction data, and object hardness data.

3. The method of claim 2, wherein, The expression of the adversarial training mechanism is as follows: ； In the formula, ； wherein, denotes the mapped fused feature vector; denotes the generator model; denotes the visual feature data; denotes the parameters of the generator model; denotes the dynamic weight factor; denotes the nonlinear transformation model; denotes the haptic feature data; denotes the parameters of the nonlinear transformation model; denotes the similarity score.

4. The method of claim 1, wherein, The trajectory prediction of the dynamic object to be grabbed using a long short-term memory network combined with a conditional random field according to the feature representation of the dynamic object to be grabbed after fusion comprises: The feature representation of the dynamic object to be grabbed after fusion is subjected to context feature conversion to obtain feature sequence data; The feature sequence data is subjected to hidden prediction by using a long short-term memory network to output a hidden state sequence of the feature sequence data; The hidden state sequence is subjected to post-processing by using a conditional random field to obtain motion trajectory prediction information of the dynamic object to be grabbed.

5. A robotic dynamic grasping system employing the method of any of claims 1-4, characterized in that, It comprises: A multi-modal fusion module is configured to collect visual data and tactile data of a dynamic object to be grabbed by a robot, and perform multi-modal fusion on the visual data and the tactile data to obtain multi-modal fusion data of the dynamic object to be grabbed; A target identification module is configured to perform target identification on the dynamic object to be grabbed by using a target detection algorithm based on region proposal according to the multi-modal fusion data to obtain position and posture information of the dynamic object to be grabbed; A state prediction module is configured to perform state prediction on the dynamic object to be grabbed based on the position and posture information to obtain state prediction information of the dynamic object to be grabbed; A path generation module is configured to generate a grabbing path of the robot by using a reinforcement learning algorithm according to the state prediction information of the dynamic object to be grabbed.

6. An electronic device, comprising: It comprises: At least one processor and a memory; The memory and the processor are connected through a bus; The memory is configured to store one or more programs; When the one or more programs are executed by the at least one processor, a multi-modal data fusion based robot dynamic grabbing method as claimed in any one of claims 1 to 4 is implemented.

7. A computing device readable storage medium characterized by, An execution program is stored thereon, and when the execution program is executed, a multi-modal data fusion based robot dynamic grabbing method as claimed in any one of claims 1 to 4 is implemented.

Images