Dynamic assembly line object grabbing method and system based on multi-modal information fusion

Abstract

The invention discloses a dynamic assembly line object grabbing method and system based on multi-modal information fusion, and belongs to the technical field of industrial robot grabbing control, and the method comprises the steps: synchronously collecting the visual information of a dynamic assembly line, the joint motion information of a robot and the touch information of a dexterous finger end, and obtaining a standardized multi-source information data set; constructing a coarse grabbing action generation sub-model and a fine grabbing action generation sub-model; the coarse grabbing action generation sub-model obtains a coarse grabbing action sequence through a de-noising diffusion implicit model; according to the coarse grabbing action sequence and the touch information, a fine grabbing action sequence is obtained through a fine grabbing action generation sub-model; adopting a self-supervised learning mode to train the two models in stages; and based on the trained model, a final fine grabbing action sequence is generated, and the robot is driven to execute dynamic grabbing. The core problems that in the prior art, the approaching precision is insufficient, the grabbing stability is poor, the model adaptability is weak, and a control closed loop is lacked are effectively solved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of robotics, artificial intelligence and industrial automation. Specifically, it belongs to the field of industrial robot grasping and control technology, and particularly relates to a dynamic assembly line object grasping method and system based on multimodal information fusion. Background Technology

[0002] In industrial automation, logistics sorting, and intelligent manufacturing, the precise and stable grasping of moving objects on dynamic production lines by robots is crucial for achieving efficient production. This scenario requires robots to respond in real-time to the continuous movement of target objects (speed range 0.05m / s-0.8m / s), the diversity of their shapes and materials, and environmental interference such as changes in lighting and occlusion. The core challenge lies in simultaneously ensuring high precision (positioning error ≤5mm) when the end effector approaches the target and high stability (no slippage, no damage) after grasping.

[0003] Currently, although robot dynamic grasping technology has evolved from "single perception" to "multimodal perception + adaptive action", significant bottlenecks still exist:

[0004] First, the multimodal information perception and fusion capabilities are insufficient. Existing solutions mostly rely on vision as the primary sensor, with other sensors playing a secondary role, lacking deep collaboration. Visual perception is susceptible to interference from lighting and occlusion, and there are inter-frame matching delays when objects are moving at high speeds; joint and tactile perception are mostly used for post-event state monitoring or grip strength adjustment, failing to proactively integrate into motion planning. In addition, the different acquisition frequencies of different modal sensors and the lack of an effective synchronization mechanism lead to temporal misalignment of multi-source data, limiting the overall perception accuracy.

[0005] Secondly, dynamic grasping action generation strategies have weak adaptability. Model-based methods (such as model predictive control) rely heavily on preset environmental parameters and dynamic models, making it difficult to adapt to dynamic changes such as sudden changes in pipeline speed, resulting in action generation lag and proximity deviation. Learning-based methods (such as reinforcement learning) mostly adopt end-to-end training, lacking hierarchical optimization design for different stages of "rapid approach" and "fine grasping", resulting in insufficient interpretability and specificity.

[0006] Finally, there is a lack of efficient model training and a closed-loop control mechanism. Mainstream supervised learning relies on massive amounts of manually labeled data, which is costly and difficult to cover all perturbation scenarios; while self-supervised learning is often limited to a single modality, resulting in insufficient model generalization ability and robustness. At the same time, most systems adopt open-loop control of "perception-planning-execution", which cannot make online action adjustments based on real-time tactile feedback, joint errors, etc. during the grasping process, and has poor fault tolerance when facing sudden disturbances.

[0007] In summary, existing technologies, due to their shortcomings in multimodal fusion, hierarchical motion generation, efficient training, and closed-loop control, struggle to achieve both high precision and high stability in dynamic grasping, thus hindering their large-scale application in complex industrial scenarios.

[0008] Therefore, there is an urgent need for a solution that can achieve deep multimodal fusion, hierarchical action generation, efficient self-supervised training, and closed-loop control capabilities. Summary of the Invention

[0009] To address the aforementioned technical problems, this invention provides a dynamic pipeline object grasping method and system based on multimodal information fusion. This method enables the direct prediction of high-precision and high-stability grasping action sequences from historical multimodal observation data, without relying on explicit object detection, pose estimation, or pipeline velocity measurement. It also possesses strong generalization ability and real-time performance.

[0010] This invention provides a dynamic pipeline object grasping method based on multimodal information fusion, comprising the following steps:

[0011] Simultaneously collect visual information from the dynamic pipeline, joint motion information from the robot, and tactile information from the dexterous fingertips, and obtain a standardized multi-source information dataset after preprocessing;

[0012] Based on the standardized multi-source information dataset, a coarse-grabbing action generation sub-model and a fine-grabbing action generation sub-model are constructed.

[0013] The coarse grasping action generation sub-model takes visual information and joint motion information from a standardized multi-source information dataset as input, and obtains a coarse grasping action sequence for dynamic targets through a denoising diffusion implicit model.

[0014] Based on the coarse grasping action sequence and tactile information in the standardized multi-source information dataset, a fine grasping action generation sub-model is used to perform cross-modal fusion and fine-tuning to obtain a fine grasping action sequence.

[0015] A self-supervised learning approach is adopted to train the coarse grasping action generation sub-model and the fine grasping action generation sub-model in stages.

[0016] Based on the trained coarse grasping action generation sub-model and the trained fine grasping action generation sub-model, the final fine grasping action sequence is generated to drive the robot to perform dynamic grasping operations.

[0017] Optionally, the process of simultaneously acquiring dynamic pipeline visual information, robot joint motion information, and dexterous fingertip tactile information, and obtaining a standardized multi-source information dataset after preprocessing includes:

[0018] The visual information of the dynamic pipeline is deblurred, region of interest is clipped, and mean-variance is standardized.

[0019] The robot's joint motion information and tactile information from the dexterous fingertips are smoothed and their maximum-minimum values ​​are standardized.

[0020] It also performs time synchronization and outlier removal for information from all modalities.

[0021] Optionally, the process by which the coarse grasping action generation sub-model takes visual information and joint motion information from a standardized multi-source information dataset as input, and obtains a coarse grasping action sequence for dynamic targets through a denoising diffusion implicit model includes:

[0022] Based on the visual information, image features are extracted by an image encoder, and dynamic change features are extracted based on the differences between consecutive image frames;

[0023] Based on the joint motion information, joint motion features are extracted using a joint feature encoder.

[0024] The image features, dynamic change features, and joint motion features are spliced ​​and fused to obtain a feature embedding that integrates multi-source information.

[0025] Based on the feature embedding, scaling factors and offset factors are generated through a linear modulation network;

[0026] Based on the scaling factor and offset factor, the intermediate layer features of the denoising diffusion implicit model are linearly modulated channel by channel.

[0027] Based on the modulated denoising diffusion implicit model, the noisy action features are backsampled in an iterative denoising manner to obtain the denoised action features.

[0028] Based on the denoised action features, the coarse grasping action sequence is decoded and generated.

[0029] Optionally, the process of obtaining a fine grasping action sequence by performing cross-modal fusion and fine-tuning through a fine grasping action generation sub-model based on the coarse grasping action sequence and tactile information in the standardized multi-source information dataset includes:

[0030] Based on the tactile information, tactile features are extracted through a tactile embedding network to obtain a tactile feature vector;

[0031] Based on the coarse grasping action sequence, the action feature sequence is obtained by encoding it through an action feature encoder;

[0032] Using the tactile feature vector as the query vector and the action feature sequence as the key vector and value vector, cross-modal fusion is performed through a multi-head attention mechanism to obtain the fused intermediate action features.

[0033] The fused intermediate action features are input into the action refinement network to fine-tune the coarse grasping action sequence frame by frame, thereby obtaining the fine grasping action sequence.

[0034] Optionally, the process of training the coarse grasping action generation sub-model and the fine grasping action generation sub-model in stages using a self-supervised learning approach includes:

[0035] Based on the real grasping data of objects with different movement speeds, shapes and materials collected by the teleoperated robot;

[0036] The training set is divided into a dataset consisting of multimodal observation data and corresponding real-world capture data action sequences according to a preset ratio.

[0037] Based on the visual and joint motion information in the training set, and using the sum of action sequence prediction error and pipeline speed estimation error as the dual loss function, the coarse grasping action generation sub-model is jointly optimized.

[0038] Based on the tactile information in the training set and the coarse motion sequence generated by the coarse grasping motion generation sub-model, and using the sum of motion fine-tuning error and grasping stability loss as the dual loss function, the fine grasping motion generation sub-model is jointly optimized.

[0039] The present invention also provides a dynamic pipeline object grasping system based on multimodal information fusion, for implementing the method, including:

[0040] The multimodal data processing module is used to synchronously collect visual information of the dynamic pipeline, joint motion information of the robot, and tactile information of the dexterous fingertips, and preprocess the collected information to output a standardized multi-source information dataset.

[0041] A hierarchical action generation module, connected to the multimodal data processing module, is used to receive the standardized multi-source information dataset and construct a coarse-grabbing action generation sub-model and a fine-grabbing action generation sub-model.

[0042] The coarse grasping action generation module is used to generate a sequence of coarse grasping actions for dynamic targets by taking visual information and joint motion information as input and using a denoising diffusion implicit model.

[0043] The fine grasping action generation module is used to perform cross-modal fusion and fine-tuning based on the coarse grasping action sequence and tactile information to generate a fine grasping action sequence.

[0044] The self-supervised training module is used to train the coarse grasping action generation sub-model and the fine grasping action generation sub-model in stages using a self-supervised learning approach.

[0045] The closed-loop control execution module, connected to the hierarchical action generation module, is used to generate the final fine grasping action sequence based on the trained coarse grasping action generation sub-model and the trained fine grasping action generation sub-model, driving the robot to perform dynamic grasping operations.

[0046] Optionally, the multimodal data processing module is used to deblur, crop the region of interest, and standardize the mean-variance of the visual information; to smooth the joint motion information and tactile information and standardize the maximum-minimum values; and to synchronize all information in time and remove outliers.

[0047] Optionally, the coarse grasping action generation module includes:

[0048] A multi-source feature embedding unit is used to extract image features and dynamic change features based on visual information, and to extract joint motion features based on joint motion information, and to splice and fuse the features into a feature embedding.

[0049] A linear modulation unit is used to generate a scaling factor and an offset factor based on the feature embedding.

[0050] The denoising diffusion implicit action generation unit, whose internal features are modulated by the scaling factor and offset factor, is used to generate the coarse grasping action sequence in an iterative denoising manner.

[0051] Optionally, the fine-grabbing action generation module includes:

[0052] A tactile feature embedding unit is used to extract tactile feature vectors based on tactile information;

[0053] An action feature encoding unit is used to encode the coarse grasping action sequence to obtain an action feature sequence;

[0054] A cross-modal attention fusion unit is used to fuse the tactile feature vector as a query and the action feature sequence as a key and value through a multi-head attention mechanism to output intermediate action features;

[0055] The motion fine-tuning unit is used to fine-tune the coarse grasping motion sequence frame by frame based on the intermediate motion features to generate the fine grasping motion sequence.

[0056] Optionally, the self-supervised training module includes:

[0057] The data partitioning unit is used to divide the dataset consisting of multimodal observation data and corresponding real grasping data action sequences into a training set according to a preset ratio, based on the real grasping data collected by the teleoperated robot for objects with different movement speeds, shapes and materials.

[0058] The first model training unit, connected to the data partitioning unit, is used to optimize the parameters of the coarse grasping action generation sub-model based on the visual information and joint motion information in the training set, with the goal of minimizing the sum of the mean square error of action sequence prediction and the absolute error of pipeline velocity estimation.

[0059] The second model training unit, connected to the data partitioning unit and the first model training unit, is used to optimize the parameters of the fine grasping action generation sub-model based on the tactile information in the training set and the coarse action sequence generated by the optimized coarse grasping action generation sub-model, with the goal of minimizing the sum of the fine action fine-tuning mean square error and the grasping stability cross-entropy loss.

[0060] Compared with the prior art, the present invention has the following advantages and technical effects:

[0061] This invention effectively overcomes the limitations of a single perception modality by synchronizing and fusing visual, joint motion, and tactile information at the time and feature level, improving the perception robustness and positioning accuracy of dynamic targets under complex lighting, occlusion, and rapid movement conditions. By constructing a hierarchical motion generation model of "coarse grasping – fine gripping," a diffusion model is used to generate coarse motion sequences adapted to pipeline speeds, and a tactile attention mechanism is introduced for fine-tuning. This achieves adaptation to the motion trajectory of dynamic targets and the material and shape of objects, balancing trajectory tracking accuracy in the approach phase and contact stability in the gripping phase. Through self-supervised learning, multimodal data acquired via teleoperation is used for phased training, reducing reliance on large-scale manually labeled data and improving the model's generalization ability in diverse objects and dynamic scenes. By establishing a closed-loop control architecture of "perception-generation-execution-feedback," the system can adjust its movements online based on real-time tactile and joint information, significantly enhancing the grasping success rate and system fault tolerance when facing sudden disturbances such as pipeline vibration and object slippage.

[0062] In summary, this invention ultimately achieves high-precision and high-stability grasping of objects with various speeds, shapes, and materials in dynamic pipeline scenarios, effectively solving the core problems of insufficient proximity accuracy, poor grasping stability, weak model adaptability, and lack of control closed loop in existing technologies. Attached Figure Description

[0063] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0064] Figure 1 This is a schematic diagram of the overall architecture of the dynamic object grasping system provided in an embodiment of the present invention;

[0065] Figure 2This is a schematic diagram of a multimodal data acquisition system provided in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the multimodal data preprocessing process provided in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram of the overall structure of the layered action generation model provided in an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram of the process for acquiring training datasets via teleoperation, provided in an embodiment of the present invention.

[0069] Figure 6 This is a schematic diagram of the coarse grasping action generation sub-model structure provided in an embodiment of the present invention;

[0070] Figure 7 This is a schematic diagram of the fine grasping action generation sub-model structure provided in an embodiment of the present invention;

[0071] Figure 8 This is a schematic diagram of the model reasoning and closed-loop control process provided in an embodiment of the present invention. Detailed Implementation

[0072] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0073] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0074] Traditional dynamic grasping methods suffer from the following drawbacks: 1. Reliance on explicit perception: Object category, pose, and velocity need to be identified in advance, and error accumulation leads to grasping failure; 2. Lagging motion planning: Based on model predictive control (MPC) or trajectory optimization methods, computation is complex and it is difficult to meet real-time requirements; 3. Poor grasping stability: It does not make full use of tactile feedback and is difficult to adapt to uncertainties such as object material, shape, and sliding; 4. Difficulty in data acquisition: Manual annotation is costly in dynamic scenes, making it difficult to build large-scale, high-quality datasets.

[0075] Example 1

[0076] To address the aforementioned issues, this embodiment provides a dynamic pipeline object grasping method based on multimodal information fusion, comprising the following steps:

[0077] Simultaneously collect visual information from the dynamic pipeline, joint motion information from the robot, and tactile information from the dexterous fingertips, and obtain a standardized multi-source information dataset after preprocessing;

[0078] Based on the standardized multi-source information dataset, a coarse-grabbing action generation sub-model and a fine-grabbing action generation sub-model are constructed.

[0079] The coarse grasping action generation sub-model takes visual information and joint motion information from a standardized multi-source information dataset as input, and obtains a coarse grasping action sequence for dynamic targets through a denoising diffusion implicit model.

[0080] Based on the coarse grasping action sequence and tactile information in the standardized multi-source information dataset, a fine grasping action generation sub-model is used to perform cross-modal fusion and fine-tuning to obtain a fine grasping action sequence.

[0081] A self-supervised learning approach is adopted to train the coarse grasping action generation sub-model and the fine grasping action generation sub-model in stages.

[0082] Based on the trained coarse grasping action generation sub-model and the trained fine grasping action generation sub-model, the final fine grasping action sequence is generated to drive the robot to perform dynamic grasping operations.

[0083] As a feasible implementation method, this embodiment simultaneously acquires visual images, robot joint states, and tactile data from dexterous fingertips to construct a closed-loop perception system. Then, it constructs a coarse grasping action generation sub-model and a fine grasping action generation sub-model. The coarse action generation model first generates an initial grasping trajectory adapted to the pipeline speed by fusing visual and joint information. The fine action generation model, based on a tactile attention mechanism, further fine-tunes and optimizes the stability of the coarse action by introducing tactile feedback. Finally, the generated action sequence is executed, and the new sensor data acquired during the execution process is sent back to the data acquisition module, forming a self-supervised closed loop.

[0084] The core of this embodiment lies in constructing a hierarchical generation model for dynamic object grasping actions based on multimodal information fusion, and training it through a self-supervised learning method. This enables the final action model to predict accurate and stable dynamic grasping actions from historical observation data, thereby driving the robot to complete accurate grasping in dynamic pipeline scenarios and solving the dual problems of insufficient proximity accuracy and poor grasping stability in dynamic scenarios.

[0085] like Figure 1As shown, in this embodiment, the main objects of the dynamic object autonomous grasping control system are simply described as a multimodal data collection system. This system integrates multimodal information to dynamically grasp action sequences, generating a hierarchical model and the robot itself. These main objects interact with each other. The multimodal data collection system collects scene data from the dynamic pipeline and robot state data, providing data input to the hierarchical model for generating the action sequences required to grasp dynamic objects. The robot executes the actions generated by the action generation model to perform dynamic object grasping operations. The data generated during the robot's operation is then collected by the data collection system and used as input to the action generation model, forming a closed-loop system.

[0086] The feasible process of simultaneously acquiring dynamic pipeline visual information, robot joint motion information, and dexterous fingertip tactile information, and obtaining a standardized multi-source information dataset after preprocessing includes:

[0087] The visual information of the dynamic pipeline is deblurred, region of interest is clipped, and mean-variance is standardized; the joint motion information and tactile information of the robot's dexterous fingers are smoothed and maximum-minimum standardized; and the information of all modalities is synchronized in time and outliers are removed.

[0088] As a feasible implementation method, the data collection system in this embodiment is as follows: Figure 2 As shown, the camera captures the dynamic assembly line scene from a preset perspective above and to the side of the assembly line, obtaining a global image including the object to be grasped, the assembly line conveyor belt, and the robot. Simultaneously, it collects real-time angle data of each joint of the robotic arm and dexterous hand, as well as tactile data from the fingertips of the dexterous hand. These three types of data are collected in real-time at a preset frequency (30Hz), and then processed as follows: Figure 3 The method shown performs time synchronization, data filtering, outlier handling, and standardization on the collected raw information, ultimately obtaining standardized global image sequence data, joint sequence data, and tactile sequence data, which serve as the input for generating a hierarchical model that meets the requirements for dynamically capturing action sequences by fusing multimodal information.

[0089] Furthermore, the visual information is deblurred, the region of interest is cropped, and the mean-variance is standardized; the joint motion information and tactile information are smoothed and the maximum-minimum standard is applied.

[0090] like Figure 4As shown, the hierarchical model for generating dynamic grasping action sequences by fusing multimodal information consists of a coarse grasping action generation model based on imitation learning and a fine grasping action generation model based on tactile information. The coarse grasping action generation model based on imitation learning takes visual and robot joint data from the multimodal standard dataset as input to generate a dynamic grasping coarse action sequence. Then, the generated coarse action sequence is combined with tactile information and input into the fine grasping action generation model to generate a fine and stable dynamic object grasping action.

[0091] Training a hierarchical model for generating dynamic grasping action sequences that integrate multimodal information requires first constructing a rich dataset of "multimodal observation-grasping actions". For example... Figure 5 As shown in the figure, this process illustrates the acquisition of dynamic object grasping datasets via human teleoperation: A teleoperation device, such as a visual teleoperation device, wearable teleoperation device, or isomorphic robotic arm, is selected and connected to the robotic arm (with a dexterous hand) performing the grasping task to achieve real-time control of the robotic arm. Simultaneously, using various sensors (such as multi-view cameras, joint state sensors, and tactile sensing devices), multimodal observation data (covering visual perception, joint motion state, and tactile information) and corresponding robot action commands are collected synchronously and continuously during the robotic arm's grasping of objects on a dynamic assembly line. This ultimately constructs a paired dataset of "multimodal observation data - robot grasping actions," providing basic sample support for the training of hierarchical models.

[0092] The feasible process of the coarse grasping action generation sub-model, which takes visual information and joint motion information from a standardized multi-source information dataset as input, and obtains a coarse grasping action sequence for dynamic targets through a denoising diffusion implicit model, includes:

[0093] Based on the visual information, image features are extracted using an image encoder, and dynamic change features are extracted based on the differences between consecutive image frames. Based on the joint motion information, joint motion features are extracted using a joint feature encoder. The image features, dynamic change features, and joint motion features are concatenated and fused to obtain a feature embedding that integrates multi-source information. Based on the feature embedding, a scaling factor and an offset factor are generated using a linear modulation network. Based on the scaling factor and the offset factor, the intermediate layer features of the denoising diffusion implicit model are linearly modulated channel by channel. Based on the modulated denoising diffusion implicit model, the noisy action features are backsampled using an iterative denoising method to obtain denoised action features. Based on the denoised action features, the coarse grasping action sequence is decoded and generated.

[0094] As a possible implementation method, such as Figure 6As shown, the coarse grasping action generation model based on imitation learning mainly consists of a four-layer core structure: a multi-source information feature embedding module, a linear modulation parameter acquisition module, an action generation module (based on DDIM), and a pipeline velocity estimation module. These modules work together to achieve the entire process of "multi-modal feature fusion → dynamic parameter modulation → action sequence generation → pipeline velocity adaptation."

[0095] The multi-source information feature embedding module consists of three parts: image feature embedding, dynamic feature embedding, and joint embedding. This module synchronously receives the latest time. The former Image observation sequence at time points (length is) Image time series and robot historical action observation sequence (length is) The joint angle time series was used as input to transform it into a latent feature representation. .

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] in, For image encoders (such as CNN networks), the output is an image feature embedding vector. ; For dynamic feature encoders (such as optical flow + CNN network), dynamic changes are captured by calculating the inter-frame differences of image sequences, and dynamic feature embedding vectors are output. ; For joint feature encoders (such as MLP networks), the output is a joint feature embedding vector. ; This represents the feature vector concatenation operation. The final result is a feature embedding that integrates information from multiple sources. And input it into the linear modulation parameter acquisition module.

[0101] The linear modulation parameter acquisition module will use the output of the feature embedding module. With the number of iterations Embedded (Indicates the current denoising iteration number) After concatenation of the encoded vectors, the data is processed through two linear layers ( Generate linear modulation parameters—scaling factor and offset factor The formula is:

[0102] ;

[0103] in , For linear transformation layers, the output is ( (Number of feature channels in the intermediate layer used for linear modulation of the action generation module). (Obtained) Then, the intermediate features of the denoising model in the action generation module are processed. ( Channel-by-channel linear modulation is performed (where the length is the characteristic sequence), and the formula is:

[0104] ;

[0105] in for The Channel characteristics, , for , The Elements. Through this modulation, the motion generation process can dynamically incorporate multi-source information from historical observations (image features, motion features, joint features).

[0106] The coarse dynamic grasping motion generation module generates motion sequences for grasping dynamic targets through diffusion denoising. Because dynamic scenes have high real-time requirements for motion control (needing to manipulate objects on a dynamic pipeline), the motion generation module employs Denoising Diffusion Implicit Models (DDIM) to generate the joint angle motion sequences required for dynamic grasping through iterative denoising. ( (This refers to the duration of the action sequence). The core of DDIM is the backsampling process, which extracts action features from noisy action sequences. (Initially random noise) Gradually denoise to obtain realistic motion features The key formula is:

[0107] ;

[0108] in for Noisy motion characteristics at any given moment; It is a denoising model (UNet network), which takes noisy action features and time steps as input. The output is the predicted noise; ( The coefficients added to the noise at each step satisfy the following conditions: , indicating the level of accumulated noise; This is an adjustable variance parameter (set to 0 in DDIM to achieve deterministic sampling, which greatly accelerates the generation process of grabbing actions). It is standard Gaussian noise (if This introduces randomness to balance the generation of diversity.

[0109] After K iterations, the final denoised action features are obtained. After decoding, the sequence of joint angle movements of the robotic arm and dexterous hand required to grasp dynamic objects on the assembly line is generated. .

[0110] The pipeline velocity estimation module is designed to make the motion generation model more adaptable to pipelines with different speeds. The pipeline velocity estimation module obtains the intermediate layer features of the motion generation model. As input, through the velocity estimation network (Optional methods include Multilayer Perceptron (MLP), Convolutional Neural Networks (CNNs), Attention Mechanisms, etc., to predict the running speed of the pipeline) The formula is:

[0111] ;

[0112] in The selection of a model is determined by a combination of "speed estimation accuracy" and "inference time". For example, MLP is suitable for scenarios with simple scenarios and single speed patterns; attention mechanisms are suitable for complex scenarios with strong pipeline speed time-series dependencies.

[0113] The feasible process of obtaining a fine grasping action sequence by performing cross-modal fusion and fine-tuning through a fine grasping action generation sub-model based on the coarse grasping action sequence and tactile information in the standardized multi-source information dataset includes:

[0114] Based on the tactile information, tactile features are extracted through a tactile embedding network to obtain a tactile feature vector; based on the coarse grasping action sequence, an action feature encoder is used to encode the action feature sequence to obtain an action feature sequence; using the tactile feature vector as a query vector and the action feature sequence as a key vector and value vector, cross-modal fusion is performed through a multi-head attention mechanism to obtain fused intermediate action features; the fused intermediate action features are input into an action refinement network to fine-tune the coarse grasping action sequence frame by frame to obtain the refined grasping action sequence.

[0115] As a feasible implementation method, after the coarse grasping action generation model based on imitation learning generates the joint angle action sequence for coarse dynamic grasping, to further improve the stability of dynamic object grasping, tactile data from a multimodal standard dataset is fused with the coarse action sequence and input into the refined dynamic object grasping action generation model. The specific model structure and data flow details of the refined grasping action generation model are as follows... Figure 7 As shown.

[0116] First, the tactile sequence data (collected by tactile sensors at the tip of dexterous fingers, containing information such as contact force and pressure distribution) in the multimodal dataset is embedded to obtain tactile feature representations, as shown in the following implementation:

[0117] ;

[0118] in, For (length is) (Tactile time series of dexterous fingertips). It is a haptic embedding network, which can be a multilayer perceptron (MLP) or similar, to capture the temporal dynamic features of haptic signals and output haptic embedding vectors. .

[0119] Fine-grasping action generation model with coarse-dynamic grasping action sequence and tactile embedding As input, a cross-modal attention fusion mechanism is used to achieve deep interaction of multimodal information, thereby generating a refined grasping action sequence. Specifically, the coarse action sequence is first processed by an action feature encoder. (Such as MLP or temporal convolutional networks) are converted into a sequence of feature vectors. .

[0120] ;

[0121] in , Number of action feature channels The length of the action sequence.

[0122] To achieve cross-modal information interaction between haptic embedding and motion features, and to learn the fine-grained regulation of motion by haptics, a multi-head attention mechanism is introduced for interaction information fusion, as specifically implemented below:

[0123] ;

[0124] Among them, query vector Embedded by tactile The transformation yields the key vector K and value vector V derived from action features. Obtained by transformation; To avoid gradient anomalies and enhance attention head dimension, multi-head attention is used to generate intermediate motion features that fuse tactile information. .

[0125] The fused intermediate action features The input motion refinement network employs residual networks, Transformer decoders, and other structures to adjust subtle changes in joint angles frame by frame, generating the final refined dynamic grasping motion sequence. each The generation process takes into account the contact state, enabling a more stable grasp of various objects.

[0126] The feasible process of training the coarse grasping action generation sub-model and the fine grasping action generation sub-model in stages using a self-supervised learning approach includes:

[0127] Based on real grasping data of objects with different movement speeds, shapes, and materials collected by the teleoperated robot, a training set is formed by dividing the dataset consisting of multimodal observation data and corresponding real grasping data action sequences according to a preset ratio. Based on the visual information and joint motion information in the training set, and using the sum of action sequence prediction error and pipeline speed estimation error as a dual loss function, a coarse grasping action generation sub-model is jointly optimized. Based on the tactile information in the training set and the coarse action sequences generated by the coarse grasping action generation sub-model, and using the sum of action fine-tuning error and grasping stability loss as a dual loss function, a fine grasping action generation sub-model is jointly optimized.

[0128] Based on the constructed "multimodal observation-grabbing action" dataset, a self-supervised learning framework is used to train the "coarse action generation model" and the "fine action generation model" in stages. The core objective is to minimize the action prediction error and the grasping stability loss. The specific process is as follows:

[0129] (1) Training dataset partitioning: Dynamic object grasping datasets with different speeds (0.05m / s-0.8m / s), different shapes (regular and irregular), and different materials (plastic, metal, and rubber) were collected by teleoperation to cover diverse grasping and contact scenarios. Then, the paired datasets of "multimodal observation data sequence (visual + joint + tactile) - robot action sequence" collected by teleoperation were divided into training set and validation set in a 7:3 ratio.

[0130] (2) Training of the coarse motion generation model (visual + joint information): The parameters of the model (image encoder) are jointly optimized using "motion sequence prediction error + pipeline velocity estimation error" as the dual loss function. Dynamic encoder DDIM denoising network Speed ​​estimation network Among them, action prediction loss. Mean squared error is used to calculate the "coarse action sequence". "and teleoperation real action sequence" The difference lies in the fact that the pipeline speed estimation loss uses absolute error to calculate the predicted speed. Compared to real pipeline speed The difference is calculated using the following formula:

[0131] ;

[0132] ;

[0133] ;

[0134] in This represents the total number of joints in the robotic arm and dexterous hand. The length of the action sequence. To jointly optimize the loss function, This is a scaling factor; by adjusting this parameter, the accuracy of motion prediction and the ability to adapt to dynamic speed can be coordinated.

[0135] (3) Training of refined motion generation model (tactile guidance): Self-supervised contrastive learning is introduced with "coarse motion sequence as input" to optimize the tactile-motion matching relationship. The loss function includes "motion fine-tuning error". "and grip stability loss" "Composition. Fine-tuning error in motion is a component of the precise motion sequence." With remote control and real motion The MSE focuses on the fine-tuning of finger joints after grasping an object, while grasp stability loss... It is based on tactile data to construct "grip stability discrimination labels" (if the tactile pressure distribution is uniform and there is no slip signal, the label is 1; otherwise, it is 0). The cross-entropy loss optimization model is used to learn the stable grip state. The specific calculation formula is as follows:

[0136] ;

[0137] ;

[0138] in For authentic and stable labels, The model predicts the stable gripping classification.

[0139] As a feasible implementation method, the process of dynamic object grasping on the pipeline by using a hierarchical generation model of dynamic grasping action sequences that integrates multimodal information includes:

[0140] like Figure 8 As shown, this example provides a model inference flow for a dynamic object grasping method based on multimodal information fusion, constructing an information transmission mechanism within the robot's actual operation control loop. This inference architecture comprises three core modules: a sensor data acquisition module, a dynamic grasping action generation model and the robot body, and a motion control model. These modules form a closed-loop feedback control mechanism.

[0141] During the reasoning process, data is collected through cameras and the robot's own sensor system. Observational information within the time period The data is fed into a dynamic grasping control model, which predicts the dynamic object grasping requirements. The module generates joint sequences for the robotic arm and dexterous hand over a given time period. These sequences are then input into a motion control model, which generates control commands based on the current pose and the sequence of actions to be performed. This control model then controls the robot's actuators to achieve precise motion control. This module can be built upon classical control methods such as MPC and QP. Finally, the robot executes specific actions based on the control signals, continuously generating new observation information from sensors during execution to re-input to the state estimation model, thus completing the closed-loop control process.

[0142] Through this reasoning structure design, the state estimation model plays a core role in the entire control loop, enabling the robot to complete the generation of the complete action of grasping dynamic objects and to complete the grasping of dynamic objects on the pipeline without the need for dynamic object detection, pose estimation, and pipeline speed estimation.

[0143] Example 2

[0144] This embodiment also provides a dynamic pipeline object grasping system based on multimodal information fusion, used to implement the method, including:

[0145] The multimodal data processing module is used to synchronously collect visual information of the dynamic pipeline, joint motion information of the robot, and tactile information of the dexterous fingertips, and preprocess the collected information to output a standardized multi-source information dataset.

[0146] A hierarchical action generation module, connected to the multimodal data processing module, is used to receive the standardized multi-source information dataset and construct a coarse-grabbing action generation sub-model and a fine-grabbing action generation sub-model.

[0147] The coarse grasping action generation module is used to generate a sequence of coarse grasping actions for dynamic targets by taking visual information and joint motion information as input and using a denoising diffusion implicit model.

[0148] The fine grasping action generation module is used to perform cross-modal fusion and fine-tuning based on the coarse grasping action sequence and tactile information to generate a fine grasping action sequence.

[0149] The self-supervised training module is used to train the coarse grasping action generation sub-model and the fine grasping action generation sub-model in stages using a self-supervised learning approach.

[0150] The closed-loop control execution module, connected to the hierarchical action generation module, is used to generate the final fine grasping action sequence based on the trained coarse grasping action generation sub-model and the trained fine grasping action generation sub-model, driving the robot to perform dynamic grasping operations.

[0151] Implementable, the multimodal data processing module is used to deblur, crop the region of interest and standardize the mean-variance of the visual information, smooth the joint motion information and tactile information and standardize the maximum-minimum value, and synchronize all information in time and remove outliers.

[0152] Implementable, the coarse grasping action generation module includes:

[0153] A multi-source feature embedding unit is used to extract image features and dynamic change features based on visual information, and to extract joint motion features based on joint motion information, and to splice and fuse the features into a feature embedding.

[0154] A linear modulation unit is used to generate a scaling factor and an offset factor based on the feature embedding.

[0155] The denoising diffusion implicit action generation unit, whose internal features are modulated by the scaling factor and offset factor, is used to generate the coarse grasping action sequence in an iterative denoising manner.

[0156] The feasible fine-grabbing action generation module includes:

[0157] A tactile feature embedding unit is used to extract tactile feature vectors based on tactile information;

[0158] An action feature encoding unit is used to encode the coarse grasping action sequence to obtain an action feature sequence;

[0159] A cross-modal attention fusion unit is used to fuse the tactile feature vector as a query and the action feature sequence as a key and value through a multi-head attention mechanism to output intermediate action features;

[0160] The motion fine-tuning unit is used to fine-tune the coarse grasping motion sequence frame by frame based on the intermediate motion features to generate the fine grasping motion sequence.

[0161] Implementable, the self-supervised training module includes:

[0162] The data partitioning unit is used to divide the dataset consisting of multimodal observation data and corresponding real grasping data action sequences into a training set according to a preset ratio, based on the real grasping data collected by the teleoperated robot for objects with different movement speeds, shapes and materials.

[0163] The first model training unit, connected to the data partitioning unit, is used to optimize the parameters of the coarse grasping action generation sub-model based on the visual information and joint motion information in the training set, with the goal of minimizing the sum of the mean square error of action sequence prediction and the absolute error of pipeline velocity estimation.

[0164] The second model training unit, connected to the data partitioning unit and the first model training unit, is used to optimize the parameters of the fine grasping action generation sub-model based on the tactile information in the training set and the coarse action sequence generated by the optimized coarse grasping action generation sub-model, with the goal of minimizing the sum of the fine action fine-tuning mean square error and the grasping stability cross-entropy loss.

[0165] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A dynamic pipeline object grasping method based on multimodal information fusion, characterized in that, Includes the following steps: Simultaneously collect visual information from the dynamic pipeline, joint motion information from the robot, and tactile information from the dexterous fingertips, and obtain a standardized multi-source information dataset after preprocessing; Based on the standardized multi-source information dataset, a coarse-grabbing action generation sub-model and a fine-grabbing action generation sub-model are constructed. The coarse grasping action generation sub-model takes visual information and joint motion information from a standardized multi-source information dataset as input, and obtains a coarse grasping action sequence for dynamic targets through a denoising diffusion implicit model. Based on the coarse grasping action sequence and tactile information in the standardized multi-source information dataset, a fine grasping action generation sub-model is used to perform cross-modal fusion and fine-tuning to obtain a fine grasping action sequence. A self-supervised learning approach is adopted to train the coarse grasping action generation sub-model and the fine grasping action generation sub-model in stages. Based on the trained coarse grasping action generation sub-model and the trained fine grasping action generation sub-model, the final fine grasping action sequence is generated to drive the robot to perform dynamic grasping operations.

2. The method according to claim 1, characterized in that, The process of simultaneously acquiring dynamic pipeline visual information, robot joint motion information, and dexterous fingertip tactile information, and obtaining a standardized multi-source information dataset after preprocessing includes: The visual information of the dynamic pipeline is deblurred, region of interest is clipped, and mean-variance is standardized. The robot's joint motion information and tactile information from the dexterous fingertips are smoothed and their maximum-minimum values ​​are standardized. It also performs time synchronization and outlier removal for information from all modalities.

3. The method according to claim 1, characterized in that, The coarse grasping action generation sub-model takes visual information and joint motion information from a standardized multi-source information dataset as input, and obtains a coarse grasping action sequence for dynamic targets through a denoising diffusion implicit model. The process includes: Based on the visual information, image features are extracted by an image encoder, and dynamic change features are extracted based on the differences between consecutive image frames; Based on the joint motion information, joint motion features are extracted using a joint feature encoder. The image features, dynamic change features, and joint motion features are spliced ​​and fused to obtain a feature embedding that integrates multi-source information. Based on the feature embedding, scaling factors and offset factors are generated through a linear modulation network; Based on the scaling factor and offset factor, the intermediate layer features of the denoising diffusion implicit model are linearly modulated channel by channel. Based on the modulated denoising diffusion implicit model, the noisy action features are backsampled in an iterative denoising manner to obtain the denoised action features. Based on the denoised action features, the coarse grasping action sequence is decoded and generated.

4. The method according to claim 1, characterized in that, Based on the coarse grasping action sequence and tactile information in the standardized multi-source information dataset, the process of obtaining the fine grasping action sequence includes cross-modal fusion and fine-tuning through a fine grasping action generation sub-model, which includes: Based on the tactile information, tactile features are extracted through a tactile embedding network to obtain a tactile feature vector; Based on the coarse grasping action sequence, the action feature sequence is obtained by encoding it through an action feature encoder; Using the tactile feature vector as the query vector and the action feature sequence as the key vector and value vector, cross-modal fusion is performed through a multi-head attention mechanism to obtain the fused intermediate action features. The fused intermediate action features are input into the action refinement network to fine-tune the coarse grasping action sequence frame by frame, thereby obtaining the fine grasping action sequence.

5. The method according to claim 1, characterized in that, The process of training the coarse grasping action generation sub-model and the fine grasping action generation sub-model in stages using a self-supervised learning approach includes: Based on the real grasping data of objects with different movement speeds, shapes and materials collected by the teleoperated robot; The training set is divided into a dataset consisting of multimodal observation data and corresponding real-world capture data action sequences according to a preset ratio. Based on the visual and joint motion information in the training set, and using the sum of action sequence prediction error and pipeline speed estimation error as the dual loss function, the coarse grasping action generation sub-model is jointly optimized. Based on the tactile information in the training set and the coarse motion sequence generated by the coarse grasping motion generation sub-model, and using the sum of motion fine-tuning error and grasping stability loss as the dual loss function, the fine grasping motion generation sub-model is jointly optimized.

6. A dynamic pipeline object grasping system based on multimodal information fusion, characterized in that, For implementing the method according to any one of claims 1-5, comprising: The multimodal data processing module is used to synchronously collect visual information of the dynamic pipeline, joint motion information of the robot, and tactile information of the dexterous fingertips, and preprocess the collected information to output a standardized multi-source information dataset. A hierarchical action generation module, connected to the multimodal data processing module, is used to receive the standardized multi-source information dataset and construct a coarse-grabbing action generation sub-model and a fine-grabbing action generation sub-model. The coarse grasping action generation module is used to generate a sequence of coarse grasping actions for dynamic targets by taking visual information and joint motion information as input and using a denoising diffusion implicit model. The fine grasping action generation module is used to perform cross-modal fusion and fine-tuning based on the coarse grasping action sequence and tactile information to generate a fine grasping action sequence. The self-supervised training module is used to train the coarse grasping action generation sub-model and the fine grasping action generation sub-model in stages using a self-supervised learning approach. The closed-loop control execution module, connected to the hierarchical action generation module, is used to generate the final fine grasping action sequence based on the trained coarse grasping action generation sub-model and the trained fine grasping action generation sub-model, driving the robot to perform dynamic grasping operations.

7. The system according to claim 6, characterized in that, The multimodal data processing module is used to deblur, crop regions of interest, and standardize the mean-variance of the visual information; to smooth and standardize the maximum-minimum values ​​of the joint motion and tactile information; and to synchronize all information in time and remove outliers.

8. The system according to claim 6, characterized in that, The coarse-grabbing action generation module includes: A multi-source feature embedding unit is used to extract image features and dynamic change features based on visual information, and to extract joint motion features based on joint motion information, and to splice and fuse the features into a feature embedding. A linear modulation unit is used to generate a scaling factor and an offset factor based on the feature embedding. The denoising diffusion implicit action generation unit, whose internal features are modulated by the scaling factor and offset factor, is used to generate the coarse grasping action sequence in an iterative denoising manner.

9. The system according to claim 6, characterized in that, The fine grasping action generation module include: A tactile feature embedding unit is used to extract tactile feature vectors based on tactile information; An action feature encoding unit is used to encode the coarse grasping action sequence to obtain an action feature sequence; A cross-modal attention fusion unit is used to fuse the tactile feature vector as a query and the action feature sequence as a key and value through a multi-head attention mechanism to output intermediate action features; The motion fine-tuning unit is used to fine-tune the coarse grasping motion sequence frame by frame based on the intermediate motion features to generate the fine grasping motion sequence.

10. The system according to claim 6, characterized in that, The self-supervised training module includes: The data partitioning unit is used to divide the dataset consisting of multimodal observation data and corresponding real grasping data action sequences into a training set according to a preset ratio, based on the real grasping data collected by the teleoperated robot for objects with different movement speeds, shapes and materials. The first model training unit, connected to the data partitioning unit, is used to optimize the parameters of the coarse grasping action generation sub-model based on the visual information and joint motion information in the training set, with the goal of minimizing the sum of the mean square error of action sequence prediction and the absolute error of pipeline velocity estimation. The second model training unit, connected to the data partitioning unit and the first model training unit, is used to optimize the parameters of the fine grasping action generation sub-model based on the tactile information in the training set and the coarse action sequence generated by the optimized coarse grasping action generation sub-model, with the goal of minimizing the sum of the fine action fine-tuning mean square error and the grasping stability cross-entropy loss.

Images