Method and system for multimodal fusion-based robot grasp failure recognition and regrasp

Abstract

The application discloses a kind of multimodal fusion's robot grasping failure identification and regrasping method and system, it is related to robot grasping and intelligent control technical field, the method includes the following steps: step 1, obtain multimodal original data;Step 2, carry out time alignment and extract multimodal feature;Step 3, multimodal feature is fused using modal reliability weighted fusion mechanism, and the way that sample level discrimination and timing level positioning are combined is used to carry out grasping failure identification, and the grasping failure category and the failure event information corresponding to the grasping failure category are output;Step 4, construct failure feedback vector;Step 5, generate regrasping correction strategy;Step 6, execute regrasping operation;Step 7, whether regrasping operation is successful, if not successful, then return step 1;Step 8, if successful, then end grasping.

Description

Technical Field

[0001] This invention relates to the field of robot grasping and intelligent control technology, and in particular to a method and system for identifying and re-grasping robot grasping failures through multimodal fusion. Background Technology

[0002] Robotic grasping is one of the fundamental tasks in the field of robot perception and manipulation, and it is widely used in industrial sorting, warehouse handling, intelligent manufacturing, service robots, and autonomous operation in complex environments. Existing robotic grasping systems typically detect, locate, and estimate the pose of target objects based on visual perception results, and then control the robotic arm and end effector to complete the grasping operation by combining preset grasping strategies or candidate grasping poses output by grasping networks.

[0003] However, in real-world scenarios, the objects to be grasped often exhibit diverse materials, irregular shapes, severe occlusion, uncertain local contact, and complex environmental interference, leading to frequent grasping failures during robot grasping. These failures include not only complete misses (absence of the target object), but also slippage after gripping, dropping during transport, collisions with the target object or surrounding environment during grasping, destructive grasping due to improper contact position or gripping method, and other abnormal contact situations. These failures significantly reduce the success rate and efficiency of robot grasping tasks, and may even damage the grasped target, the end effector, and the surrounding environment.

[0004] To address the issue of grasping failures, existing technologies typically employ the following approaches: One approach involves a simple retry based solely on a success / failure binary classification result after the grasping process concludes; that is, after determining that the grasping has failed, the original grasping strategy is invoked once or multiple times to perform the grasping. Another approach monitors the grasping status based on a single perceptual modality, such as using visual images to determine whether the target has moved out of its original position, or using end-effector force signals to detect whether abnormal contact has occurred. A third approach corrects some known failure scenarios using pre-set heuristic rules, such as increasing the gripper closure amount when the grip is unstable, or readjusting the grasping point when the target deviates.

[0005] However, existing technologies still have the following shortcomings: First, the granularity of failure perception is relatively coarse. Many existing methods can only determine whether the grasp was successful or only roughly identify a few obvious anomalies, making it difficult to further distinguish between different grasp failure categories such as missing, slipping, falling, collision, destructive grasping, and contact anomalies. Since the causes of different failure categories are different, simple coarse-grained results are insufficient to provide a sufficient basis for subsequent correction.

[0006] Secondly, single-modal perception lacks robustness. The robot grasping process is essentially a dynamic interactive process involving stages such as approach, contact, closing, lifting, and transport. Relying solely on visual information makes it difficult to promptly capture hidden contact anomalies; relying solely on force information makes it difficult to obtain changes in target position and spatial relationships; and relying solely on auditory information is easily interfered with by environmental noise. Current technologies do not adequately utilize the combined use of multimodal information such as vision, force, and hearing, making it difficult to form stable, reliable, and discriminative evidence of grasping failure from heterogeneous perceptual information.

[0007] Secondly, the ability to characterize the failure process is limited. Grasping failures are usually not single, instantaneous results, but rather dynamic events that gradually emerge throughout the entire grasping process. For example, the impact when the grippers close, the slippage signs during lifting, and the unstable contact before falling during the handling phase all have obvious temporal characteristics. Existing technologies often lack the ability to identify the moment of failure, key time periods, key frame locations, or event window ranges, thus failing to effectively characterize the failure process and hindering targeted corrective decisions.

[0008] Furthermore, post-failure correction strategies lack specificity and feasibility. Existing methods often directly regenerate crawling candidates or simply repeat the original actions after a crawling failure, lacking a mechanism to generate structured correction information based on the cause of failure and further transform it into an executable re-crawl correction strategy. In other words, existing technologies typically cannot answer key questions such as "why did it fail," "when and where did the failure occur," and "how should the next crawling action be changed based on the cause of failure?" This results in re-crawls lacking specificity and easily leading to repeated occurrences of the same or similar failures.

[0009] Therefore, those skilled in the art are dedicated to developing a new method and system for identifying and re-grabbing robot grasping failures, in order to address the aforementioned shortcomings in the existing technology. Summary of the Invention

[0010] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to construct a technical solution that can uniformly process multimodal perception information in the robot grasping process, identify different grasping failure categories and their corresponding failure event information, form a failure feedback vector that can characterize the cause of grasping failure and its correction basis, and on this basis generate re-grasping correction strategies such as local contact zone, grasping risk distribution information, candidate grasping constraint information, and grasping parameter increment, thereby improving the success rate of robot grasping tasks, the corrective targeting and the robustness of closed-loop execution.

[0011] To achieve the above objectives, this invention provides a method for multimodal fusion-based robot grasping failure identification and re-grasping, comprising the following steps: Step 1: Acquire multimodal raw data, including visual data, force data, and auditory data; Step 2: Perform time alignment and extract multimodal features, including visual features, force features, and auditory features; Step 3: Use a modal reliability weighted fusion mechanism to perform feature fusion on the multimodal features, and use a combination of sample-level discrimination and temporal-level localization to identify capture failures, and output the capture failure category and the failure event information corresponding to the capture failure category; Step 4: Construct a failure feedback vector, which should include at least the failure category, failure confidence, and modal reliability information; Step 5: Generate a re-grabbing correction strategy, which includes at least one of the following: local contact zone, grabbing risk distribution information, candidate grabbing constraint information, grabbing parameter increment, and structured semantic correction instructions; Step 6: Perform a re-fetch operation; Step 7: Determine whether the re-fetching operation was successful. If it was unsuccessful, return to step 1. Step 8: If successful, end the crawling process; The failure event information in step 3 includes, but is not limited to, failure confidence, modal reliability weight, failure peak time, event window range, keyframe index, and cross-modal conflict degree.

[0012] Further, the visual data in step 1 includes one or more of the following: single-frame images, multi-frame image sequences, color images, and depth images before, during, or after grasping; the force data includes force signals, torque signals, or six-dimensional force / torque timing signals from the robot's end effector during the grasping process; and the auditory data includes audio time-domain signals, frequency-domain signals, or time-spectrum signals during the grasping process.

[0013] Furthermore, the time alignment in step 2 is to perform synchronous alignment of the visual data, force data, and auditory data at different sampling frequencies under a unified time reference; The visual features in the multimodal features are static visual features or temporal visual features extracted from the visual data; the force features are statistical features, rate of change features, contact impact features, or temporal dynamic features extracted from the force data; and the auditory features are Mel-spectral features, short-time Fourier transform features, time-frequency texture features, or combinations thereof extracted from the auditory data.

[0014] Furthermore, the modal reliability weighted fusion mechanism in step 3 includes the following process: Let the multimodal set be ,in, Representing visual modality, Represents force perception mode, Represents auditory modality; for any modality The feature corresponding to this mode is denoted as ; First, the features are gating using a modal feature gate function. The gated modal features are obtained by filtering. : , in, For the first Feature gating vectors corresponding to each modality This represents element-wise multiplication; The feature gate vector We obtain it from the following formula: , in, and For learnable parameters, For the Sigmoid function; Then, the modal features Input the corresponding modal encoder to obtain the modal implicit representation. And the prediction result of the failure category of the corresponding modality is obtained through the corresponding modality classifier: , , in, For the first The mode encoder corresponding to each mode. For the first A modality classifier corresponding to each modality. Indicates the first The failure categories of each modal output are logits; Next, temperature calibration is performed on the failure categories (logits) of each mode to obtain the calibrated mode prediction results: , in, For the first The temperature parameters corresponding to each mode are used to calibrate the prediction confidence of that mode; Subsequently, based on the calibrated modality prediction results, the class probability of the mode for each failure category is calculated using the Softmax function. : , Next, the corresponding modal reliability score is determined based on the category probability of each modality. ; Finally, based on the reliability scores of each mode... Modal reliability weights are obtained through a normalized exponential function. : , in, Indicates the first The reliability weight of each modality in the current crawled sample This is a reliability sharpening factor used to adjust the degree of difference between reliability weights for different modes. For the first The reliability score of each modality in the current crawled sample; The reliability weights for each mode satisfy: .

[0015] Furthermore, the modal reliability score for: , in, Indicates the first The modality pair of the first The predicted probability of each failure category, Indicates the first The reliability score of each modality in the current crawled sample.

[0016] Furthermore, the prediction results of each mode are weighted and fused based on the modal reliability weights to obtain sample-level failure discrimination results: , And according to The maximum probability category output is the category of the failed crawling of the current crawled sample: , in, This refers to the category of failed crawling for the currently crawled sample. The grab failure categories include one or more of the following: missing grab, slipping, falling, collision, destructive grab, and contact anomaly.

[0017] Furthermore, in step 3, the temporal-level localization involves inputting the visual features, force features, and auditory features obtained in step 2 into their respective temporal encoders to obtain the temporal latent representations of each modality at different times. : , in, For the first The timing encoder corresponding to each mode For the first The modality at time... The temporal characteristics; First, for any given moment Frame-level event saliency scores are calculated based on the temporal implicit representations of each modality. , used to indicate the first The modality at time... The degree of contribution to the failure event; Then, based on each mode at time... Calculating cross-modal conflict degree using differential classification probability ; Next, the temporal latent representations of each modality are concatenated or weighted and fused, and then input into the event detection head to obtain the event occurrence probability at each time step. : , in, Indicates the event detection header, Representing visual modalities Force mode and auditory modality At any moment The splicing result of the temporal implicit representation; Then, the probability of the event occurring. Compared with the frame-level event saliency score and the cross-modal conflict degree Together they form a comprehensive event score. : , in, and These are the weighting coefficients; Next, based on the comprehensive event score... The peak time of failure events is determined by the peak time of failure events: , Finally, based on the comprehensive event score... A failure event window is defined as a continuous time period exceeding a preset proportion threshold; wherein, the start and end times of the failure event window are... and satisfy: , in, The failure event window is set to a preset ratio threshold. The peak time of the failure events Together with the keyframe index, they constitute the failure event information.

[0018] Furthermore, Obtained from a frame-level scoring function and normalized over time: , , in, For the first The modality at time... Frame-level rating, and The parameters of the frame-level scoring function are... To The transpose of the corresponding matrix, This represents the time index during the crawling process; For any two modes and Calculate at time The cross-modal conflict degree is obtained by averaging the differences in the category probability distributions of all modal pairs. : , in, Represents the set of all modal pairs. The function representing the difference between two class probability distributions is... distance, distance, divergence or One type of divergence, Indicates the first The modality at time... The predicted probability distribution for each failure category, Indicates the first The modality at time... The predicted probability distribution for each failure category.

[0019] The present invention also provides a multimodal fusion system for robot grasping failure identification and re-grasping, the system comprising: a multimodal data acquisition module, a feature extraction and time alignment module, a grasping failure identification module, a failure feedback vector construction module, a re-grasping strategy generation module, and an execution control module; The multimodal data acquisition module acquires multimodal raw data during the grasping process, including visual data, force data, and auditory data; The feature extraction and time alignment module is connected to the multimodal data acquisition module and is used to preprocess, time align and extract features from the visual data, the force data and the auditory data to obtain visual features, force features and auditory features. The grasping failure identification module is connected to the feature extraction and time alignment module. It is used to fuse the visual features, the force features and the auditory features, and to identify grasping failures by combining sample-level discrimination and temporal-level localization. It outputs the grasping failure category and the failure event information corresponding to the grasping failure category. The failure feedback vector construction module is connected to the crawling failure identification module and is used to construct a failure feedback vector that characterizes the cause of crawling failure and the basis for correction; the failure feedback vector includes at least failure category, failure confidence and modal reliability information; The re-crawl strategy generation module is connected to the failure feedback vector construction module and is used to generate a re-crawl correction strategy based on the failure feedback vector; the re-crawl correction strategy includes at least one of the following: local contact zone, crawling risk distribution information, candidate crawling constraint information, crawling parameter increment, and structured semantic correction instructions; The execution control module is connected to the re-grabbing strategy generation module and is used to control the robot to perform re-grabbing operations according to the re-grabbing correction strategy.

[0020] Furthermore, the system is configured to perform the multimodal fusion method for robot grasping failure identification and re-grasping as described in any of the preceding items.

[0021] The method and system for identifying and re-grabbing robot grasping failures through multimodal fusion provided by this invention have at least the following technical effects: 1. The technical solution provided by this invention can improve the granularity of grasping failure identification. It can not only determine whether grasping has failed, but also distinguish different grasping failure categories such as grasping empty, slipping, falling, collision, destructive grasping and contact abnormality, and output failure event information corresponding to the grasping failure category, thereby providing a more granular basis for subsequent correction.

[0022] 2. The technical solution provided by this invention can enhance the robustness of multimodal failure perception. By jointly utilizing visual, force, and auditory multimodal raw data, through unified time alignment, feature extraction and fusion processing, and by introducing modal reliability information, it can reduce the recognition error caused by occlusion, noise, missing or abnormal effects on a single modality, thereby improving the stability and accuracy of grasping failure recognition.

[0023] 3. The technical solution provided by this invention can characterize the failure process and its causes. By combining sample-level failure discrimination with time-series failure event localization, it can not only output the type of capture failure, but also output information such as the time of failure event occurrence, key time period, key frame position or event window range, thereby more accurately depicting the capture failure process.

[0024] 4. The technical solution provided by this invention can form a structured failure feedback representation. By constructing a failure feedback vector to characterize the reasons for crawling failure and the basis for correction, the crawling failure results are no longer limited to a simple classification level, but are further formed into a structured intermediate representation that can be directly used by the decision-making module. This is beneficial to improving the interpretability, scalability and subsequent strategy generation capabilities of the system.

[0025] 5. The technical solution provided by this invention can improve the pertinence and executability of re-grabbing correction. Based on the failure feedback vector, it generates local contact zones, grasping risk distribution information, candidate grasping constraint information and grasping parameter increments, etc., and can make targeted adjustments to the execution quantities such as gripper opening degree, gripper posture, end approach direction, grasping contact position, grasping feed depth and candidate grasping sorting according to different grasping failure categories, avoiding repeated failures caused by simply repeating the original grasping action.

[0026] 6. The technical solution provided by this invention can support closed-loop re-grasping. After the robot fails to grasp for the first time, it continues to perform grasping failure identification, failure feedback vector construction and re-grasping correction strategy generation based on new multimodal raw data, forming a closed-loop iterative re-grasping mechanism, thereby improving the overall completion rate and execution robustness of grasping tasks in complex scenarios.

[0027] 7. The technical solution provided by this invention has a wide range of applications and can be applied to various robot grasping scenarios such as industrial grasping, warehouse sorting, service robots, and autonomous operation. It has good adaptability and promotional value for target objects with different shapes, materials, contact conditions, and environmental interference.

[0028] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0029] Figure 1 This is a flowchart of a preferred embodiment of the multimodal fusion robot grasping failure identification and re-grasping method of the present invention; Figure 2 This is a schematic diagram of a system structure for multimodal fusion robot grasping failure identification and re-grasping according to a preferred embodiment of the present invention. Detailed Implementation

[0030] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0031] This invention combines visual, force, and auditory data from the robot's grasping process, outputting grasping failure categories and corresponding failure event information by employing a combination of sample-level failure discrimination and time-series failure event localization. Furthermore, it constructs a failure feedback vector to characterize the causes of grasping failures and their correction criteria. Based on this, a re-grasping correction strategy is generated according to the failure feedback vector, controlling the robot to perform targeted re-grasping operations, thereby improving the accuracy of grasping failure perception, the targeted nature of failure correction, and the robustness of grasping task execution.

[0032] Example 1 like Figure 1 As shown in the figure, a method for identifying and re-grabbing robot grasping failures through multimodal fusion, provided by an embodiment of the present invention, includes the following steps: Step 1: Acquire multimodal raw data, including visual data, force data, and auditory data; Step 2: Perform time alignment and extract multimodal features, including visual features, force features, and auditory features; Step 3: Use a modal reliability weighted fusion mechanism to perform feature fusion on multimodal features, and use a combination of sample-level discrimination and temporal-level localization to identify capture failures, and output the capture failure category and the failure event information corresponding to the capture failure category; Step 4: Construct a failure feedback vector, which should include at least the failure category, failure confidence, and modal reliability information; Step 5: Generate a re-grabbing correction strategy, which includes at least one of the following: local contact zone, grabbing risk distribution information, candidate grabbing constraint information, grabbing parameter increment, and structured semantic correction instructions; Step 6: Perform a re-fetch operation; Step 7: Determine if the re-fetching operation was successful. If it was unsuccessful, return to step 1. Step 8: If successful, end the crawling process.

[0033] The failure event information in step 3 includes, but is not limited to, failure confidence, modal reliability weight, failure peak time, event window range, keyframe index, and cross-modal conflict degree.

[0034] Example 2 Based on Example 1, step 1 acquires multimodal raw data of the robot during the grasping process, including at least visual data, force data, and auditory data. Visual data is acquired by RGB-D cameras positioned above or to the side of the grasping scene, including one or more of the following: single-frame images, multi-frame image sequences, color images, and depth images before, during, or after grasping. Force data is acquired by a six-dimensional force / torque sensor at the end effector, and auditory data is acquired by a microphone, including force signals, torque signals, or six-dimensional force / torque timing signals from the robot's end effector during the grasping process. Auditory data includes audio time-domain signals, frequency-domain signals, or time-spectrum signals during the grasping process. Multimodal raw data can be acquired before, during, and after the grasping action to cover the entire process of grasping approach, contact, closing, lifting, and transport. The existing process already supports simultaneous triggering of visual, force, and audio acquisition via unified control commands.

[0035] Example 3 Based on Example 1 or 2, the time alignment in step 2 is to perform synchronous alignment of visual data, force data and auditory data at different sampling frequencies under a unified time reference.

[0036] Visual features in multimodal features are static visual features or temporal visual features extracted from visual data; force features are statistical features, rate of change features, contact impact features, or temporal dynamic features extracted from force data; auditory features are Mel-spectrum features, short-time Fourier transform features, time-frequency texture features, or combinations thereof extracted from auditory data.

[0037] Example 4 Building upon Examples 1-3, sample-level discrimination is used to identify the overall failure category of the current grasped sample from the multimodal features corresponding to the complete grasping process. Specifically, for the visual modality, force modality, and auditory modality, corresponding modal feature encoders are constructed respectively. The visual modality encoder is used to extract visual latent representations from changes in the image before and after grasping, changes in the target region, changes in local motion, or changes in depth; the force modality encoder is used to extract force latent representations from the peak value, mean, variance, rate of change, impact amplitude, or contact change trend of the end force / torque; and the auditory modality encoder is used to extract auditory latent representations from audio peak values, spectral energy, Mel spectral coefficients, or time-frequency texture.

[0038] Specifically, the modal reliability weighted fusion mechanism in step 3 includes the following process: Let the multimodal set be ,in, Representing visual modality, Represents force perception mode, Represents auditory modality; for any modality The feature corresponding to this mode is denoted as ; First, the features are gated using a modal feature gate function. The gated modal features are obtained by filtering. : , in, For the first Feature gating vectors corresponding to each modality This represents element-wise multiplication; Feature Gating Vector We obtain it from the following formula: , in, and For learnable parameters, This is the Sigmoid function.

[0039] Modal feature gating functions are used to suppress feature components in the modality that are irrelevant to the identification of capture failures or are greatly affected by noise, thereby improving the stability of subsequent failure category identification.

[0040] Then, modal features Input the corresponding modal encoder to obtain the modal implicit representation. And the prediction result of the failure category of the corresponding modality is obtained through the corresponding modality classifier: , , in, For the first The mode encoder corresponding to each mode. For the first A modality classifier corresponding to each modality. Indicates the first The failure categories of each modal output are logits; Next, temperature calibration is performed on the failure categories (logits) of each mode to obtain the calibrated mode prediction results: , in, For the first The temperature parameters corresponding to each mode are used to calibrate the prediction confidence of that mode; Subsequently, based on the calibrated modality prediction results, the class probability of the mode for each failure category is calculated using the Softmax function. : , Next, the corresponding modal reliability score is determined based on the category probability of each modality. ; Finally, based on the reliability scores of each mode... Modal reliability weights are obtained through a normalized exponential function. : , in, Indicates the first The reliability weight of each modality in the current crawled sample This is a reliability sharpening factor used to adjust the degree of difference between reliability weights for different modes. For the first The reliability score of each modality in the current crawled sample; The reliability weights for each mode satisfy: .

[0041] When a modality is affected by occlusion, noise, abnormal acquisition, or missing signal, resulting in a decrease in the confidence of its failure category prediction, the reliability weight of that modality in the fusion result is reduced accordingly; when a modality has more stable and clearer failure discrimination information in the current failed capture samples, the reliability weight of that modality in the fusion result is increased accordingly.

[0042] In a preferred implementation, the modal reliability score for: , in, Indicates the first The modality pair of the first The predicted probability of each failure category, Indicates the first The reliability score of a modality in the currently captured samples. In other words, a modality gets a higher reliability score when it has a higher calibration confidence in its prediction of the failure category.

[0043] Example 5 Based on Example 4, the prediction results of each mode are weighted and fused according to the modal reliability weight to obtain the sample-level failure discrimination result: , And according to The maximum probability category output is the category of the failed crawling of the current crawled sample: , in, This refers to the category of failed crawling for the currently crawled sample. Among them, the categories of grab failure include one or more of the following: missing grab, slipping, falling, collision, destructive grab, and contact anomaly.

[0044] Example 6 Based on Examples 1-5, in step 3, temporal-level localization is used to determine the moment of failure, key time period, or event window range during the crawling process. Specifically, temporal-level localization involves inputting the visual, force, and auditory features obtained in step 2 into the corresponding temporal encoders to obtain the temporal latent representations of each modality at different times. : , in, For the first The timing encoder corresponding to each mode For the first The modality at time... The temporal features; the temporal encoder can employ one or more of the following: one-dimensional temporal convolutional network, dilated temporal convolutional network, recurrent neural network, or temporal attention network.

[0045] First, for any given moment Frame-level event saliency scores are calculated based on the temporal implicit representations of each modality. , used to indicate the first The modality at time... The degree of contribution to the failure event; Then, based on each mode at time... Calculating cross-modal conflict degree using differential classification probability ; Next, the temporal latent representations of each modality are concatenated or weighted and fused before being input into the event detection head to obtain the event occurrence probability at each time step. : , in, Indicates the event detection header, Representing visual modalities Force mode and auditory modality At any moment The splicing result of the temporal implicit representation; After that, the probability of the event occurring Compared with frame-level event saliency score and cross-modal conflict degree Together they form a comprehensive event score. : , in, and These are the weighting coefficients; Next, based on the comprehensive event score The peak time of failure events is determined by the peak time of failure events: , Finally, based on the comprehensive event score A failure event window is defined as a continuous time period exceeding a preset percentage threshold; wherein, the start and end times of the failure event window are... and satisfy: , in, For the preset ratio threshold, the failure event window Peak time of failure events Together with the keyframe index, they constitute the failure event information.

[0046] In particular, Obtained from a frame-level scoring function and normalized over time: , , in, For the first The modality at time... Frame-level rating, and For the parameters of the frame-level scoring function, To The transpose of the corresponding matrix; This represents the time index during the crawling process; In particular, for any two modes and Calculate at time The difference in category probability distributions is calculated, and the average difference in distributions across all modal pairs is used to obtain the cross-modal conflict degree. : , in, Represents the set of all modal pairs. The function representing the difference between two class probability distributions is... distance, distance, divergence or One type of divergence, Indicates the first The modality at time... The predicted probability distribution for each failure category, Indicates the first The modality at time... The predicted probability distribution for each failure category. When different modalities produce significantly inconsistent judgments about the failure category at the same time, An increase in this value indicates that the moment may correspond to a critical failure event such as abnormal contact, slippage, falling, or collision.

[0047] Failure event information includes one or more of the following: the time of occurrence of the failure event, the duration of the failure event, the keyframe index, the key time period index, and the event window range.

[0048] Specifically, in the process of locating time-series failure events, the modal reliability weights obtained from sample-level failure discrimination are... It can also be used to modulate the temporal implicit representation of each mode, so that the more reliable modes contribute more to event localization. Preferably, the temporal implicit representation is modulated as follows: , in, For the modulated first Modal temporal implicit representation For the first The modulation coefficients corresponding to each mode. In this way, the reliability estimation in sample-level failure discrimination can be transferred to the time-series failure event localization, thus forming a coupling between sample-level category judgment and time-series event evidence.

[0049] Therefore, step 3 can output not only the category of capture failure, but also information such as failure confidence, modal reliability weight, failure peak time, event window range, keyframe index and cross-modal conflict degree, providing structured input for the subsequent step 4 to construct the failure feedback vector.

[0050] Example 7 Based on Examples 1-6, the modal reliability weight is also used to generate a failure feedback vector, so that the failure feedback vector includes not only the failure category and failure confidence, but also the reliability contribution of the visual modality, force modality and auditory modality in the current failure identification process, thereby providing interpretable multimodal evidence for subsequent re-grabbing correction strategies.

[0051] In the above manner, the embodiments of the present invention do not use a complex domain gated attention mechanism to generate comprehensive object description information, nor do they use a Kalman filter to dynamically update the object state. Instead, they focus on the tasks of grasping failure identification and re-grabbing, determine the modal reliability weights based on the calibration confidence of each modality's judgment of the failure category, and use the modal reliability weights to fuse the failure category identification results and failure feedback vectors to construct the system.

[0052] The failure feedback vector constructed in step 4 includes at least failure category, failure confidence, and modal reliability information; preferably, it also includes one or more of the following: failure event start and end location, keyframe location, event intensity, modal conflict degree, risk intensity index, contact anomaly indication information, and strategy suggestion label. The purpose of this step is to transform the crawling failure identification result into a structured feedback representation, so that subsequent re-crawl corrections no longer rely on isolated judgments based on manual experience rules, but instead receive the failure reasons and correction basis in a unified format.

[0053] When generating the re-grab correction strategy in step 5, at least one or more of the following gripping execution parameters are adjusted: gripper opening degree, gripper posture, end-effector approach direction, gripping contact position, gripping feed depth, gripping region selection, and candidate gripping sorting. Specifically, the re-grab correction strategy includes one or more of the following: (1) Generate a local contact zone to define the area suitable for re-contacting the target object; (2) Generate and capture risk distribution information to represent high-risk and low-risk contact areas; (3) Generate candidate grasping constraint information, which is used to filter and reorder candidate grasping points or candidate grasping postures; (4) Generate gripping parameter increments to adjust the gripper opening degree, gripper posture, end approach direction, gripping contact position and gripping feed depth; (5) Generate structured semantic correction instructions to represent correction suggestions such as "move the gripping position to the edge of the target", "increase the depth of the gripper insertion", and "adjust the end gripping angle".

[0054] Specifically, step 5 further includes: generating one or more of the following based on the failure feedback vector: a local contact zone, a grasping risk distribution map, candidate grasping area constraint information, or grasping parameter increments; and generating a re-grab correction strategy based on the local contact zone, the grasping risk distribution map, candidate grasping area constraint information, or grasping parameter increments. When the failure feedback vector indicates slippage or unstable gripping, the re-grab correction strategy includes one or more of the following: increasing the gripper's coverage contact area, adjusting the gripper's feed depth, or changing the contact area position; when the failure feedback vector indicates collision or excessively high contact risk, the re-grab correction strategy includes one or more of the following: adjusting the end-effector approach direction, avoiding high-risk areas, or reducing local contact risk; when the failure feedback vector indicates missing the target or target positioning deviation, the re-grab correction strategy includes one or more of the following: adjusting the grasping area, reselecting grasping candidate points, or correcting the end-effector alignment position.

[0055] Specifically, the re-grab correction strategy also includes outputting structured semantic correction instructions. These instructions characterize grasping adjustment suggestions for the current grasping failure category, and include at least one of the following: moving the grasping position towards the target edge, increasing the gripper insertion depth, decreasing the gripper insertion depth, adjusting the end-effector angle, or avoiding local interference areas.

[0056] Specifically, in step 6, the robot is controlled to perform a re-grasping. Based on the re-grasping correction strategy generated in step 5, the robot is controlled to perform the corrected re-grasping operation. In a preferred embodiment, the control includes: controlling the robotic arm to move to the corrected approach pose, controlling the gripper to contact and close according to the corrected opening and closing degree and feed depth, and then performing lifting or transporting actions to complete the re-grasping process.

[0057] Specifically, step 7 determines whether the grasping was successful. After the robot performs a re-grasp, the success or failure of the current grasping result is determined. If the grasping is successful, step 8 is executed; if the grasping fails, step 1 is returned to acquire new multimodal raw data, and grasping failure identification and failure feedback vector construction are performed again based on the new multimodal raw data; if the re-grasp is still determined to be unsuccessful, a new re-grasp correction strategy is generated iteratively until the grasping is successful or the preset termination condition is reached, thus forming a closed-loop re-grasp mechanism.

[0058] Example 8 like Figure 2 As shown, this embodiment of the invention provides a multimodal fusion robot grasping failure identification and re-grasping system, including: a multimodal data acquisition module, a feature extraction and time alignment module, a grasping failure identification module, a failure feedback vector construction module, a re-grasping strategy generation module, and an execution control module; The multimodal data acquisition module acquires multimodal raw data during the grasping process, including visual data, force data, and auditory data. In this embodiment, visual data can be acquired by an RGB camera, a depth camera, or an RGB-D camera; force data can be acquired by a six-dimensional force / torque sensor at the robot's end effector; and auditory data can be acquired by a microphone. In existing pipelines, A / B camera images, six-dimensional force data, and audio data can be acquired before, during, and after grasping, and synchronous acquisition is triggered by a unified control signal.

[0059] The feature extraction and time alignment module, connected to the multimodal data acquisition module, is used to preprocess, time-align, and extract features from visual, force, and auditory data to obtain visual, force, and auditory features. Specifically, visual data can be used to extract single-frame visual features or temporal visual features; force data can be used to extract statistical features, rate of change features, contact impact features, or temporal dynamic features; and auditory data can be used to extract spectrogram features, Mel-spectrum features, time-frequency texture features, or combinations thereof. For modal data at different sampling frequencies, it is preferable to synchronize and align them using a unified time reference before inputting them into subsequent modules.

[0060] The grasping failure identification module, connected to the feature extraction and time alignment module, fuses visual, force, and auditory features. It employs a combination of sample-level discrimination and temporal-level localization to identify grasping failures, outputting the grasping failure category and corresponding failure event information. The grasping failure categories include at least one or more of the following: missing grasp, slipping, falling, collision, destructive grasping, and contact anomaly. Failure event information includes one or more of the following: the time of occurrence of the failure event, the duration of the failure event, the keyframe position, the key time period position, or the event window range. In this embodiment, the grasping failure identification module can also output reliability information related to each modality to reduce the impact of abnormal modalities, missing modalities, or noisy modalities on the final identification result. Existing models have already constructed reliabilityweights, start / end / event probabilities, and event scores during inference, which can be used as a specific implementation method for this module.

[0061] A failure feedback vector construction module, connected to the crawling failure identification module, is used to construct a failure feedback vector characterizing the causes of crawling failures and the basis for correction. The failure feedback vector includes at least failure category, failure confidence level, and modal reliability information. Preferably, the failure feedback vector also includes one or more of the following: failure event occurrence time, event interval, modal conflict degree, risk intensity index, contact anomaly indication information, and strategy suggestion label. In this embodiment, the failure feedback vector is a structured intermediate representation used to further transform the "identification result" into "feedback basis that can be directly used for strategy correction," thereby avoiding the problem in existing technologies where only success or failure labels are output without guiding subsequent re-crawls.

[0062] The re-grabbing strategy generation module is connected to the failure feedback vector construction module and is used to generate a re-grabbing correction strategy based on the failure feedback vector. The re-grabbing correction strategy includes at least one of the following: a local contact zone, grasping risk distribution information, candidate grasping constraint information, grasping parameter increments, and structured semantic correction instructions. Specifically, the local contact zone is used to determine contact areas suitable for re-grabbing; the grasping risk distribution information is used to characterize unfavorable contact areas or high-risk areas in the local space; and the grasping parameter increments are used to correct the gripper opening degree, gripper posture, end-effector approach direction, grasping contact position, and grasping feed depth. In a preferred embodiment, the local contact zone can establish a contact zone coordinate system based on the candidate grasping pose, gripper width, height, and depth, and filter the point cloud through a capsule or strip-shaped region to obtain the local contact area related to re-grabbing. The current contact zone program has already extracted the contact zone coordinate system, capsule parameters, and in-strip point cloud index based on the executed grasping pose, and can be used as a specific implementation of this module.

[0063] The execution control module, connected to the re-grasping strategy generation module, is used to control the robot to perform re-grasping operations based on the re-grasping correction strategy. In this embodiment, the execution control module can drive the robotic arm and gripper to perform corrected approach, closing, lifting, and transporting actions. The robot can be a multi-degree-of-freedom robotic arm system equipped with a gripper, such as a gripping platform including the robotic arm body, gripper, camera, force sensor, and audio acquisition device. Existing machine scripts already include a collaborative execution flow of the JAKA robotic arm, AG-95 gripper, RealSense camera, and external acquisition process.

[0064] Specifically, the failure identification module is also used to output one or more of the following information: failure confidence, key frame position, key time period position, or event window range; the failure feedback vector construction module is used to encapsulate failure category, failure confidence, modal reliability information, and failure event information into a structured failure feedback vector.

[0065] Specifically, the re-grab strategy generation module includes one or more of the following: a contact zone generation unit, a risk field generation unit, and a parameter increment generation unit. The contact zone generation unit is used to determine the contact area suitable for re-grab; the risk field generation unit is used to generate spatial distribution information related to local contact risks; and the parameter increment generation unit is used to generate one or more adjustment amounts for gripper opening degree, gripper posture, end-effector approach direction, and gripping feed depth.

[0066] Specifically, the execution control module is configured to drive the robotic arm and gripper to perform at least one adaptive re-grasping based on the re-grasping correction strategy output by the re-grasping strategy generation module after the robot's first grasping fails, and to continue closed-loop iteration when the re-grasping fails.

[0067] Specifically, the system is configured to perform a method for identifying and re-grasping robot grasping failures using multimodal fusion, as described in any of the preceding embodiments.

[0068] Example 9 The robotic grasping platform used in this embodiment includes a multi-degree-of-freedom robotic arm, a parallel gripper, an RGB-D camera, a six-dimensional force / torque sensor, and a microphone. The RGB-D camera is used to collect visual data during the grasping process, the six-dimensional force / torque sensor is used to collect contact force data between the robot's end effector and the target object or environment, and the microphone is used to collect auditory data during the grasping process. In each experiment, the robot performs one grasping action, simultaneously recording visual, force, and auditory data during the grasping process.

[0069] This embodiment collected data from 2428 real-world grasping failure tests, covering five categories of grasping failures: slippage, drop, collision, destructive grasping, and missing grasp. See Table 1 for details. Table 1 Number of Failure Samples for Each Category In this embodiment, visual, force, and auditory data are preprocessed and aligned to the same time reference before being input into the grasping failure identification module of this invention. The grasping failure identification module outputs the grasping failure category, failure category confidence, modal reliability weight, and failure event information, and further constructs a failure feedback vector. To verify the effectiveness of the sample-level failure discrimination, time-series failure event localization, and modal reliability weighted fusion mechanism in this invention, the following control experiments are set up: The first set of experiments was used to verify the impact of different branch structures on the recognition performance of grasping failures. The control methods included: using only the confidence calibration branch, using only the fine-grained feedback branch, and using the complete structure of this invention. The experimental results are shown in Table 2: Table 2 As shown in the table above, when only the confidence calibration branch is used, the grasping failure recognition effect is low due to the lack of temporal event characterization of the failure process. When only the fine-grained feedback branch is used, the failure recognition performance can be improved by utilizing temporal event information. When the complete structure of this invention is adopted, the sample-level failure discrimination, temporal-level failure event localization, and modal reliability weighted fusion work together to achieve an accuracy of 88.12% and a macro-average F1 of 87.41%. These results indicate that this invention can effectively improve the robot's ability to recognize different grasping failure categories.

[0070] The second set of experiments was used to verify the roles of visual, force, and auditory data in grasping failure recognition. Grasping failure recognition was performed using single-modal, bimodal, and trimodal combinations, respectively. The experimental results are shown in Table 3. Table 3 As shown in the table above, a single modality is insufficient to fully represent all abnormal information during a grasping failure process. For example, force data is more sensitive to collisions, slippage, and abnormal contact; visual data helps determine changes in object position and target area; and auditory data helps capture transient events such as collisions, friction, and falls. When the three modalities are combined, the accuracy reaches 87.50%, and the macro-average F1 score reaches 87.14%, which is higher than either a single modality or any bimodal combination. This result demonstrates that the present invention, employing visual, force, and auditory multimodal evidence fusion, can improve the completeness and stability of grasping failure identification.

[0071] Furthermore, the table above also shows that some bimodal combinations are not necessarily superior to a single force-sensing modality, indicating that abnormal, noisy, or low-reliability modalities may interfere with the recognition results during multimodal fusion. Therefore, this invention employs a modal reliability weighted fusion mechanism, determining the corresponding modal reliability weight based on the calibration confidence of each modality in the current sample, thereby reducing the impact of abnormal or low-reliability modalities on the final failure category judgment.

[0072] The third set of experiments was used to verify the confidence reliability of the present invention. The expected calibration error, negative log-likelihood, and Brier score were calculated for different output results. The results are shown in Table 4. Table 4 Among these, the expected calibration error, negative log-likelihood, and Brier score are all better the lower they are.

[0073] As shown in the table above, the expected calibration error of the complete structure output of this invention is 0.058, the negative log-likelihood is 0.493, and the Brier score is 0.206, all of which are significantly better than the fusion output of only the confidence calibration branch. This result indicates that this invention not only improves the accuracy of identifying failed capture categories but also enhances the reliability of the failure category confidence, making the failure category confidence and modal reliability weights a valid basis for generating subsequent re-capture correction strategies.

[0074] The fourth set of experiments was used to verify the technical effectiveness of the present invention in real-world robot re-grasping tasks. The test subjects included 12 everyday objects, categorized into simple, medium, and difficult objects based on their grasping difficulty. Each object underwent multiple real-world robot grasping trials. After the first grasping failure, the success rate of the second re-grasping attempt was compared between the method of the present invention and two control methods. The results are shown in Table 5: Table 5 As shown in the table above, in the second re-grabbing after the initial failure, the overall success rate of the method of this invention reached 80.56%, significantly higher than that of control method one (43.33%) and control method two (47.22%). Especially for medium-sized and difficult objects, the method of this invention maintains a high success rate, indicating that the re-grabbing correction strategy based on the failure feedback vector can effectively avoid repeating similar erroneous grabbing actions and improve the recovery capability after failure.

[0075] Furthermore, the percentage of tasks that were successfully completed within a limited number of retries after the initial failure to crawl was statistically analyzed.

[0076] The results are shown in Table 6: Table 6 As shown in the table above, the overall completion rate of the method of the present invention is 89.17% within 3 attempts after the first failed crawl and 96.67% within 5 attempts, both higher than the control method. This result indicates that the present invention, by transforming the multimodal failure identification results into a re-crawl correction strategy through a failure feedback vector, can improve the task completion rate within a limited number of retries and reduce the number of repeated trials.

[0077] In summary, the experimental data of this embodiment show that the present invention has at least the following technical effects: First, it can improve the robot's accuracy in identifying the type of grasping failure; second, it can reduce the impact of abnormal modes or noisy modes on the recognition results through modal reliability weighted fusion; third, it can output a failure feedback vector with high confidence reliability; fourth, it can transform the failure feedback vector into a re-grasping correction strategy, thereby improving the success rate of the second re-grasping after the first failure and the task completion rate within a limited number of attempts.

[0078] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for identifying and re-grasping robot grasping failures through multimodal fusion, characterized in that, The method includes the following steps: Step 1: Acquire multimodal raw data, including visual data, force data, and auditory data; Step 2: Perform time alignment and extract multimodal features, including visual features, force features, and auditory features; Step 3: Use a modal reliability weighted fusion mechanism to perform feature fusion on the multimodal features, and use a combination of sample-level discrimination and temporal-level localization to identify capture failures, and output the capture failure category and the failure event information corresponding to the capture failure category; Step 4: Construct a failure feedback vector, which should include at least the failure category, failure confidence, and modal reliability information; Step 5: Generate a re-grabbing correction strategy, which includes at least one of the following: local contact zone, grabbing risk distribution information, candidate grabbing constraint information, grabbing parameter increment, and structured semantic correction instructions; Step 6: Perform a re-fetch operation; Step 7: Determine whether the re-fetching operation was successful. If it was unsuccessful, return to step 1. Step 8: If successful, end the crawling process; The failure event information in step 3 includes, but is not limited to, failure confidence, modal reliability weight, failure peak time, event window range, keyframe index, and cross-modal conflict degree.

2. The method for identifying and re-grasping robot grasping failures through multimodal fusion as described in claim 1, characterized in that, The visual data in step 1 includes one or more of the following: single-frame images, multi-frame image sequences, color images, and depth images before, during, or after grasping; the force data includes force signals, torque signals, or six-dimensional force / torque timing signals from the robot's end effector during the grasping process; and the auditory data includes audio time-domain signals, frequency-domain signals, or time-spectrum signals during the grasping process.

3. The method for identifying and re-grasping robot grasping failures through multimodal fusion as described in claim 1, characterized in that, The time alignment in step 2 is to perform synchronous alignment of the visual data, force data and auditory data at different sampling frequencies under a unified time reference. The visual features in the multimodal features are static visual features or temporal visual features extracted from the visual data; the force features are statistical features, rate of change features, contact impact features, or temporal dynamic features extracted from the force data; and the auditory features are Mel-spectral features, short-time Fourier transform features, time-frequency texture features, or combinations thereof extracted from the auditory data.

4. The method for identifying and re-grasping robot grasping failures through multimodal fusion as described in claim 1, characterized in that, The modal reliability weighted fusion mechanism in step 3 includes the following process: Let the multimodal set be ,in, Representing visual modality, Represents force perception mode, Represents auditory modality; for any modality The feature corresponding to this mode is denoted as ; First, the features are gating using a modal feature gate function. The gated modal features are obtained by filtering. : ， in, For the first Feature gating vectors corresponding to each modality This represents element-wise multiplication; The feature gate vector We obtain it from the following formula: ， in, and For learnable parameters, For the Sigmoid function; Then, the modal features Input the corresponding modal encoder to obtain the modal implicit representation. And the prediction result of the failure category of the corresponding modality is obtained through the corresponding modality classifier: ， ， in, For the first The mode encoder corresponding to each mode. For the first A modality classifier corresponding to each modality. Indicates the first The failure categories of each modal output are logits; Next, temperature calibration is performed on the failure categories (logits) of each mode to obtain the calibrated mode prediction results: ， in, For the first The temperature parameters corresponding to each mode are used to calibrate the prediction confidence of that mode; Subsequently, based on the calibrated modality prediction results, the class probability of the mode for each failure category is calculated using the Softmax function. : ， Next, the corresponding modal reliability score is determined based on the category probability of each modality. ; Finally, based on the reliability scores of each mode... Modal reliability weights are obtained through a normalized exponential function. : ， in, Indicates the first The reliability weight of each modality in the current crawled sample This is a reliability sharpening factor used to adjust the degree of difference between reliability weights for different modes. For the first The reliability score of each modality in the current crawled sample; The reliability weights for each mode satisfy: 。 5. The method for identifying and re-grasping robot grasping failures through multimodal fusion as described in claim 4, characterized in that, Modal reliability score for: ， in, Indicates the first The modality pair of the first The predicted probability of each failure category, Indicates the first The reliability score of each modality in the current crawled sample.

6. The method for identifying and re-grasping robot grasping failures through multimodal fusion as described in claim 4, characterized in that, Based on the modal reliability weights, the prediction results of each modality are weighted and fused to obtain the sample-level failure discrimination results: ， And according to The maximum probability category output is the category of the failed crawling of the current crawled sample: ， in, This refers to the category of failed crawling for the currently crawled sample. The grab failure categories include one or more of the following: missing grab, slipping, falling, collision, destructive grab, and contact anomaly.

7. The method for multimodal fusion-based robot grasping failure identification and re-grasping as described in claim 4, characterized in that, In step 3, the temporal localization involves inputting the visual features, force features, and auditory features obtained in step 2 into the corresponding temporal encoders to obtain the temporal latent representations of each modality at different times. : ， in, For the first The timing encoder corresponding to each mode For the first The modality at time... The temporal characteristics; First, for any given moment Frame-level event saliency scores are calculated based on the temporal implicit representations of each modality. , used to indicate the first The modality at time... The degree of contribution to the failure event; Then, based on each mode at time... Calculating cross-modal conflict degree using differential classification probability ; Next, the temporal latent representations of each modality are concatenated or weighted and fused, and then input into the event detection head to obtain the event occurrence probability at each time step. : ， in, Indicates the event detection header, Representing visual modalities Force mode and auditory modality At any moment The splicing result of the temporal implicit representation; Then, the probability of the event occurring. Compared with the frame-level event saliency score and the cross-modal conflict degree Together they form a comprehensive event score. : ， in, and These are the weighting coefficients; Next, based on the comprehensive event score... The peak time of failure events is determined by the peak time of failure events: ， Finally, based on the comprehensive event score... A failure event window is defined as a continuous time period exceeding a preset proportion threshold; wherein, the start and end times of the failure event window are... and satisfy: ， in, The failure event window is set to a preset ratio threshold. The peak time of the failure events Together with the keyframe index, they constitute the failure event information.

8. The method for identifying and re-grasping robot grasping failures through multimodal fusion as described in claim 7, characterized in that, Obtained from a frame-level scoring function and normalized over time: ， ， in, For the first The modality at time... Frame-level rating, and The parameters of the frame-level scoring function are... To The transpose of the corresponding matrix, This represents the time index during the crawling process; For any two modes and Calculate at time The cross-modal conflict degree is obtained by averaging the differences in the category probability distributions of all modal pairs. : ， in, Represents the set of all modal pairs. The function representing the difference between two class probability distributions is... distance, distance, divergence or One type of divergence, Indicates the first The modality at time... The predicted probability distribution for each failure category, Indicates the first The modality at time... The predicted probability distribution for each failure category.

9. A system for identifying and re-grasping robot grasping failures through multimodal fusion, characterized in that, The system includes: a multimodal data acquisition module, a feature extraction and time alignment module, a crawling failure identification module, a failure feedback vector construction module, a re-crawl strategy generation module, and an execution control module; The multimodal data acquisition module acquires multimodal raw data during the grasping process, including visual data, force data, and auditory data; The feature extraction and time alignment module is connected to the multimodal data acquisition module and is used to preprocess, time align and extract features from the visual data, the force data and the auditory data to obtain visual features, force features and auditory features. The grasping failure identification module is connected to the feature extraction and time alignment module. It is used to fuse the visual features, the force features and the auditory features, and to identify grasping failures by combining sample-level discrimination and temporal-level localization. It outputs the grasping failure category and the failure event information corresponding to the grasping failure category. The failure feedback vector construction module is connected to the crawling failure identification module and is used to construct a failure feedback vector that characterizes the cause of crawling failure and the basis for correction; the failure feedback vector includes at least failure category, failure confidence and modal reliability information; The re-crawl strategy generation module is connected to the failure feedback vector construction module and is used to generate a re-crawl correction strategy based on the failure feedback vector; the re-crawl correction strategy includes at least one of the following: local contact zone, crawling risk distribution information, candidate crawling constraint information, crawling parameter increment, and structured semantic correction instructions; The execution control module is connected to the re-grabbing strategy generation module and is used to control the robot to perform re-grabbing operations according to the re-grabbing correction strategy.

10. The system for multimodal fusion-based robot grasping failure identification and re-grasping as described in claim 9, characterized in that, Each module is configured to perform the multimodal fusion robot grasping failure identification and re-grasping method as described in any one of claims 1 to 8.

Images