Industrial robot self-adaptive grabbing method and system based on multi-mode perception

Abstract

The invention provides an industrial robot self-adaptive grabbing method based on multi-modal sensing, which comprises the following steps of: 1, acquiring multi-modal data of an object and an environment through a multi-modal sensing module; the multi-mode sensing module comprises at least two sensing modules of a visual sensing unit, a touch sensing unit, a force sensing unit and an auditory sensing unit; 2, preprocessing and feature extraction are carried out on the multi-modal data, feature information of different modals is fused through a multi-modal fusion algorithm, and comprehensive object description information is generated; 3, generating an optimal grabbing strategy based on the comprehensive object description information; wherein the grabbing strategy comprises the position and posture of a grabbing point, a grabbing path and grabbing force; and fourthly, the industrial robot executes the optimal grabbing strategy, the first step to the third step are repeated, and the grabbing strategy is adjusted in a self-adaptive mode till grabbing is completed. Stable and efficient grabbing can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial robots, in particular to an industrial robot adaptive grasping method and system based on multi-modal perception. BACKGROUND

[0002] In current industrial scenarios, industrial robots mostly rely on rigid programming logic when performing grasping tasks, that is, object positioning is completed through pre-set coordinate systems or fixed programming logic. Such methods can maintain a certain stability in structured environments (such as standardized parts scenes in assembly lines), but their adaptability is significantly exposed when facing unstructured working conditions (such as sudden changes in lighting, surface defects, stacking obstructions, or material heterogeneity). Existing grasping systems based on multi-modal perception generally have the following limitations: first, the fusion method for different modal data is simple, and it is difficult to fully utilize the advantages of each modality; second, there is a lack of real-time adaptive adjustment capability for dynamic changes during grasping; third, the grasping strategy generation for complex objects is not flexible enough to meet diverse needs. Experimental tests have found that existing vision-based grasping systems have a 37% increase in positioning error when the lighting changes more than ±15%; when the object surface friction coefficient μ<0.3, the failure rate of traditional grasping methods is as high as 42%.

[0003] Therefore, it is necessary to provide a stable and efficient grasping method and system. SUMMARY

[0004] The purpose of the present application is to provide an industrial robot adaptive grasping method and system based on multi-modal perception, which can achieve stable and efficient grasping.

[0005] In a first aspect, the present application provides an industrial robot adaptive grasping method based on multi-modal perception, comprising: Step 1: Collecting multi-modal data of objects and environments through a multi-modal perception module; the multi-modal perception module includes at least two of a visual perception unit, a tactile perception unit, a force perception unit, and a hearing perception unit; Step 2: Preprocessing and feature extraction of the multi-modal data, fusion of feature information of different modalities through a multi-modal fusion algorithm, and generation of comprehensive object description information; Step 3: Generating an optimal grasping strategy based on the comprehensive object description information; wherein the grasping strategy includes grasping point position and attitude, grasping path, and grasping force; Step 4: Industrial robot executes the optimal grasping strategy, repeats steps 1-3, and adaptively adjusts the grasping strategy until the grasping is completed.

[0006] In a possible implementation, after the optical imaging sensor array performs denoising, enhancement and normalization processing on the image, a target detection algorithm based on a convolutional neural network (CNN) is used to detect the target object; wherein the loss function of target detection is: ; wherein, is the coordinate loss of the th target prediction box in the th grid; is the confidence loss of the th non-target prediction box in the th grid; is the category loss of the th grid; and are weight coefficients.

[0007] In a possible implementation, the six-dimensional force-moment sensor outputs a generalized external force vector in real time through a nonlinear decoupling algorithm (based on Wrench-Space projection); The moment is calculated based on the generalized external force vector, and the calculation formula is: ; wherein, is the moment, is the distance vector from the force point to the rotation axis, is the generalized external force vector; the controller sends the calculated moment as an output instruction to the joint motor to offset the inertial disturbance of the robot arm.

[0008] In a possible implementation, in step 2, an adaptive multi-source sensing integration algorithm is used, and a spatio-temporal alignment feature encoder is constructed to realize the synchronization of different sampling frequency sensing data in the time domain and the spatial domain, to obtain the feature vectors of each modality; and a complex domain gating attention mechanism is used to dynamically adjust the weight coefficients of the modality feature vectors, and the modality feature vectors are fused to obtain comprehensive object description information; wherein the calculation formula of the complex domain gating attention mechanism is: ; wherein, is the weight coefficient of the th modality, is the score of the feature vector of the th modality, is the number of modalities; wherein, , represents the real part of , represents the imaginary part of , is a complex unit; ; ; wherein, , and , are learnable weights and biases; the integrated object description information is: ; wherein, is the integrated object description information, is the feature vector of the i-th modality.

[0009] In a possible implementation, a Kalman filter is used to dynamically update the integrated object description information.

[0010] The application can solve the fusion distortion problem caused by the time-space asynchrony of multi-source sensing information when the traditional industrial robot grasps an object.

[0011] In a possible implementation, the step 3 comprises: based on the integrated object description information, generating an optimal grasping point position and pose using a hybrid learning framework based on deep deterministic policy gradient; wherein the hybrid learning framework updates the policy in reinforcement learning in combination with a Q-learning algorithm; planning a collision-free and minimum-cost grasping path according to the grasping point position and the current pose of the industrial robot; adjusting the grasping force according to the feedback information of the force perception unit and the tactile perception unit.

[0012] In a possible implementation, the path planning is performed using the A* algorithm, and the heuristic function is: ; wherein, is the total cost of the node , is the actual cost from the starting point to the node , and is the estimated cost from the node to the end point; represents the i-th node on the path ; The goal of path planning is to find a path with the minimum cost: ; wherein, is the path, is the number of nodes on the path .

[0013] ​​​​​In a possible implementation, the grasping force adjustment unit adopts PID control, dynamically adjusts the control output according to the deviation of the actual grasping force from the expected grasping force, and realizes real-time optimization of the grasping force; wherein the actual grasping force is measured by the force perception unit; the expected grasping force and the parameters of the PID control are dynamically adjusted according to the feedback information of the tactile perception unit.

[0014] In a possible implementation, the method further comprises: triggering a safety protection mechanism when an abnormality is detected during the grasping process; In a possible implementation, during the grasping process, the pressure distribution asymmetry degree β of the end effector is detected by a distributed piezoresistive sensing network, and when β is greater than a preset threshold, a warning is triggered and the safety protection mechanism is started; The starting of the safety protection mechanism comprises: using a Lyapunov energy function-based gradual braking unit to control the robot to gradually (i.e. smoothly and without impact) stop; specifically, it comprises: calculating the braking torque required by each joint in real time according to the Lyapunov energy function and the current state (position, speed) of the robot; sending the calculated braking torque instruction to the joint motor to gradually reduce the speed of the robot; during the braking process, the speed and energy state of the robot are continuously monitored until it is confirmed to be completely stopped.

[0015] In a second aspect, the application provides an industrial robot adaptive grasping system based on multi-modal perception, comprising: A multi-modal perception module, comprising at least two of a visual perception unit, a tactile perception unit, a force perception unit and a hearing perception unit, for collecting multi-modal data of objects and environments; A data fusion and processing module for pre-processing and feature extraction of the multi-modal data, fusing feature information of different modalities through a multi-modal fusion algorithm to generate comprehensive object description information; A grasping strategy generation module for generating an optimal grasping strategy based on the comprehensive object description information; wherein the grasping strategy comprises grasping point position and attitude, grasping path and grasping force; A feedback control module for controlling the industrial robot to execute the optimal grasping strategy, and adaptively adjusting the grasping strategy based on the multi-modal perception module, the data fusion and processing module and the grasping strategy generation module until the grasping is completed.

[0016] The system is used to implement the above-mentioned industrial robot adaptive grasping system based on multi-modal perception.

[0017] In a possible implementation, the visual perception unit comprises an optical imaging sensor array. The optical imaging sensor array can comprise a color camera and a depth camera, for acquiring two-dimensional images and three-dimensional point cloud data of the object, and identifying the target object through a convolutional neural network.

[0018] The tactile perception unit comprises a distributed piezoresistive sensing network. The distributed piezoresistive sensing network can be a flexible pressure-shear dual-mode film embedded on the surface of the end effector, based on a carbon nanotube / PDMS composite layer, and outputs piezoresistive-capacitive fusion signals to analyze sub-millimeter contact topography.

[0019] The force perception unit comprises a six-dimensional force torque sensing group. The six-dimensional force torque sensing group can output a generalized external force vector in real time through a nonlinear decoupling algorithm. The auditory perception unit comprises a voiceprint feature extraction device. The voiceprint feature extraction device can extract the sound signal features in the grabbing process through a short-time Fourier transform.

[0020] The above multi-modal perception module is a heterogeneous sensing information cooperative acquisition system, integrating an optical imaging sensor array, a distributed piezoresistive sensing network, a six-dimensional force torque sensing group, and a voiceprint feature extraction device, to form a complementary perception system for acquiring multi-modal data of the object and the environment.

[0021] In a possible implementation, the data fusion and processing module can comprise a feature extraction unit, a multi-modal fusion unit, and a dynamic updating unit, for pre-processing, feature extraction, multi-modal fusion, and dynamic updating of the heterogeneous sensing data acquired by the multi-modal perception module, to generate comprehensive object description information. In a possible implementation, the grabbing strategy generation module can comprise a grabbing point planning unit, a grabbing path planning unit, and a grabbing force adjustment unit, for generating optimal grabbing point positions and poses, collision-free grabbing paths based on the comprehensive object description information, and adjusting the grabbing force according to the real-time perception of force and tactile information. In a possible implementation, the feedback control module can comprise a real-time monitoring unit, an adaptive adjustment unit, and a Lyapunov energy function-based gradual braking unit, for real-time monitoring of multi-modal data and industrial robot states in the grabbing process, adaptive adjustment of the grabbing strategy according to feedback information, and triggering of a safety protection mechanism when an abnormality is detected.

[0022] The specific implementation of the above second aspect of the application can refer to the implementation of the above first aspect, which will not be described here.

[0023] Advantages: The application obtains a more comprehensive and robust description of the object by fusing visual, tactile, force sense and other multi-source information, and has high-precision environmental perception capability. This solves the limitations of single sensor in complex scenes such as light change, occlusion, and non-obvious surface features. The introduction of Kalman filter can dynamically update the comprehensive description information of the object according to real-time perception data, so that the system's understanding of the object state is time-varying and optimal, and can adapt to the dynamic changes such as small movement and deformation of the object during grasping. The complex domain gating attention mechanism is adopted to realize adaptive weight allocation of different modal information. The system can automatically judge and focus on more reliable information sources (such as relying more on tactile information when grasping soft objects) in different scenes, improving the discriminability of fused features. A PID control architecture based on tactile perception information for target setting and force sense perception information for execution adjustment is established. Tactile information is used for intelligent setting of expected grasping force, and force sense perception information is used for real-time closed-loop adjustment, realizing compliant and efficient grasping of objects with different materials and shapes, and effectively avoiding object slipping or damage caused by improper force. An optimized multi-task loss function is used to accurately balance the loss weight of target / background, ensuring high detection rate and positioning accuracy of target objects (such as parts) and potential obstacles (such as human hands) in complex industrial environments, providing reliable guarantee for grasping path planning. A safety warning mechanism based on pressure distribution asymmetry (β) is designed, and is linked with the gradual braking unit based on Lyapunov energy function. The system can trigger soft braking at the moment of abnormality (such as foreign object clamping), smoothly consume kinetic energy and stop, and has a fast response abnormal handling function, which maximizes the safety of human-machine cooperation. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A total flowchart in an embodiment of the application; Figure 2 An industrial robot adaptive grasping system in an embodiment of the application; Figure 3 A grasping process schematic diagram in an embodiment of the application; Figure 4 A multi-modal data fusion and processing flowchart in an embodiment of the application; Figure 5 A working principle schematic diagram in an embodiment of the application. DETAILED DESCRIPTION

[0025] In order to enable personnel in the technical field to better understand the scheme of the present application, the technical scheme of the present application will be further described in detail below in conjunction with the embodiments of the present application and the drawings.

[0026] The specific embodiments according to the present application will be described below with reference to the drawings. Figure 1 The specific embodiments according to the present application will be described below with reference to the drawings.

[0027] Embodiment one: The embodiment of the present application provides a kind of based on multi-modal perception's industrial robot adaptive grasping method, comprising: Step 1: by multi-modal perception module, the multi-modal data of object and environment are collected;The multi-modal perception module includes at least two perception modules in visual perception unit, tactile perception unit, force perception unit and auditory perception unit; Step 2: the multi-modal data are preprocessed and feature extraction, different modal feature information is fused by multi-modal fusion algorithm, and comprehensive object description information is generated; Step 3: based on the comprehensive object description information, the optimal grasping strategy is generated;Wherein, grasping strategy includes grasping point position and posture, grasping path and grasping force; Step 4: industrial robot executes optimal grasping strategy, repeats step 1~step 3, and adaptively adjusts grasping strategy until grasping is completed.

[0028] In some embodiments, the method in the embodiment is integrated by optical imaging sensor array (visual perception unit), distributed piezoresistive sensing network (tactile perception unit), six-dimensional force torque sensing group (force perception unit) and voiceprint feature extraction device (auditory perception unit), to form complementary perception system, for obtaining multi-modal data of object and environment.

[0029] In some embodiments, in step 3, according to comprehensive object description information, the optimal grasping point position and posture are generated by using Deep Deterministic Policy Gradient-Hybrid (DDPG-H) based on deep deterministic policy gradient hybrid learning framework, the collision-free grasping path is planned, and the grasping force is adjusted according to the real-time perception of force and tactile information.

[0030] In some embodiments, in step 4, industrial robot executes grasping action according to the generated grasping strategy, monitors each modal perception data and industrial robot state information in real time during grasping process, and adaptively adjusts grasping strategy according to feedback information, to ensure the smooth progress of grasping process. When the pressure distribution asymmetry β is greater than the preset threshold (such as 0.25), protection mechanism is triggered immediately.

[0031] Embodiment two: The embodiment of the present application provides a kind of based on multi-modal perception's industrial robot adaptive grasping system, comprising: Multi-modal perception module includes at least two perception modules in visual perception unit, tactile perception unit, force perception unit and auditory perception unit, for collecting multi-modal data of object and environment; The data fusion and processing module is used to preprocess and extract features from the multimodal data, and to fuse feature information from different modalities through a multimodal fusion algorithm to generate comprehensive object description information. The grasping strategy generation module is used to generate an optimal grasping strategy based on the comprehensive object description information; wherein, the grasping strategy includes the grasping point position and posture, the grasping path and the grasping force; The feedback control module is used to control the industrial robot to execute the optimal grasping strategy. Based on the multimodal perception module, data fusion and processing module, and grasping strategy generation module, it adaptively adjusts the grasping strategy until the grasping is completed.

[0032] The system is used to implement the industrial robot adaptive grasping system based on multimodal perception in Embodiment 1 above.

[0033] The technical content of the two embodiments in this application can be referred to each other. A detailed description follows: (1) Multimodal sensing module: As a heterogeneous sensor information collaborative acquisition system, this system innovatively integrates an optical imaging sensor array (vision), a distributed piezoresistive sensor network (tactile), a six-dimensional force and torque sensor group (force), and a voiceprint feature extraction device (hearing) to form a complementary sensing system for acquiring multimodal data of objects and the environment.

[0034] The optical imaging sensor array acquires 2D images and 3D point cloud data of the object using a color camera and a depth camera. The images undergo denoising, enhancement, and normalization to improve the recognition accuracy of subsequent deep learning models. A target object detection algorithm based on a deep learning model—a convolutional neural network (CNN)—is then used to detect the target object.

[0035] In some embodiments, the loss function for object detection is: ; in, It is the first In the grid, the th The coordinate loss term for a target bounding box; for a target bounding box, this loss term represents the difference between the predicted box and the ground truth box in the center coordinates (x, y) and width and height (w, h). It is used to optimize the position and size of the predicted box so that it fits snugly against the target object.

[0036] It is the first In the grid, the th Confidence loss for bounding boxes without targets; for bounding boxes in a grid that do not contain target objects, their predicted "confidence" should tend to 0. This loss term penalizes predictions that are much greater than 0. Cross-entropy loss is typically used. Used to reduce false positives in background regions.

[0037] is the class loss for the th grid; for grids that contain target objects, their predicted class probability distribution should align with the true class label. Cross-entropy loss is typically used. This loss term is used to ensure that the model can correctly identify the class of the target object in the grid.

[0038] and are weight coefficients.

[0039] Distributed piezoresistive sensing network: A distributed flexible pressure-shear dual-mode film (based on carbon nanotube / PDMS composite layer) is embedded on the surface of the end effector, and its output is a piezoresistive-capacitive fusion signal, which can analyze sub-millimeter contact topography. Its pressure distribution can be represented as: ; where is the reading of the th pressure sensor, is the number of sensors.

[0040] Joint dynamics field: A six-axis force-torque sensor outputs a generalized external force vector in real time through a nonlinear decoupling algorithm (based on Wrench-Space projection); Based on the generalized external force vector, the torque can be calculated, and the calculation formula is: ; where is the torque, is the distance vector from the force point to the rotation axis, is the generalized external force vector. The controller sends the calculated torque as an output command to the joint motor to offset the inertial disturbance of the robotic arm and achieve compliant motion.

[0041] Voiceprint feature extraction device: Capture the sound signal between the object and the environment during the grasping process.

[0042] Feature extraction of sound signal can use short-time Fourier transform (STFT), whose formula is: ; where is the frequency spectrum, is the time series signal, is the window function, is the window length, is the frequency index, is the time index.

[0043] (2) Data fusion and processing module: The data fusion and processing module includes a feature extraction unit, a multi-modal fusion unit, and a dynamic updating unit, which are used for processing and fusing the collaborative decoupling data of heterogeneous sensor information.

[0044] Feature extraction unit: pre-process the raw data collected by each modal perception unit, and extract the feature information related to the grabbing task.

[0045] Among them, the contour features of the object are extracted from the visual data using the edge detection algorithm (Canny algorithm): ; Among them, is the gradient of the image , and are the partial derivatives of the image in the and directions.

[0046] Multi-modal fusion unit: adopt adaptive multi-source sensor integration algorithm (AMSI), which adopts attention weight distribution mechanism, realizes the synchronization of different sampling frequency sensor data in time domain and space domain by constructing time-space alignment feature encoder, obtains the feature vector of each modal; and adopts complex domain gating attention mechanism to dynamically adjust the weight coefficient of each modal, and fuses the modal feature vector to obtain comprehensive object description information; wherein the calculation formula of the complex domain gating attention mechanism is: ; Among them, is the weight coefficient of the th modal, is the score of the feature vector of the th modal, is the number of modes.

[0047] Among them, , represents the real part of , represents the imaginary part of , is the complex unit. The calculation of the score can be realized by two parallel fully connected layers (FC), one layer calculates the real part of the score , which mainly encodes the intensity or amplitude information of the feature. Another layer calculates the imaginary part of the score , which mainly encodes the phase or time correlation information of the feature. This process can be represented as: ; ; in, , and , These are learnable weights and biases.

[0048] The integrated object description information (fused features) is represented as follows: ; in, To synthesize object description information, It is the first The feature vectors of each modality.

[0049] Dynamic update unit: Dynamically updates the comprehensive object description information based on real-time changes in perceived data.

[0050] The dynamic update unit uses a Kalman filter to synthesize object description information. Dynamic updates, specifically including: State prediction, the formula is: ; in, yes The predicted state vector (prior estimate) at time (current time); for The optimal estimated state vector (posterior estimate) at time (previous time), i.e. The comprehensive object description information after Kalman filtering at any time; yes The state transition matrix at each moment is used to describe the evolution of the comprehensive object description information in the time dimension (such as the position change and attitude adjustment of the object due to external disturbances), and reflects the correlation of the comprehensive object description information at different moments. yes The control input matrix at each time step, yes The control input at any time reflects the impact of the industrial robot's grasping actions (such as the movement of the robotic arm and the adjustment of the end effector) on the comprehensive object description information (such as the displacement of the object's position due to the operation of the robotic arm during the grasping process). yes The process noise at any given moment represents the impact of environmental interference (such as light fluctuations and sensor errors) on the accuracy of the comprehensive object description information, and is compensated for by the noise model of the Kalman filter.

[0051] Covariance prediction, the formula is: ; In the formula express The covariance matrix of the predicted state at each time step; express The covariance matrix of the optimal state at time step. Indicates process noise The covariance matrix.

[0052] Calculate the Kalman gain: ; In the formula Kalman gain; The measurement matrix maps the state space to the measurement space. To measure noise The covariance matrix; yes Measurement noise at any given moment represents the interference of sensor error on the measurement results. It needs to be suppressed by a Kalman filter to improve the accuracy of the overall object description information.

[0053] Calculate the updated state estimate: ; In the formula, for The optimal estimated state vector at time t, i.e. The comprehensive object description information after Kalman filtering at any time; yes The measured value at a given time corresponds to the multimodal data collected in real time by the multimodal sensing module, which is a direct observation result of the object's state; yes The measurement matrix at time is used to map the comprehensive object description information (state vector) to the measurement space, and to establish a mathematical relationship between abstract object features and specific sensor measurements. ; For noise measurement, the actual measured value is used. Compared with theoretical measurements based on predicted states The difference between them, Kalman gain Multiply by the measurement noise to obtain the desired predicted state. The amount of correction performed.

[0054] Update the covariance matrix: .

[0055] In the formula, for The covariance matrix of the optimal state at time step.

[0056] (3) Crawling strategy generation module: The crawling strategy generation module includes a crawling point planning unit, a crawling path planning unit, and a crawling intensity adjustment unit, which are used to generate the optimal crawling strategy.

[0057] Grasp point planning unit: Based on comprehensive object description information, using the Hybrid Learning Framework Based on Depth Deterministic Policy Gradient (DDPG-H) algorithm, combined with factors such as object shape, size, and surface characteristics, it automatically generates the optimal grip point position and pose.

[0058] The hybrid learning framework incorporates the Q-learning algorithm to update the strategy during reinforcement learning; the Q-learning algorithm update formula is as follows: ; in, It is a combination of new observations ( and )right The Q value obtained after the update The state is based on historical experience. Take action below The Q value is used to reflect the effectiveness of the current crawling strategy; express The state at any given moment, i.e., the comprehensive description information of the object; express The actions at any given moment, i.e., the grasping strategy; The learning rate affects the convergence speed of the grasping strategy optimization. yes The reward value is generated based on the actual effect of the grasping strategy. For example, if the grasping is stable (no slippage, no collision), the reward value is positive. If the object slips, is damaged due to excessive force, or there is a collision on the path, the reward value is negative. It is used to give feedback on whether the grasping action is successful or not. It is a discount factor used to weigh immediate rewards against future rewards; express The state at any given moment corresponds to the action to be performed. Subsequently, the comprehensive object description information updated by the multimodal perception module reflects the changes in the object's state or environment after the grasping strategy is executed (such as object position movement, posture change, etc.). Indicates the state Next, take all possible actions. The Q-value of the optimal action corresponds to the industrial robot's prediction of the optimal grasping strategy in the future.

[0059] The application generates a grasping strategy in combination with a hybrid learning framework based on deep deterministic policy gradient, associates the physical state, execution action and feedback reward in the grasping process, realizes autonomous learning and optimization of the grasping strategy of the industrial robot, so that the generated grasping point, path and force are more suitable for complex industrial scenes, and the grasping strategy can be compatible with regular and irregular objects.

[0060] The grasping point position and posture can be represented by a vector as follows: ; Wherein, is the position coordinate of the grasping point, is the angle of the grasping posture.

[0061] Grasping path planning unit: according to the grasping point position and the current posture of the industrial robot, a collision-free grasping path is planned.

[0062] The A* algorithm can be used for path planning, and the heuristic function is as follows: ; Wherein, is the total cost of the node , is the actual cost from the starting point to the node , is the estimated cost from the node to the end point; represents the i-th node on the path . The goal of path planning is to find a path with the minimum cost:

[0063] Wherein, is the path, is the number of nodes on the path . Grasping force adjustment unit: according to the feedback information of the force perception unit and the tactile perception unit, the grasping force is adjusted in real time. PID control is adopted, and the grasping force adjustment formula is as follows:

[0064] ;

[0065] ; ; Wherein, is the control output, that is, the control signal for adjusting the grasping force of the end effector, which directly determines the clamping force of the mechanical gripper; is the proportional coefficient of PID control, is the integral coefficient of PID control, ​​​​error signal of time, desired gripping force, actual gripping force; integral coefficient of PID control, derivative coefficient of PID control; time factor.

[0066] wherein measured by the force perception unit.

[0067] In some embodiments, the parameters of PID control and the desired gripping force are dynamically adjusted according to the feedback information of the tactile perception unit .

[0068] In some embodiments, that is, the pressure distribution asymmetry detected by the tactile perception unit and the sliding index dynamically changes. The sliding index , that is, the norm of the pressure distribution gradient detected by the tactile perception unit, the larger the gradient, the higher the possibility of sliding.

[0069] In some embodiments, , wherein is the basic gripping force estimated based on the weight of the object and the friction coefficient, is the adjustment amount calculated in real time according to the tactile feedback.

[0070] This formula dynamically adjusts the control output according to the deviation between the actual gripping force and the desired gripping force through the PID control algorithm, realizes the real-time optimization of the gripping force, and avoids the object from sliding due to insufficient force or damaging the object due to excessive force.

[0071] (4) Feedback control module: including a real-time monitoring unit (monitoring the data collected by each modal perception unit and the state information of the industrial robot executor in the grabbing process, and timely discovering problems in the grabbing process, such as grabbing deviation and object sliding), an adaptive adjustment unit (according to the feedback information of the real-time monitoring unit, the grabbing strategy is adaptively adjusted, such as adjusting the grabbing path, increasing the gripping force, and re-planning the grabbing point), and a gradual braking unit based on Lyapunov energy function (detecting abnormal conditions in the grabbing process, triggering a warning when the pressure distribution asymmetry β is greater than a preset threshold (such as 0.25), and immediately triggering a protection mechanism to ensure human-machine safety).

[0072] The present application has the following characteristics: The comprehensive design of the multi-modal perception module, including the coordinated work of visual, tactile, force sense and auditory perception units, provides rich heterogeneous sensing information for industrial robots to decouple information.

[0073] The application of deep learning algorithms in the data fusion and processing module realizes effective fusion and dynamic updating of different modal data, providing accurate input for the generation of grasping strategies.

[0074] The use of a hybrid learning framework based on deep deterministic policy gradient in the grasping strategy generation module generates optimal grasping point positions and poses, as well as collision-free grasping paths and real-time adjustments of grasping force.

[0075] The real-time monitoring, adaptive adjustment and Lyapunov energy function progressive braking function of the feedback control module ensure the smooth progress of the grasping process and the safety of man-machine.

[0076] Compared with the prior art, the present application has the following advantages: The present application adds a tactile / force sense feedback mechanism, solving the problem that the prior art cannot detect grasping stability (such as object slipping). Through the dynamic updating unit, the grasping point is corrected in real time to avoid positioning failure caused by object displacement. The hybrid learning framework based on deep deterministic policy gradient generates grasping strategies, which can be compatible with regular and irregular objects, breaking through the template limitation of the prior art; auditory perception assists in judging the grasping state (such as collision sound), making up for the deficiency of single visual modal; the grasping force is adjusted in combination with surface characteristics (such as friction), avoiding the cascading failure risk caused by the mismatch of mechanical properties of the geometric center; a "three-level safety protection system" is proposed: early warning → adjustment → emergency stop.

[0077] It should be understood that the above-mentioned numbers S1-S4 are only used to distinguish and facilitate the expression of each different step, and do not necessarily constitute a limitation on the execution order between each step.

[0078] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal perception based adaptive grasping method for industrial robots, characterized in that, The method comprises the following steps: Step 1: collecting multi-modal data of objects and environment through a multi-modal perception module; the multi-modal perception module comprises at least two of a visual perception unit, a tactile perception unit, a force perception unit and an auditory perception unit; Step 2: pre-processing and feature extraction are performed on the multi-modal data, and different modal feature information is fused through a multi-modal fusion algorithm to generate comprehensive object description information; Step 3: generating an optimal grasping strategy based on the comprehensive object description information; wherein the grasping strategy comprises grasping point position and posture, grasping path and grasping force; Step 4: the industrial robot executes the optimal grasping strategy, and repeats steps 1-3 to adaptively adjust the grasping strategy until the grasping is completed.

2. The system of claim 2, wherein, After the optical imaging sensor array denoises, enhances and normalizes the image, a target detection algorithm based on a convolutional neural network (CNN) is used to detect the target object; wherein the loss function of target detection is: ； wherein, is the coordinate loss of the th target prediction box in the th grid; is the confidence loss of the th non-target prediction box in the th grid; is the class loss of the th grid; and are weight coefficients.

3. The method of claim 1, wherein, The six-dimensional force-torque sensor outputs a generalized external force vector in real time through a nonlinear decoupling algorithm; The torque is calculated based on the generalized external force vector, and the calculation formula is: ; wherein, is the torque, is the distance vector from the action point of the force to the rotation axis, is the generalized external force vector; the controller sends the calculated torque as an output instruction to the joint motor to offset the inertia disturbance of the robot arm.

4. The method of claim 1, wherein, In step 2, an adaptive multi-source sensing integration algorithm is used to realize the synchronization of different sampling frequency sensing data in time domain and space domain through the construction of a time-space aligned feature encoder, to obtain feature vectors of each modality; and a complex domain gated attention mechanism is used to dynamically adjust the weight coefficients of the feature vectors of each modality, and the feature vectors of each modality are fused to obtain comprehensive object description information; wherein the calculation formula of the complex domain gated attention mechanism is: ； wherein, is a weight coefficient of the th modality, is a score of the th modality's feature vector, is the number of modalities; wherein, , represents the real part of represents the imaginary part of is the complex unit; ; ; wherein, , and , are learnable weights and biases; The comprehensive object description information is: ； wherein, is a comprehensive object description information, is the first modal feature vector.

5. The method of claim 1, wherein, A Kalman filter is used to dynamically update the comprehensive object description information.

6. The method of claim 1, wherein, Step 3 comprises: Based on the comprehensive object description information, an optimal grasping point position and posture are generated using a hybrid learning framework based on deep deterministic policy gradient; wherein the hybrid learning framework updates the policy in reinforcement learning by combining a Q-learning algorithm; According to the grasping point position and the current posture of the industrial robot, a collision-free and minimum-cost grasping path is planned; According to the feedback information of the force perception unit and the tactile perception unit, the grasping force is adjusted.

7. The method of claim 4, wherein, Using An algorithm is used for path planning, whose heuristic function is: ； wherein, is the total cost of the node , is the actual cost from the start to the node , is the estimated cost from the node to the end; denotes the thnode on the path ; The goal of path planning is to find a path with minimum cost: ； wherein, is the path, is the number of nodes on the path .

8. The method of claim 4, wherein, The grasping force adjustment unit uses PID control to dynamically adjust the control output according to the deviation between the actual grasping force and the expected grasping force, so as to realize real-time optimization of the grasping force; wherein the actual grasping force is measured by the force perception unit; the expected grasping force and the parameters of the PID control are dynamically adjusted according to the feedback information of the tactile perception unit.

9. The method of claim 1, wherein, The method further comprises: triggering a safety protection mechanism when an anomaly is detected during the grasping process; During the grasping process, the pressure distribution asymmetry degree β of the end effector is detected through a distributed piezoresistive sensing network, and when β is greater than a preset threshold, a warning is triggered and the safety protection mechanism is started; The starting safety protection mechanism comprises an asymptotic braking unit based on a Lyapunov energy function, which controls the robot to stop gradually, specifically comprising: calculating the braking torque required by each joint in real time according to the Lyapunov energy function and the current state of the robot; sending the calculated braking torque instruction to the joint motor to gradually reduce the speed of the robot; during the braking process, the speed and energy state of the robot are continuously monitored until it is confirmed to be completely stopped.

10. A multimodal perception based adaptive grasping system for industrial robots, characterized in that, Comprise: A multi-modal perception module comprising at least two of a visual perception unit, a tactile perception unit, a force perception unit, and an auditory perception unit, for collecting multi-modal data of objects and environments; A data fusion and processing module for pre-processing and feature extraction of the multi-modal data, and for fusing feature information of different modalities through a multi-modal fusion algorithm to generate comprehensive object description information; A grasping strategy generation module for generating an optimal grasping strategy based on the comprehensive object description information; wherein the grasping strategy comprises a grasping point position and attitude, a grasping path, and a grasping force; A feedback control module for controlling the industrial robot to execute the optimal grasping strategy, and for adaptively adjusting the grasping strategy based on the multi-modal perception module, the data fusion and processing module, and the grasping strategy generation module until the grasping is completed.

Images