A multi-modal perception fusion system and method for a link-type dexterous hand
By integrating multimodal sensors and fusion algorithms into the dexterous hand, the problem of incomplete and inaccurate information in existing dexterous hand perception systems in complex environments is solved, enabling accurate perception and adaptive operation of objects and improving its application capabilities in complex tasks.
Patent Information
- Application Number
- CN202511520243.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-23
- Publication Date
- 2026-06-23
AI Technical Summary
Existing dexterous hand sensing systems struggle to accurately acquire object information in complex lighting environments or when objects have similar surface textures. Conflicts and redundancies exist among data from multiple sensors, failing to provide reliable sensing support and limiting their application in complex tasks.
It employs an array of tactile sensors distributed on the finger contact surface, pressure sensors installed on the finger joints and grasping parts, proximity sensors arranged at the fingertips, and visual sensors configured on the main body or externally. Combined with a multimodal perception fusion algorithm unit, including a preprocessing module, a feature extraction module, a fusion decision module, and a decision output module, it performs data fusion and control through mean filtering, improved DS evidence theory, and the maximum probability principle.
It achieves precise perception of the environment and the object being manipulated, enhancing the dexterous hand's operational capabilities in complex tasks. Through the collaborative layout of multimodal sensors and noise suppression, it enables adaptive and precise operation, eliminates sensor signal interference, and provides unified decision support.
Smart Images
Figure CN122253159A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot perception and control technology, and more specifically, to a multimodal perception fusion system and method for a linkage-type dexterous hand. Background Technology
[0002] In robotic applications, linkage-type dexterous hands are widely used in industrial production, medical assistance, and scientific research due to their structural advantages. However, existing dexterous hand perception systems have significant shortcomings.
[0003] Traditional single-sensor methods, such as relying solely on visual sensors, struggle to accurately acquire object information in complex lighting environments or when objects have similar surface textures. Conversely, relying solely on tactile sensors fails to capture the overall shape and spatial position of objects. Even with simple combinations of multiple sensors, the lack of an effective data fusion mechanism leads to conflicts and redundancy between different sensor types, resulting in incomplete and inaccurate sensory information. This fails to provide reliable support for the precise operation of dexterous hands, limiting the application of linkage-based dexterous hands in complex tasks.
[0004] Therefore, there is an urgent need for a system and method that can deeply integrate multiple sensing technologies to enhance the dexterous hand's ability to perceive the environment and the objects being manipulated. Summary of the Invention
[0005] The purpose of this invention is to provide a multimodal perception fusion system and method for a linkage-type dexterous hand to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention aims to provide a multimodal perception fusion system for a linkage-type dexterous hand, comprising:
[0007] An array of tactile sensors distributed on the surface that the finger touches is used to collect information on the pressure distribution and surface texture of the object in real time;
[0008] A pressure sensor installed between the finger joints and the gripping area is used to measure the magnitude of the gripping force.
[0009] A proximity sensor placed at the tip of the finger is used to detect the distance between the finger and an object using infrared or ultrasonic principles.
[0010] Visual sensors, configured on or outside the dexterous hand, are used to collect visual information about the shape, position, and posture of objects.
[0011] The multimodal perception fusion algorithm unit includes a preprocessing module, a feature extraction module, a fusion decision module, and a decision output module, which are used to preprocess sensor data, extract features, make fusion decisions, and perform closed-loop control.
[0012] As a further improvement to this technical solution, the preprocessing module adopts a mean filtering algorithm, and the filtering formula is:
[0013] Filtered signal ,in, The radius of the filtering window. The original signal sequence is used to eliminate high-frequency noise from tactile and pressure sensors.
[0014] As a further improvement to this technical solution, the feature extraction module includes a pressure change rate calculation unit, the formula of which is:
[0015] Pressure change rate ,in, and These are pressure values at adjacent time points, used to monitor the slippage trend of objects in real time.
[0016] As a further improvement to this technical solution, the fusion decision module adopts an improved DS evidence theory, and the fusion formula is as follows:
[0017] Basic probability assignment function ,in, The normalization constant is , , The evidence consists of tactile, visual, and proximity sensors, respectively.
[0018] As a further improvement to this technical solution, the decision output module selects propositions that meet the conditions based on the principle of maximum probability. As a result of the decision:
[0019] ,in, For the set of all propositions.
[0020] As a further improvement to this technical solution, the proximity sensor triggers the visual sensor to focus and capture images when the distance to the object is set to a threshold, and at the same time activates the tactile sensor array to enter a high-sensitivity mode.
[0021] This invention also provides a multimodal perception fusion method for a linkage-type dexterous hand, comprising the following steps:
[0022] S1. Multimodal data synchronous acquisition, specifically:
[0023] The pressure distribution and surface texture signals of the object's contact surface are collected using a tactile sensor array;
[0024] The gripping force signal of the finger joints is collected by a pressure sensor;
[0025] Real-time distance signals between a finger and an object are collected using a proximity sensor;
[0026] The shape, position, and posture image information of objects are acquired through a visual sensor;
[0027] S2. Data preprocessing: Mean filtering and noise reduction are performed on the tactile and pressure signals, and grayscale conversion, noise reduction and edge detection are performed on the visual images.
[0028] S3. Dynamic Feature Extraction: Calculate the rate of pressure change to monitor the slippage trend of objects;
[0029] S4. Multimodal evidence fusion decision: The improved DS evidence theory is used to fuse tactile, visual and proximity sensor data, and the decision results are output based on the principle of maximum probability.
[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0031] 1. In this multimodal perception fusion system and method for a linkage-type dexterous hand, the spatial collaborative layout of multimodal sensors and the noise suppression preprocessing module are set up to achieve a comprehensive improvement in the accuracy of environmental perception. A dense array of tactile sensors is arranged on the finger contact surface to accurately capture the microscopic texture features of the object surface. High-sensitivity pressure sensors are embedded in the joint load-bearing nodes to provide real-time feedback on changes in grasping force. A proximity sensor is integrated into the fingertip to provide millimeter-level distance warning before contact with the object. A three-dimensional spatial positioning is constructed in conjunction with a global vision sensor. To address the sensor signal interference problem, a sliding window mean filtering technique is used to effectively suppress abnormal fluctuations caused by electromagnetic noise.
[0032] 2. In the multimodal perception fusion system and method of the linkage-type dexterous hand, an improved evidence fusion mechanism and dynamic response closed-loop control are set up to achieve the effect of adaptive and precise operation. In response to the problem of multi-sensor data conflict, an evidence fusion algorithm based on probability conflict analysis is adopted to generate a unified decision by integrating tactile texture information, visual material judgment and distance parameters. When the slipping trend of an object is detected, the control closed loop is triggered by calculating the pressure change rate in real time. The proportional control unit responds quickly to the force deviation, the integral unit eliminates historical accumulated errors, and the differential unit predicts the slipping trend and adjusts in advance. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of the multimodal perception fusion system of the linkage-type dexterous hand of the present invention. Detailed Implementation
[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0035] In one specific embodiment, such as Figure 1 As shown, this invention provides a multimodal perception fusion system for a linkage-type dexterous hand, featuring an innovative hardware layout. A tactile sensor array, pressure sensor, and proximity sensor are integrated into the finger area of the dexterous hand, while a visual sensor is configured at a suitable location on the main body or external part of the dexterous hand.
[0036] 1. Tactile Sensor Array: Composed of multiple miniature tactile sensors distributed across the surface area where the finger contacts the object. Each tactile sensor can collect information such as pressure distribution and surface texture at the contact point in real time, converting it into an electrical signal output. For example, a tactile sensor can sense whether the object's surface is rough sandpaper or smooth glass, providing surface characteristic information for subsequent operations.
[0037] 2. Pressure sensor: Installed between the finger joint and the gripping part, it is used to accurately measure the force applied by the fingers when gripping an object. Its output signal is linearly related to the gripping force, and can provide real-time feedback of the gripping force value, making it easy for the control system to adjust the gripping force according to the weight and material of the object.
[0038] 3. Proximity sensor: Using infrared or ultrasonic principles, it is placed at the tip of the finger. It can detect the distance between the finger and an object, providing distance information in advance when the finger approaches the object, helping the dexterous hand to adjust its movement posture in advance and avoid collisions.
[0039] 4. Vision Sensor: A high-definition camera or depth camera is selected and installed on top of the dexterous hand or fixed in a suitable position in the working environment. It can collect visual information such as the shape, color, position, and posture of objects, and extract key features of the objects, such as contours and dimensions, through image recognition algorithms.
[0040] The multimodal sensing fusion algorithm is as follows:
[0041] 1. Data preprocessing stage - Derivation and function of mean filtering formula.
[0042] (1) Derivation process.
[0043] In actual data acquisition, sensor signals are easily affected by environmental noise, leading to data fluctuations. Mean filtering smooths the signal and reduces the impact of noise by calculating the average value of data within a certain window. Its basic idea is to utilize the correlation between adjacent data points and replace the current data point with the mean value to achieve noise reduction.
[0044] (2) Derivation process.
[0045] Let the original signal sequence be , where n represents the sampling time. The filter window radius is... This means that, centered on the current sampling point x(n), M data points before and after it are selected for calculation. The filtered signal y(n) is the average value of the data within the window, calculated using the following formula:
[0046] Filtered signal This is used to eliminate high-frequency noise from tactile and pressure sensors. The derivation process is as follows:
[0047] The number of data points within the window is 2M+1 (including the current point x(n) and the M points before and after it). The average of these summed data points is used to obtain y(n). For example, when M=1, The signal is smoothed by replacing the current point's data with the average of three adjacent points.
[0048] (3) Functions and Operations.
[0049] This formula is primarily used to process the electrical signals output by tactile and pressure sensors. In practical applications, it effectively eliminates high-frequency jitter caused by electromagnetic interference and sensor noise, resulting in smoother pressure values and tactile signals, providing reliable data for subsequent feature extraction. For example, when grasping objects, it avoids misjudgments due to sudden pressure changes caused by noise, ensuring the accuracy of grasping force control.
[0050] 2. Feature extraction stage - Derivation and function of pressure change rate formula.
[0051] (1) Derivation process.
[0052] During the grasping process, the dynamic changes in pressure can reflect the object's state (such as whether it slips or whether the contact is stable). The rate of change of pressure extracts the dynamic characteristics of pressure by calculating how quickly the pressure value changes between adjacent moments, providing a basis for judging the grasping state.
[0053] (2) Derivation process.
[0054] Assuming adjacent time points and The pressure values are respectively and Pressure change rate The calculation formula is:
[0055] This formula is based directly on the definition of the rate of change, dividing the pressure difference by the time interval to obtain the pressure change per unit time.
[0056] (3) Functions and Operations.
[0057] Pressure change rate characteristics can be used to monitor pressure dynamics in real time during the grasping process. When an object slips, the pressure decreases rapidly. By calculating the pressure change rate, this change can be detected in a timely manner, triggering the control system to adjust the grasping posture or force to prevent the object from falling. Simultaneously, when placing an object, the pressure change rate can be used to determine whether the object is making stable contact with the surface, assisting in precise operations.
[0058] 3. Data Fusion Stage - Derivation and Functionality of Improved DS Evidence Theory Formula
[0059] (1) Derivation process
[0060] DS evidence theory is an effective method for handling uncertain information fusion. In multimodal sensing, different sensors make uncertainties and conflicts in their judgments of the same proposition (such as "the object is made of metal"). The improved DS evidence theory resolves the evidence conflict problem by introducing a normalization constant K, and reasonably fuses the judgment information from various sensors to obtain a more accurate conclusion.
[0061] (2) Derivation process
[0062] Let the evidence corresponding to tactile and pressure data be... The evidence corresponding to visual data is The evidence corresponding to the proximity sensor data is For any proposition A in the proposition set, the formula for calculating the fused basic probability assignment function m(A) is:
[0063] ,in Let be the normalization constant. The derivation process is as follows: First,
[0064] The derivation process is as follows:
[0065] first, This indicates that the three sensor evidence bodies relate to the proposition. (and The support product of all sensors represents the degree to which they collectively support proposition A. Summing the results from different sensor data yields the unnormalized support for A. However, conflicting evidence from different sensors may exist (i.e.,...). This would lead to a non-zero sum of support values, so a normalization constant K is introduced to adjust it into a probabilistic form to ensure... ( (For the set of all propositions).
[0066] (3) Functions and Operations.
[0067] This formula plays a central role in multimodal data fusion. Taking the determination of an object's material as an example, the probability that a tactile sensor might classify an object as having a "smooth material" is... The probability that the visual sensor identifies it as "metallic material" is 100%. Distance information provided by proximity sensors can also influence the judgment. By fusing information from various sensors using this formula, a final probability judgment of whether an object is made of a certain material can be obtained, providing an accurate basis for decision output and avoiding erroneous decisions caused by misjudgments from a single sensor.
[0068] 4. Decision Output Stage - Derivation and Function of the Maximum Probability Principle Formula.
[0069] (1) Derivation process.
[0070] After obtaining the basic probability assignment functions for each proposition through data fusion, it is necessary to select the most reliable conclusion from multiple possible outcomes. The maximum probability principle, based on the idea of maximizing probability, selects the proposition with the highest probability as the final decision to determine the state or property of an object.
[0071] (2) Derivation process.
[0072] Based on the fused fundamental probability assignment function m(A), for all proposition sets If proposition A in the given text exists, then proposition A is true. satisfy:
[0073] Then the proposition As the final decision outcome.
[0074] (3) Functions and Operations.
[0075] This formula is used to transform the fused probabilistic information into explicit decision instructions. In practice, after the fusion algorithm obtains the probability that an object may be "fragile," "ordinary," or "heavy," it determines the object's attribute based on the principle of maximum probability and then outputs corresponding control instructions to the control system. For example, if the object is determined to be "fragile," the control system reduces the gripping force; if it is determined to be "heavy," it increases the gripping force and adjusts the gripping posture, thus achieving intelligent operation of the dexterous hand.
[0076] Data preprocessing: The raw data collected by each sensor is preprocessed. For the electrical signals output from the tactile and pressure sensors, filtering is performed to remove noise interference. A mean filtering algorithm is used. Let the original signal sequence be x(n), and the formula for calculating the filtered signal y(n) is:
[0077] in, This represents the radius of the filtering window. For image data acquired by the vision sensor, processing steps such as grayscale conversion, noise reduction, and edge detection are performed to enhance image features and facilitate subsequent feature extraction.
[0078] (III) Perception-control closed loop.
[0079] Based on a multimodal perception fusion system, a perception-control closed loop is constructed. After receiving the decision results output by the fusion algorithm, the control system generates control commands according to the preset operation tasks and strategies, adjusting the motor drive system of the dexterous hand to control the movement of the fingers and the grasping force. Simultaneously, the perception system continuously collects information in real time and feeds it back to the control system. The control system adjusts the control commands in real time based on the new feedback information, forming a closed-loop control. For example, if the object slips during grasping, the tactile and pressure sensors detect changes in pressure distribution and magnitude. After making a decision through the fusion algorithm, the control system immediately adjusts the gripping posture and force of the fingers to ensure stable object grasping.
[0080] Derivation of the closed-loop formula for sensing and control of a linkage-type dexterous hand:
[0081] In the perception-control closed loop constructed based on a multimodal perception fusion system, a series of formulas are used to achieve precise control of the dexterous hand, ensuring that it can dynamically adjust its movements according to changes in the environment and the object being manipulated. The following section derives the formulas from aspects such as deviation calculation, control quantity generation, system output, and feedback updates.
[0082] I. Deviation Calculation.
[0083] The control system needs to adjust control commands based on the difference between the decision results output by the fusion algorithm and the target state. Let the vector formed by the current object state information (such as position, gripping force, etc.) output by the fusion algorithm be denoted as . The preset target state vector is Then the state deviation vector The calculation formula is:
[0084] Taking gripping force control as an example, if the target gripping force is The actual gripping force is currently obtained by processing the pressure sensor through a fusion algorithm. Then the gripping force deviation This deviation reflects the difference between the current state and the target state, and is the basis for subsequent control variable calculations.
[0085] II. Control quantity generation.
[0086] The proportional-integral-derivative (PID) control algorithm is used to generate control commands. This is a commonly used and effective closed-loop control algorithm. The formula for calculating the control quantity u is:
[0087] in:
[0088] This is a proportional gain, used for rapid response to deviations. Its function is to adjust the control input proportionally to the magnitude of the current deviation, thereby accelerating the system's response speed. For example, when the gripping force deviation is large, a larger proportional gain is used. The value enables the control system to rapidly increase or decrease the motor output in order to quickly reduce the gripping force deviation.
[0089] The integral term is primarily used to eliminate steady-state error. During long-term control processes, even small deviations can persist and cause the final result to deviate from the target. The integral term accumulates the deviation over time, continuously adjusting the control input until the steady-state error reaches zero.
[0090] The differential term is used to predict the trend of deviation changes and adjust the system in advance to enhance system stability. For example, when an object is about to slip and the gripping force deviation begins to change rapidly, the differential term can adjust the control quantity in advance according to the rate of change of the deviation to prevent the deviation from increasing further.
[0091] Taking dexterity hand finger joint angle control as an example, let the current joint angle deviation be... The control quantity u calculated by the above PID formula can be converted into the motor speed or torque adjustment value, thereby controlling the joint angle to move closer to the target angle.
[0092] III. System Output and Feedback Updates.
[0093] The control variable u, after conversion, is used to adjust the motor drive system of the dexterous hand. Assuming the input-output relationship of the motor is linear (which can be more accurately described in practice using a motor model), the output torque or speed of the motor... The relationship with the control variable u is as follows:
[0094] Where G is the gain coefficient of the motor drive system, which reflects the degree to which the control quantity amplifies or reduces the motor output, and is determined by the characteristics of the motor and the drive circuit. The torque or speed output by the motor drives the finger movement, changing the finger's position, posture, and gripping force.
[0095] The sensing system continuously collects state information of fingers and objects in real time, and then inputs the newly collected data back into the multimodal sensing fusion system for processing to obtain an updated state vector. The control system recalculates the deviation based on the new state vector and the target state vector, that is:
[0096] The control input is then adjusted again using the PID algorithm based on the new deviation value, and this cycle is repeated to form a closed-loop control. For example, if the object slips during the grasping process, the pressure sensor and tactile sensor collect the pressure distribution and magnitude changes. After multimodal perception fusion, a new state vector is obtained. The calculated new deviation prompts the control system to adjust the control input, change the motor output, and adjust the finger gripping posture and force to ensure stable grasping of the object.
[0097] Through the synergistic effect of the above formulas, a complete closed-loop control process from perception, decision-making, control to feedback is realized for the linkage-type dexterous hand, enabling the dexterous hand to accurately and stably complete various operational tasks in complex and ever-changing operating environments.
[0098] The specific implementation method is as follows:
[0099] (a) Hardware installation and debugging.
[0100] On the finger surface of the linkage-type dexterous hand, a tactile sensor array is installed according to a predetermined layout, ensuring that each sensor fits tightly against the finger surface without affecting normal finger movement. Pressure sensors are installed at key stress points of the finger joints and precisely calibrated to ensure accurate pressure measurements. Proximity sensors are installed at appropriate positions on the fingertips, and the detection angle and range are adjusted. A vision sensor is fixed on top of the dexterous hand or in the working environment, and the camera angle and parameters are adjusted to ensure clear acquisition of object image information. After installation, all sensors are uniformly debugged to check whether data acquisition and transmission are normal.
[0101] (ii) Algorithm training and optimization.
[0102] 1. Data Acquisition: In different operating scenarios, use dexterous hands to grasp various types of objects, including items of different shapes, materials, and weights, and collect raw data from various sensors during the grasping process to build a dataset.
[0103] 2. Algorithm Training: The multimodal perception fusion algorithm is trained using the collected dataset. The parameters of the basic probability assignment function in the DS evidence theory are optimized using machine learning algorithms to improve the accuracy of the fusion results. For the CNN network, a large amount of image data is used for training, and network parameters are adjusted to enhance image feature extraction capabilities.
[0104] 3. System Testing and Optimization: The trained algorithm is applied to a real-world dexterous hand system to test various operational tasks, such as grasping, placing, and transporting. Based on the test results, problems in the system are analyzed, and algorithm parameters and system configuration are further optimized to improve the dexterous hand's perception and operational performance.
[0105] In one specific embodiment, the present invention also provides a multimodal perception fusion method for a linkage-type dexterous hand, comprising the following steps:
[0106] S1. Multimodal data synchronous acquisition, specifically:
[0107] The pressure distribution and surface texture signals of the object's contact surface are collected using a tactile sensor array;
[0108] The gripping force signal of the finger joints is collected by a pressure sensor;
[0109] Real-time distance signals between a finger and an object are collected using a proximity sensor;
[0110] The shape, position, and posture image information of objects are acquired through a visual sensor;
[0111] S2. Data preprocessing: Mean filtering and noise reduction are performed on the tactile and pressure signals, and grayscale conversion, noise reduction and edge detection are performed on the visual images.
[0112] S3. Dynamic Feature Extraction: Calculate the rate of pressure change to monitor the slippage trend of objects;
[0113] S4. Multimodal evidence fusion decision: The improved DS evidence theory is used to fuse tactile, visual and proximity sensor data, and the decision results are output based on the principle of maximum probability.
[0114] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.
Claims
1. A multimodal perception fusion system for a linkage-type dexterous hand, characterized in that, include: An array of tactile sensors distributed on the surface that the finger touches is used to collect information on the pressure distribution and surface texture of the object in real time; A pressure sensor installed between the finger joints and the gripping area is used to measure the magnitude of the gripping force. A proximity sensor placed at the tip of the finger is used to detect the distance between the finger and an object using infrared or ultrasonic principles. Visual sensors, configured on or outside the dexterous hand, are used to collect visual information about the shape, position, and posture of objects. The multimodal perception fusion algorithm unit includes a preprocessing module, a feature extraction module, a fusion decision module, and a decision output module, which are used to preprocess sensor data, extract features, make fusion decisions, and perform closed-loop control.
2. The multimodal perception fusion system of the linkage-type dexterous hand according to claim 1, characterized in that, The preprocessing module employs a mean filtering algorithm, and the filtering formula is as follows: Filtered signal ,in, The radius of the filtering window. The original signal sequence is used to eliminate high-frequency noise from tactile and pressure sensors.
3. The multimodal perception fusion system of the linkage-type dexterous hand according to claim 1, characterized in that, The feature extraction module includes a pressure change rate calculation unit, with the following formula: Pressure change rate ,in, and These are pressure values at adjacent time points, used to monitor the slippage trend of objects in real time.
4. The multimodal perception fusion system of the linkage-type dexterous hand according to claim 1, characterized in that, The fusion decision module adopts an improved DS evidence theory, and the fusion formula is as follows: Basic probability assignment function ,in, The normalization constant is , , The evidence consists of tactile, visual, and proximity sensors, respectively.
5. The multimodal perception fusion system of the linkage-type dexterous hand according to claim 1, characterized in that, The decision output module selects propositions that meet the conditions based on the principle of maximum probability. As a result of the decision: ,in, For the set of all propositions.
6. The multimodal perception fusion system of the linkage-type dexterous hand according to claim 1, characterized in that, When the proximity sensor triggers the visual sensor to focus and capture an image at a set threshold distance from the object, it simultaneously activates the tactile sensor array to enter a high-sensitivity mode.
7. A multimodal perception fusion method for a linkage-type dexterous hand, characterized in that, Includes the following steps: S1. Synchronous acquisition of multimodal data, specifically: The pressure distribution and surface texture signals of the object's contact surface are collected using a tactile sensor array; The gripping force signal of the finger joints is collected by a pressure sensor; Real-time distance signals between a finger and an object are collected using a proximity sensor; The shape, position, and posture image information of objects are acquired through a visual sensor; S2. Data preprocessing: Mean filtering and noise reduction are performed on the tactile and pressure signals, and grayscale conversion, noise reduction and edge detection are performed on the visual images. S3. Dynamic Feature Extraction: Calculate the rate of pressure change to monitor the slippage trend of objects; S4. Multimodal evidence fusion decision: The improved DS evidence theory is used to fuse tactile, visual and proximity sensor data, and the decision results are output based on the principle of maximum probability.