Robot hand adaptive control system based on multi-modal sensing and reinforcement learning

Abstract

The invention discloses a robot hand self-adaptive control system based on multi-modal sensing and reinforcement learning, and relates to the technical field of robots, the robot hand self-adaptive control system comprises a sensing layer, a signal processing layer and a calculation layer, the signal processing layer is correspondingly connected with an integrated temperature and humidity compensation module through each sensor unit, and the integrated temperature and humidity compensation module is connected with the calculation layer. The pressure detection value change of each sensor unit due to temperature and humidity is compensated, so that the detection precision is improved; through a geometric constraint introduction module and a pressure field reconstruction module, the geometric constraint introduction module uses a geometric shape of an object contact surface as prior information to correct a pressure field, and geometric constraint is introduced to ensure that the pressure field accords with a physical law at an object edge or a curved surface area; the pressure field reconstruction module is used for converting discrete sensor data into a continuous pressure distribution field, meanwhile, geometric constraint information is fused into the continuous pressure distribution field, and the mechanical property of the contact area of an object and the hand of the robot can be accurately described through reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a robot hand adaptive control system based on multimodal sensing and reinforcement learning. Background Art

[0002] The reason why an intelligent robot is called an intelligent robot is that it has a fairly developed "brain". The central processing unit works in the brain, and this computer has a direct connection with the person who operates it. The most important thing is that such a computer can perform actions arranged according to the purpose. Because of this, we say that this kind of robot is a real robot, although their appearance may be different. Intelligent robots are the concentrated embodiment of scientific and technological innovation. Among them, the grasping action is one of the most core operations of intelligent robots.

[0003] According to the announcement number: CN118003362B - A robot hand device with adaptive grasping function and method thereof, which records "a first pressure sensor is embedded in the first chuck (13); a second pressure sensor is embedded in the second chuck (14); when the first pressure sensor or the second pressure sensor detects pressure, counting is performed, and the controller obtains the total count according to the counting result; the controller determines whether the total count is less than the quantity threshold, and when the total count is less than the quantity threshold, the pressure threshold is increased." The technical personnel in this field can know that although the reference patent monitors the pressure at the chuck end in real time, it is well known that the robot has many working environments and the surface temperature of the grasped objects is different, which may easily lead to inaccurate pressure detection due to temperature changes when the robot grasps.

[0004] According to CN114347033B-Robot object grasping method, device, robot and storage medium, it records "a dynamic grasping module, which is used to obtain the current posture of the robot and the three-dimensional environmental information near the target object fed back by the visual sensor when the robot moves to the optimal grasping point and the target object is within the optimal grasping range, and synchronously transmit the current posture of the robot, the current 3D position of the target object, the current posture of the target object and the three-dimensional environmental information near the target object to a cloud server connected to the robot, so that the robot digital twin in the cloud server can grasp the target object according to the current posture of the robot, the current 3D position of the target object, the current posture of the target object and the three-dimensional environmental information near the target object. The invention relates to a method for grasping an object by performing obstacle avoidance planning in three-dimensional space based on three-dimensional environmental information, and obtaining an initial grasping path and an initial grasping action; receiving the initial grasping path and the initial grasping action sent by the cloud server; and in the process of approaching the target object according to the initial grasping path and the initial grasping action, continuously adjusting the initial grasping path and the initial grasping action according to the distance between the robot arm and the target object, the current 3D position of the target object, the current posture of the target object and the three-dimensional environmental information near the target object until the target object is grasped. Those skilled in the art can know that the grasping of objects in the reference patent is only considered from the perspective of the grasping path and the grasping action, but improvements are made on the deformation of the object or the adjustment of the grasping force after the grasping, which may easily cause damage to the grasped object.

[0005] In summary, a robot hand adaptive control system based on multimodal sensing and reinforcement learning is designed. Summary of the invention

[0006] In order to overcome the above-mentioned deficiencies, the present invention provides a robot hand adaptive control system based on multimodal sensing and reinforcement learning.

[0007] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0008] A robot hand adaptive control system based on multimodal sensing and reinforcement learning, comprising a sensing layer, a signal processing layer and a computing layer, wherein the sensing layer is a flexible piezoresistive sensor array, and the sensing layer transmits the collected signal to the computing layer through the signal processing layer;

[0009] The computing layer includes sensor data preprocessing module, thin plate spline interpolation module, geometric constraint fusion module and dynamic analysis contact module;

[0010] The sensor data preprocessing module uses wavelet noise reduction or Kalman filtering to eliminate noise in the sensor data after filtering;

[0011] The thin plate spline interpolation module builds a thin plate spline interpolation model based on sensor positions and readings:

[0012]

[0013] Among them, (x i ,y i ) is the position coordinate of the i-th sensor, ω i is the weight coefficient of the i-th sensor,

[0014] It is a thin plate spline kernel function, which is used to describe the influence relationship between sensors, describes the spatial relationship between sensors, and reflects the local influence of sensors on target points. When the target point is far away from the sensor, the kernel function value approaches zero, indicating that the influence is weakened.

[0015] a0, a1, and a2 are linear term coefficients. The linear term a0+a1x+a2y is used to fit the global trend of the pressure field, reflecting the overall change law of the pressure distribution;

[0016] Solve the weight coefficient and linear term coefficient by the least square method;

[0017] The geometric constraint fusion module includes a geometric constraint introduction module and a pressure field reconstruction module. The geometric constraint introduction module uses the geometric shape of the contact surface of the object as prior information to correct the pressure field. By introducing geometric constraints, it can ensure that the pressure field conforms to the physical laws at the edge or curved surface of the object. The pressure field reconstruction module converts discrete sensor data into a continuous pressure distribution field and integrates geometric constraint information at the same time. Through reconstruction, it can accurately describe the mechanical properties of the contact area between the object and the robot hand. The introduction of geometric constraints ensures the rationality of the pressure field at the edge or curved surface of the object and avoids pressure distribution that does not conform to the physical laws.

[0018] The dynamic analysis contact module monitors the mechanical interaction between the object and the robot hand in real time, including the slip detection module and the grip optimization module to ensure the stability and safety of the grasping.

[0019] Preferably, the flexible piezoresistive sensor array forms a detection grid composed of 10×10 pressure sensor units, covering the contact area of ​​the robot hand. Each sensor unit works independently and supports multi-point pressure detection. Each pressure sensor unit transmits data to the computing layer through the signal processing layer.

[0020] Preferably, the sensor unit spacing is 2 mm to achieve high-density pressure detection. The surface of the sensor uses polyimide or polydimethylsiloxane as a flexible substrate, and the sensor is equipped with a stretchable wire to ensure that the sensor is bendable and stretchable, and to ensure that the sensor can fit the complex curved surface of the robot hand. The shape of the sensor can be 3D printed according to the application scenario.

[0021] Preferably, each of the sensor units is connected to an integrated temperature and humidity compensation module, which performs real-time temperature and humidity compensation on the detection data of the corresponding sensor unit, and the temperature and humidity compensation includes temperature compensation and humidity compensation.

[0022] The temperature compensation formula is P2 = P1·(1+αΔT), where P1 is the original pressure value, ΔT is the temperature change, and α is the temperature compensation coefficient. Real-time temperature compensation is achieved through an embedded processor to ensure measurement accuracy, significantly reduce the impact of temperature on sensor output, and improve the robustness of the system in complex environments.

[0023] The humidity compensation formula is P2 = P1·(1+βΔH), where P1 is the original pressure value, ΔH is the humidity change, and β is the temperature compensation coefficient. Real-time temperature compensation is achieved through an embedded processor to ensure measurement accuracy.

[0024] The temperature and humidity compensation formula is P2=P1·(1+αΔT+βΔH).

[0025] Preferably, the geometric constraint introduction module comprises the following steps:

[0026] S11, obtaining geometric shape information, obtaining the geometric shape of the contact surface of the object through a visual sensor (such as a camera) or a CAD model. For example, the contact surface of the object can be expressed as a surface equation S(x, y, z)=0;

[0027] S12, geometric constraint modeling, converting geometric shape information into constraint conditions to form a corrected pressure field. For example, in the edge area of ​​the object, the pressure value should be close to zero; in the curved surface area, the pressure distribution should conform to the curvature change;

[0028] S13, the constraints are integrated into the difference model, the geometric constraints are integrated into the thin plate spline interpolation model, the interpolation result is corrected, and the pressure value of the interpolation function is forced to be zero in the edge area of ​​the object: f(x,y)=0, when in Indicates the edge of the contact surface of the object.

[0029] Preferably, the pressure field reconstruction module comprises the following steps:

[0030] S21, Geometric constraint fusion, integrates geometric constraints into the thin plate spline interpolation model, corrects the interpolation result, and adjusts the weight of the interpolation kernel function according to the curvature in the surface area: ω' i =ω i ·k(x i ,y i ) where k(x i ,y i ) indicates that the surface is at (x i ,y i ) at the point of curvature;

[0031] S22, generating a pressure field, generating a continuous pressure distribution map in the target area, that is, the surface of the robot hand, for example, calculating the pressure value of each point through an interpolation function: P(x, y) = f(x, y).

[0032] Preferably, the slip detection module comprises the following steps:

[0033] S31. Pressure center calculation. The pressure center is the weighted average position of the pressure distribution in the contact area. It is used to describe the overall trend of the pressure field. The position of the pressure center reflects the distribution characteristics of the pressure field. If the pressure center is offset greatly, it may indicate that the object is slipping.

[0034] The calculation formula is

[0035] Among them, p i is the pressure value of the ith sensor, (x i ,y i ) is the position coordinate of the i-th sensor, COP x , COP y are the coordinates of the pressure center in the x and y directions;

[0036] S32, slip detection index, is used to quantify the migration rate of the pressure center and determine whether the object slips. The calculation formula is:

[0037] Where T is the time window length, is the rate of change of the pressure center over time, ‖·‖2 is the Euclidean norm, and the user calculates the rate of change. The larger the CMI value, the faster the pressure center migrates and the higher the possibility of object slippage. By setting the threshold, real-time warning of slippage can be issued.

[0038] Preferably, the grip strength optimization module comprises the following steps:

[0039] S41, pressure uniformity assessment, used to quantify the uniformity of pressure distribution to ensure that the object is subjected to uniform force during grasping. The calculation formula is: Among them, pi is the normalized pressure value, n is the total number of sensors, the smaller the J1 value, the more uniform the pressure distribution. When optimizing grip force, the goal is to minimize the J1 value;

[0040] S42, material deformation constraint modeling, used to determine the safe grip range to prevent objects from being damaged or slipping, the calculation formula is F min ≤f≤F max , where F min F is the minimum safe grip to prevent objects from slipping. max is the maximum safe grip force to prevent object damage, and f is the current grip force value. By constraining the grip force range, the safety and stability of the grasping process are ensured;

[0041] S43, multi-objective optimization, is used to find the optimal grip value between pressure uniformity and material deformation constraints. The calculation formula is: minimize‖f-J1‖2subiect to F min ≤f≤F max , where f is the optimized grip force value, J1 is the pressure uniformity index, ‖·‖2 is the Euclidean norm, and the user calculates the size of the objective function and finds the optimal grip force value that can ensure pressure uniformity and meet the material deformation constraint through optimization.

[0042] Preferably, the working steps of the computing layer are as follows:

[0043] S1, signal acquisition, real-time detection of multiple signals through the sensing layer, including pressure, ambient temperature, and humidity;

[0044] S2, signal processing, temperature and humidity compensation of the sensor pressure detection signal through the signal processing layer;

[0045] S3, data preprocessing, after filtering, using wavelet noise reduction or Kalman filtering to eliminate noise in sensor data;

[0046] S4, thin plate spline interpolation, based on the sensor position and readings, builds a thin plate spline interpolation model;

[0047] S5. Geometric constraint fusion, through the geometric constraint introduction module and the pressure field reconstruction module. The geometric constraint introduction module uses the geometric shape of the contact surface of the object as prior information to correct the pressure field. By introducing geometric constraints, it can ensure that the pressure field conforms to the physical laws at the edge or curved surface of the object. The pressure field reconstruction module converts discrete sensor data into a continuous pressure distribution field and integrates geometric constraint information. Through reconstruction, the mechanical properties of the contact area between the object and the robot hand can be accurately described.

[0048] S6. Dynamic analysis of contact: The dynamic analysis of contact module monitors the mechanical interaction between the object and the robot hand in real time, including the slip detection module and the grip optimization module, to ensure the stability and safety of grasping.

[0049] The beneficial effects of the present invention are as follows: in the robot hand adaptive control system based on multimodal sensing and reinforcement learning:

[0050] 1. In the signal processing layer, each sensor unit is connected to an integrated temperature and humidity compensation module to compensate for the changes in the pressure detection value of each sensor unit due to temperature and humidity, thereby improving the detection accuracy;

[0051] 2. Through the geometric constraint introduction module and the pressure field reconstruction module, the geometric constraint introduction module uses the geometric shape of the contact surface of the object as prior information to correct the pressure field. By introducing geometric constraints, it can ensure that the pressure field conforms to the physical laws at the edge or curved surface of the object. The pressure field reconstruction module converts discrete sensor data into a continuous pressure distribution field and integrates geometric constraint information. Through reconstruction, the mechanical properties of the contact area between the object and the robot hand can be accurately described;

[0052] 3. The dynamic analysis contact module monitors the mechanical interaction between the object and the robot hand in real time, including the slip detection module and the grip optimization module, to ensure the stability and safety of the grasping. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0054] Figure 1 It is a working step diagram of the computing layer of the present invention;

[0055] Figure 2 It is a working step diagram of the geometric constraint introduction module of the present invention;

[0056] Figure 3 is a working step diagram of the pressure field reconstruction module of the present invention;

[0057] Figure 4 is a working step diagram of the slip detection module of the present invention;

[0058] Figure 5 It is a working step diagram of the grip strength optimization module of the present invention. DETAILED DESCRIPTION

[0059] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0060] like Figure 1-Figure 5 As shown, a robot hand adaptive control system based on multimodal sensing and reinforcement learning includes a sensing layer, a signal processing layer and a computing layer. The sensing layer is a flexible piezoresistive sensor array. The sensing layer transmits the collected signal to the computing layer through the signal processing layer.

[0061] The computing layer includes sensor data preprocessing module, thin plate spline interpolation module, geometric constraint fusion module and dynamic analysis contact module;

[0062] The sensor data preprocessing module uses wavelet noise reduction or Kalman filtering to eliminate noise in the sensor data after filtering;

[0063] The thin plate spline interpolation module builds a thin plate spline interpolation model based on sensor positions and readings:

[0064]

[0065] Among them, (x i ,y i ) is the position coordinate of the i-th sensor, ω i is the weight coefficient of the i-th sensor,

[0066] It is a thin plate spline kernel function, which is used to describe the influence relationship between sensors, describes the spatial relationship between sensors, and reflects the local influence of sensors on target points. When the target point is far away from the sensor, the kernel function value approaches zero, indicating that the influence is weakened.

[0067] a0, a1, and a2 are linear term coefficients. The linear term a0+a1x+a2y is used to fit the global trend of the pressure field, reflecting the overall change law of the pressure distribution;

[0068] Solve the weight coefficient and linear term coefficient by the least square method;

[0069] The geometric constraint fusion module includes a geometric constraint introduction module and a pressure field reconstruction module. The geometric constraint introduction module uses the geometric shape of the contact surface of the object as prior information to correct the pressure field. By introducing geometric constraints, it can ensure that the pressure field conforms to the physical laws at the edge or curved surface of the object. The pressure field reconstruction module converts discrete sensor data into a continuous pressure distribution field and integrates geometric constraint information at the same time. Through reconstruction, it can accurately describe the mechanical properties of the contact area between the object and the robot hand. The introduction of geometric constraints ensures the rationality of the pressure field at the edge or curved surface of the object and avoids pressure distribution that does not conform to the physical laws.

[0070] The dynamic analysis contact module monitors the mechanical interaction between the object and the robot hand in real time, including the slip detection module and the grip optimization module to ensure the stability and safety of the grasping.

[0071] Specifically, the flexible piezoresistive sensor array forms a detection grid composed of 10×10 pressure sensor units, covering the contact area of ​​the robot hand. Each sensor unit works independently and supports multi-point pressure detection. Each pressure sensor unit transmits data to the computing layer through the signal processing layer.

[0072] Specifically, the sensor unit spacing is 2mm to achieve high-density pressure detection. The surface of the sensor uses polyimide or polydimethylsiloxane as a flexible substrate, and the sensor is equipped with a stretchable wire to ensure that the sensor is bendable and stretchable, and to ensure that the sensor can fit the complex curved surface of the robot's hand. The shape of the sensor can be 3D printed according to the application scenario.

[0073] Specifically, each sensor unit is connected to an integrated temperature and humidity compensation module, which performs real-time temperature and humidity compensation on the detection data of the corresponding sensor unit. The temperature and humidity compensation includes temperature compensation and humidity compensation.

[0074] The temperature compensation formula is P2 = P1·(1+αΔT), where P1 is the original pressure value, ΔT is the temperature change, and α is the temperature compensation coefficient. Real-time temperature compensation is achieved through an embedded processor to ensure measurement accuracy, significantly reduce the impact of temperature on sensor output, and improve the robustness of the system in complex environments.

[0075] The humidity compensation formula is P2 = P1·(1+βΔH), where P1 is the original pressure value, ΔH is the humidity change, and β is the temperature compensation coefficient. Real-time temperature compensation is achieved through an embedded processor to ensure measurement accuracy.

[0076] The temperature and humidity compensation formula is P2=P1·(1+αΔT+βΔH).

[0077] Specifically, the geometric constraint introduction module includes the following steps:

[0078] S11, obtaining geometric shape information, obtaining the geometric shape of the contact surface of the object through a visual sensor (such as a camera) or a CAD model. For example, the contact surface of the object can be expressed as a surface equation S(x, y, z)=0;

[0079] S12, geometric constraint modeling, converting geometric shape information into constraint conditions to form a corrected pressure field. For example, in the edge area of ​​the object, the pressure value should be close to zero; in the curved surface area, the pressure distribution should conform to the curvature change;

[0080] S13, the constraints are integrated into the difference model, the geometric constraints are integrated into the thin plate spline interpolation model, the interpolation result is corrected, and the pressure value of the interpolation function is forced to be zero in the edge area of ​​the object: f(x,y)=0, when in Indicates the edge of the contact surface of the object.

[0081] Specifically, the pressure field reconstruction module includes the following steps:

[0082] S21, Geometric constraint fusion, integrates geometric constraints into the thin plate spline interpolation model, corrects the interpolation result, and adjusts the weight of the interpolation kernel function according to the curvature in the surface area: ω' i =ω i ·k(x i ,y i ) where k(x i ,y i ) indicates that the surface is at (x i ,y i ) at the point of curvature;

[0083] S22, generating a pressure field, generating a continuous pressure distribution map in the target area, that is, the surface of the robot hand, for example, calculating the pressure value of each point through an interpolation function: P(x, y) = f(x, y).

[0084] Specifically, the slip detection module includes the following steps:

[0085] S31. Pressure center calculation. The pressure center is the weighted average position of the pressure distribution in the contact area. It is used to describe the overall trend of the pressure field. The position of the pressure center reflects the distribution characteristics of the pressure field. If the pressure center is offset greatly, it may indicate that the object is slipping.

[0086] The calculation formula is

[0087] Among them, p i is the pressure value of the ith sensor, (x i ,y i ) is the position coordinate of the i-th sensor, COP x , COP y are the coordinates of the pressure center in the x and y directions;

[0088] S32, slip detection index, is used to quantify the migration rate of the pressure center and determine whether the object slips. The calculation formula is:

[0089] Where T is the time window length, is the rate of change of the pressure center over time, ‖·‖2 is the Euclidean norm, and the user calculates the rate of change. The larger the CMI value, the faster the pressure center migrates and the higher the possibility of object slippage. By setting the threshold, real-time warning of slippage can be issued.

[0090] Specifically, the grip strength optimization module includes the following steps:

[0091] S41, pressure uniformity assessment, used to quantify the uniformity of pressure distribution to ensure that the object is subjected to uniform force during grasping. The calculation formula is: Among them, p i is the normalized pressure value, n is the total number of sensors, the smaller the J1 value, the more uniform the pressure distribution. When optimizing grip force, the goal is to minimize the J1 value;

[0092] S42, material deformation constraint modeling, used to determine the safe grip range to prevent objects from being damaged or slipping, the calculation formula is F min ≤f≤F max , where F min The minimum safety grip to prevent objects from slipping, F max is the maximum safe grip force to prevent object damage, and f is the current grip force value. By constraining the grip force range, the safety and stability of the grasping process are ensured;

[0093] S43, multi-objective optimization, is used to find the optimal grip value between pressure uniformity and material deformation constraints. The calculation formula is: minimize‖f-J1‖2subiect to F min ≤f≤F max , where f is the optimized grip force value, J1 is the pressure uniformity index, ‖·‖2 is the Euclidean norm, and the user calculates the size of the objective function and finds the optimal grip force value that can ensure pressure uniformity and meet the material deformation constraint through optimization.

[0094] Specifically, the working steps of the calculation layer are as follows:

[0095] S1, signal acquisition, real-time detection of multiple signals through the sensing layer, including pressure, ambient temperature, and humidity;

[0096] S2, signal processing, temperature and humidity compensation of the sensor pressure detection signal through the signal processing layer;

[0097] S3, data preprocessing, after filtering, using wavelet noise reduction or Kalman filtering to eliminate noise in sensor data;

[0098] S4, thin plate spline interpolation, based on the sensor position and readings, builds a thin plate spline interpolation model;

[0099] S5. Geometric constraint fusion, through the geometric constraint introduction module and the pressure field reconstruction module. The geometric constraint introduction module uses the geometric shape of the contact surface of the object as prior information to correct the pressure field. By introducing geometric constraints, it can ensure that the pressure field conforms to the physical laws at the edge or curved surface of the object. The pressure field reconstruction module converts discrete sensor data into a continuous pressure distribution field and integrates geometric constraint information. Through reconstruction, the mechanical properties of the contact area between the object and the robot hand can be accurately described.

[0100] S6. Dynamic analysis of contact: The dynamic analysis of contact module monitors the mechanical interaction between the object and the robot hand in real time, including the slip detection module and the grip optimization module, to ensure the stability and safety of grasping.

[0101] Example 1: Industrial robot grabbing irregular metal parts

[0102] Application scenarios: On automated production lines, industrial robots need to grasp metal parts with irregular shapes and uneven surfaces, ensuring that the surface of the parts is not damaged during the grasping process while maintaining a stable grip.

[0103] Working principle:

[0104] 1. Signal acquisition: The pressure distribution in the contact area between metal parts and the robot hand is detected in real time through a flexible piezoresistive sensor array.

[0105] 2. Signal processing: Perform temperature and humidity compensation on sensor data to ensure measurement accuracy.

[0106] 3. Data preprocessing: Use Kalman filtering to eliminate noise in sensor data and ensure data reliability.

[0107] 4. Thin plate spline interpolation: Based on the sensor position and readings, a thin plate spline interpolation model is constructed to reconstruct the continuous pressure field.

[0108] 5. Geometric constraint fusion: Obtain the geometric shape information of metal parts through visual sensors, integrate geometric constraints into the interpolation model, and correct the distribution of pressure field on the surface of parts.

[0109] 6. Dynamic contact analysis: Real-time monitoring of the mechanical interaction between metal parts and the robot hand to optimize grip force and ensure stable grasping without damaging the part surface.

[0110] Implementation Effect

[0111] Achieve fine gripping of metal parts, reducing the slip rate to less than 0.1%.

[0112] The grip force control accuracy reaches ±0.1N to avoid surface damage of parts.

[0113] Example 2: Medical robot grabbing biological tissue

[0114] Application scenarios: In minimally invasive surgery, medical robots need to grasp biological tissue (such as blood vessels) to ensure that the tissue is not damaged during the grasping process while maintaining a stable grip.

[0115] How it works

[0116] 1. Signal acquisition: The pressure distribution in the contact area between biological tissue and the robot hand is detected in real time through a flexible piezoresistive sensor array.

[0117] 2. Signal processing: Perform temperature and humidity compensation on sensor data to ensure measurement accuracy.

[0118] 3. Data preprocessing: Use wavelet noise reduction to eliminate noise in sensor data and ensure data reliability.

[0119] 4. Thin plate spline interpolation: Based on the sensor position and readings, a thin plate spline interpolation model is constructed to reconstruct the continuous pressure field.

[0120] 5. Geometric constraint fusion: Obtain the geometric shape information of biological tissues (such as the curved structure of blood vessels) through visual sensors, integrate geometric constraints into the interpolation model, and correct the distribution of pressure fields in the vascular area.

[0121] 6. Dynamic contact analysis: Real-time monitoring of the mechanical interaction between biological tissue and the robot hand to optimize grip strength and ensure stable grasping without damaging the tissue.

[0122] Implementation Effect

[0123] It can achieve fine grasping of biological tissues and reduce the slip rate to less than 0.1%.

[0124] The grip force control accuracy reaches ±0.1N to avoid tissue damage.

[0125] Example 3: Service robot grabs fragile objects

[0126] Application scenarios: In a home or commercial environment, a service robot needs to grab fragile objects (such as glasses) and ensure that the objects are not damaged during the grabbing process while maintaining a stable grip.

[0127] How it works

[0128] 1. Signal acquisition: The pressure distribution in the contact area between the glass and the robot hand is detected in real time through a flexible piezoresistive sensor array.

[0129] 2. Signal processing: Perform temperature and humidity compensation on sensor data to ensure measurement accuracy.

[0130] 3. Data preprocessing: Use Kalman filtering to eliminate noise in sensor data and ensure data reliability.

[0131] 4. Thin plate spline interpolation: Based on the sensor position and readings, a thin plate spline interpolation model is constructed to reconstruct the continuous pressure field.

[0132] 5. Geometric constraint fusion: The geometric shape information of the glass is obtained through the visual sensor, the geometric constraints are integrated into the interpolation model, and the distribution of the pressure field on the surface of the glass is corrected.

[0133] 6. Dynamic contact analysis: Real-time monitoring of the mechanical interaction between the glass and the robot hand to optimize grip force and ensure stable grasping without damaging the glass.

[0134] Implementation Effect

[0135] It can achieve fine grasping of fragile objects and reduce the slip rate to less than 0.1%.

[0136] The grip force control accuracy reaches ±0.1N to avoid damage to items.

[0137] To summarize, in this patent, through the steps of signal acquisition, signal processing, data preprocessing, thin plate spline interpolation, geometric constraint fusion and dynamic contact analysis, the system can achieve high-precision and high-stability grasping operations and adapt to the needs of objects of different shapes and materials.

[0138] The above is based on the present invention as an inspiration. Through the above description, relevant staff can make various changes and modifications without departing from the technical idea of ​​this invention. The technical scope of this invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A robot hand adaptive control system based on multimodal sensing and reinforcement learning, characterized by: It includes a sensing layer, a signal processing layer and a computing layer. The sensing layer is a flexible piezoresistive sensor array. The sensing layer transmits the collected signals to the computing layer through the signal processing layer. The computing layer includes sensor data preprocessing module, thin plate spline interpolation module, geometric constraint fusion module and dynamic analysis contact module; The sensor data preprocessing module uses wavelet noise reduction or Kalman filtering to eliminate noise in the sensor data after filtering; The thin plate spline interpolation module builds a thin plate spline interpolation model based on sensor positions and readings: Among them, (x i ,y i ) is the position coordinate of the i-th sensor, ω i is the weight coefficient of the i-th sensor, It is a thin plate spline kernel function, which is used to describe the influence relationship between sensors, describes the spatial relationship between sensors, and reflects the local influence of sensors on target points. When the target point is far away from the sensor, the kernel function value approaches zero, indicating that the influence is weakened. a0, a1, and a2 are linear term coefficients. The linear term a0+a1x+a2y is used to fit the global trend of the pressure field, reflecting the overall change law of the pressure distribution; Solve the weight coefficient and linear term coefficient by the least square method; The geometric constraint fusion module includes a geometric constraint introduction module and a pressure field reconstruction module. The geometric constraint introduction module uses the geometric shape of the contact surface of the object as prior information to correct the pressure field. By introducing geometric constraints, it can ensure that the pressure field conforms to the physical laws at the edge or curved surface of the object. The pressure field reconstruction module converts discrete sensor data into a continuous pressure distribution field and integrates geometric constraint information. Through reconstruction, the mechanical properties of the contact area between the object and the robot hand can be accurately described. The dynamic analysis contact module monitors the mechanical interaction between the object and the robot hand in real time, including the slip detection module and the grip optimization module to ensure the stability and safety of the grasping.

2. The robot hand adaptive control system based on multimodal sensing and reinforcement learning according to claim 1, characterized in that: The flexible piezoresistive sensor array forms a detection grid composed of 10×10 pressure sensor units, covering the contact area of ​​the robot hand. Each sensor unit works independently and supports multi-point pressure detection. Each pressure sensor unit transmits data to the computing layer through the signal processing layer.

3. The robot hand adaptive control system based on multimodal sensing and reinforcement learning according to claim 2, characterized in that: The sensor unit spacing is 2 mm, the surface of the sensor uses polyimide or polydimethylsiloxane as a flexible substrate, and the sensor is equipped with a stretchable wire.

4. The robot hand adaptive control system based on multimodal sensing and reinforcement learning according to claim 1, characterized in that: Each sensor unit is connected to an integrated temperature and humidity compensation module, which performs real-time temperature and humidity compensation on the detection data of the corresponding sensor unit. The temperature and humidity compensation includes temperature compensation and humidity compensation. The temperature compensation formula is P2 = P1·(1+αΔT), where P1 is the original pressure value, ΔT is the temperature change, and α is the temperature compensation coefficient. Real-time temperature compensation is achieved through an embedded processor to ensure measurement accuracy. The humidity compensation formula is P2 = P1·(1+βΔH), where P1 is the original pressure value, ΔH is the humidity change, and β is the temperature compensation coefficient. Real-time temperature compensation is achieved through an embedded processor to ensure measurement accuracy. The temperature and humidity compensation formula is P2=P1·(1+αΔT+βΔH).

5. The robot hand adaptive control system based on multimodal sensing and reinforcement learning according to claim 1, characterized in that: The geometric constraint introduction module includes the following steps: S11, obtaining geometric shape information, obtaining the geometric shape of the contact surface of the object through a visual sensor (such as a camera) or a CAD model; S12, geometric constraint modeling, converting geometric shape information into constraint conditions to form a modified pressure field; S13, integrating constraints into the difference model, integrating geometric constraints into the thin plate spline interpolation model, and correcting the interpolation results.

6. The robot hand adaptive control system based on multimodal sensing and reinforcement learning according to claim 1, characterized in that: The pressure field reconstruction module comprises the following steps: S21, geometric constraint fusion, integrating geometric constraint conditions into the thin plate spline interpolation model to correct the interpolation results; S22, generating a pressure field, generating a continuous pressure distribution map in the target area, that is, the surface of the robot hand.

7. The robot hand adaptive control system based on multimodal sensing and reinforcement learning according to claim 1, characterized in that: The slip detection module comprises the following steps: S31. Pressure center calculation. The pressure center is the weighted average position of the pressure distribution in the contact area. It is used to describe the overall trend of the pressure field. The position of the pressure center reflects the distribution characteristics of the pressure field. If the pressure center is offset greatly, it may indicate that the object is slipping. The calculation formula is Among them, p i is the pressure value of the ith sensor, (x i ,y i ) is the position coordinate of the i-th sensor, COP x , COP y are the coordinates of the pressure center in the x and y directions; S32, slip detection index, is used to quantify the migration rate of the pressure center and determine whether the object slips. The calculation formula is: Where T is the time window length, is the rate of change of the pressure center over time, ‖·‖2 is the Euclidean norm, and the user calculates the magnitude of the rate of change.

8. The robot hand adaptive control system based on multimodal sensing and reinforcement learning according to claim 1, characterized in that: The grip strength optimization module comprises the following steps: S41, pressure uniformity assessment, used to quantify the uniformity of pressure distribution to ensure that the object is subjected to uniform force during grasping. The calculation formula is: Among them, p i is the normalized pressure value, n is the total number of sensors; S42, material deformation constraint modeling, used to determine the safe grip range to prevent objects from being damaged or slipping, the calculation formula is F min ≤f≤F max , where F min F is the minimum safe grip to prevent objects from slipping. max is the maximum safe grip force to prevent object damage, and f is the current grip force value; S43, multi-objective optimization, is used to find the optimal grip value between pressure uniformity and material deformation constraints. The calculation formula is: minimize‖f-J1‖2subiect to F min ≤f≤F max , where f is the optimized grip strength value, J1 is the pressure uniformity index, ‖·‖2 is the Euclidean norm, and the user calculates the size of the objective function.

9. The robot hand adaptive control system based on multimodal sensing and reinforcement learning according to claim 1, characterized in that: The working steps of the calculation layer are as follows: S1, signal acquisition, real-time detection of multiple signals through the sensing layer, including pressure, ambient temperature, and humidity; S2, signal processing, temperature and humidity compensation of the sensor pressure detection signal through the signal processing layer; S3, data preprocessing, after filtering, using wavelet noise reduction or Kalman filtering to eliminate noise in sensor data; S4, thin plate spline interpolation, based on the sensor position and readings, builds a thin plate spline interpolation model; S5. Geometric constraint fusion, through the geometric constraint introduction module and the pressure field reconstruction module. The geometric constraint introduction module uses the geometric shape of the contact surface of the object as prior information to correct the pressure field. By introducing geometric constraints, it can ensure that the pressure field conforms to the physical laws at the edge or curved surface of the object. The pressure field reconstruction module converts discrete sensor data into a continuous pressure distribution field and integrates geometric constraint information. Through reconstruction, the mechanical properties of the contact area between the object and the robot hand can be accurately described. S6. Dynamic analysis of contact: The dynamic analysis of contact module monitors the mechanical interaction between the object and the robot hand in real time, including the slip detection module and the grip optimization module, to ensure the stability and safety of grasping.

Images