Machine physical object grasping adaptive learning system based on visual and tactile feedback

Abstract

The present application relates to the technical field of robot control, and especially relates to a robot object grasping adaptive learning system based on visual and tactile feedback. The robot object grasping adaptive learning system based on visual and tactile feedback comprises a data acquisition module, which is used for acquiring adaptive control data of a robot; a joint adjustment module, which is used for acquiring joint compensation of the robot according to the adaptive control data of the robot, and adjusting grasping movement of the robot to obtain final joint adjustment; a visual feedback module, which is used for acquiring rough grasping force estimation of visual information by using a visual feedback formula according to visual data of the robot; and a tactile feedback module, which is used for acquiring correction force of tactile feedback by using a tactile feedback formula according to tactile data of the robot. Through multi-modal information fusion and modular design, the present application realizes high-precision and adaptive grasping of the robot on thermochromic objects.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to an adaptive learning system for robot object grasping based on visual and tactile feedback. Background Technology

[0002] With the rapid development of industrial automation and intelligent manufacturing technologies, robots are increasingly widely used in manufacturing, medical assistance, and service industries. Among these applications, the high-precision and high-reliability grasping of objects of various shapes, materials, and temperature characteristics has become an important issue for improving industrial production efficiency and product quality. However, traditional robot grasping systems mostly rely on vision or force perception as a single sensing method, which has poor adaptability to environmental changes and is easily affected by factors such as lighting, visual obstruction, object surface condition, and temperature changes, resulting in inaccurate grasping force and a high failure rate. At the same time, for the grasping of heat-sensitive materials or thermochromic objects, existing technologies are still unable to monitor and compensate for the object surface temperature in real time and accurately, further increasing the errors and risks in the grasping process. Summary of the Invention

[0003] To overcome the shortcomings of lacking accurate grasping of thermochromic objects, this invention provides an adaptive learning system for robotic object grasping based on visual and tactile feedback.

[0004] The technical implementation scheme of the present invention is as follows: an adaptive learning system for robot object grasping based on visual and tactile feedback, comprising:

[0005] The data acquisition module is used to acquire adaptive control data for the robot;

[0006] The joint adjustment module is used to obtain the joint compensation amount of the robot based on the robot's adaptive control data, and to adjust the robot's grasping motion to obtain the final joint adjustment degree.

[0007] The visual feedback module is used to obtain a rough estimate of the grasping force based on the robot's visual data using a visual feedback formula.

[0008] The tactile feedback module is used to obtain the corrective force of tactile feedback based on the robot's tactile data using a tactile feedback formula;

[0009] The temperature gain module is used to obtain the temperature influence factor based on the surface temperature and the calibration reference temperature of the thermochromic object using the temperature gain function.

[0010] The grasping adjustment module is used to obtain the robot's grasping force using a grasping force model formula based on the coarse grasping force estimate of the visual information, the correction force of the tactile feedback, and the temperature influence factor.

[0011] The final grasping module grasps the thermochromic object based on the final joint adjustment and the robot's grasping force.

[0012] Preferably, the data acquisition module is used to acquire adaptive control data of the robot, including: acquiring adaptive control data of the robot, wherein the adaptive control data of the robot includes visual data, tactile data and joint-related data of the robot, wherein the joint-related data of the robot includes joint angle control data of the robot, thermal sensitivity coefficient of each joint and reference temperature of each joint in a calibration environment.

[0013] Preferably, the joint adjustment module is used to obtain the robot's joint compensation amount based on the robot's adaptive control data, and adjust the robot's grasping motion to obtain the final joint adjustment degree, including: obtaining the surface temperature and calibration reference temperature of the grasped thermochromic object based on the robot's visual data; obtaining the robot's joint compensation amount using a thermal compensation formula based on the surface temperature of the thermochromic object and the robot's joint-related data; and adjusting the robot's grasping motion based on the robot's joint compensation amount to obtain the final joint adjustment degree, wherein the thermal compensation formula is:

[0014]

[0015] In the formula, Δθ i This represents the angle compensation amount for the i-th joint of the robot. Ti is the thermal sensitivity coefficient of the i-th joint of the robot; T0 is the reference temperature of the robot joint under the calibration environment; Ti is the thermal sensitivity coefficient of the i-th joint of the robot. obj The surface temperature of the thermochromic object.

[0016] Preferably, the step of obtaining the surface temperature and calibration reference temperature of the thermochromic object to be grasped based on the robot's visual data includes: acquiring an original image of the target thermochromic object in the grasping area using a high-resolution RGB camera or multispectral camera mounted on the robot; performing image enhancement and correction on the original image to obtain a pre-processed image; using a pre-trained deep learning network on the pre-processed image to obtain the surface temperature of the thermochromic object; and retrieving the calibration reference temperature of the object from internal storage or a cloud database by identifying a QR code, RFID tag, or externally visible identifier carried by the object in the image.

[0017] Preferably, the visual feedback module is used to obtain a rough estimate of the grasping force of the visual information using a visual feedback formula based on the robot's visual data. This includes: obtaining image data of the target object using the robot's visual data, and processing the image data to obtain visual information data, wherein the visual information data includes the object's feature dimensions, surface friction coefficient, and material stiffness; and obtaining a rough estimate of the grasping force of the visual information using a visual feedback formula based on the visual information data, wherein the visual feedback formula is:

[0018]

[0019] In the formula, F V D is a rough estimate of the ability to grasp visual information. obj μ is the characteristic dimension of the object. vis k is the surface friction coefficient of the object. vis K represents the material stiffness of the object. size K fric K stiff These are the weighting coefficients for characteristic size, surface friction coefficient, and material stiffness, respectively; ε is the adjustment parameter; G(k vis ) is the stiffness influence function for obtaining the material stiffness influence factor.

[0020] Preferably, the stiffness data acquisition unit is used to acquire stiffness-related data of an object, including: acquiring stiffness-related data of the object, wherein the stiffness-related data includes a first stiffness threshold, a second stiffness threshold, and a reference value of the maximum indentation stiffness that the material can withstand without damage; the first stiffness threshold is a lower limit of stiffness that can ignore deformation differences; the second stiffness threshold is an upper limit that requires caution when entering the fragile high stiffness range; and a material stiffness influence factor is obtained by using a stiffness influence function based on the stiffness-related data.

[0021] Preferably, the stiffness factor acquisition unit obtains the material stiffness influence factor based on the stiffness-related data using a stiffness influence function, including: wherein the stiffness influence function is:

[0022]

[0023] In the formula, G(k) vis The stiffness influence function is used to obtain the material stiffness influence factor; k thresh1 k thresh2 These are the first stiffness threshold and the second stiffness threshold, respectively; k vis k represents the material stiffness of the object. max_safe ρ is the reference value for maximum indentation stiffness; ρ and δ are adjustment parameter values.

[0024] Preferably, the tactile feedback module is used to obtain a corrective force for tactile feedback based on the robot's tactile data using a tactile feedback formula, including: obtaining the robot's initial contact force and sliding tendency measure based on the robot's tactile data, and obtaining the corrective force for tactile feedback using a tactile feedback formula, wherein the tactile feedback formula is:

[0025] F T =K slip *s slip +K init *f init ;

[0026] In the formula, F T For the corrective force of tactile feedback; s slip For measuring the tendency to slide; f init K represents the initial contact force. slip K init These are the weighting coefficients for the sliding tendency measure and the initial contact force, respectively.

[0027] Preferably, the temperature gain module is used to obtain a temperature influence factor using a temperature gain function based on the surface temperature of the thermochromic object and its calibration reference temperature, including: wherein the temperature gain function is:

[0028] g(T obj )=1+γ|T obj ―T ref |;

[0029] In the formula, g(T) obj T is the temperature gain function used to obtain the temperature influence factor; obj T is the surface temperature of a thermochromic object. ref γ is the calibration reference temperature for thermochromic objects; γ is the temperature gain sensitivity coefficient.

[0030] Preferably, the grasping adjustment module is used to obtain the robot's grasping force using a grasping force model formula based on the coarse grasping force estimate from the visual information, the corrective force from the tactile feedback, and the temperature influence factor, including: wherein the grasping force model formula is:

[0031] F = g(T) obj )*[α*F V +β*F T ], F min ≤F≤F max ;

[0032] In the formula, F represents the robot's grasping force; g(T) obj F is the temperature gain function for obtaining the temperature influence factor; V F is a rough estimate of the ability to grasp visual information.T For the corrective force of tactile feedback; F min F represents the lower limit of the robot's grasping force. max α represents the upper limit of the robot's grasping force; α and β are the fusion weight coefficients of visual and tactile feedback, and α+β=1.

[0033] The beneficial effects are:

[0034] 1. This invention acquires the robot's visual data, tactile data, and joint-related data simultaneously through a data acquisition module. Based on visual feedback formulas and tactile feedback formulas, it estimates the rough grasping force and tactile correction force, respectively. Combined with the temperature influence factor output by the temperature gain module, it forms the input to the grasping force model formula, realizing the organic integration of vision, touch, and temperature. This significantly reduces the force estimation deviation caused by relying on only a single perception method, thereby improving the success rate and safety of robot grasping.

[0035] 2. This invention uses the joint compensation amount calculated by introducing a thermal compensation formula to correct the robot's grasping motion in real time, thereby obtaining a more accurate final joint adjustment degree. This effectively eliminates the mechanical error caused by changes in the working environment temperature of the joints, and helps to maintain the stability and accuracy of the grasping motion trajectory.

[0036] 3. The grasping force model formula is used to combine visual prior estimation with tactile posterior correction. At the same time, a temperature gain factor is introduced to further ensure the robustness of the grasping process under different temperature conditions. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the adaptive learning system for robot object grasping according to the present invention;

[0038] Figure 2 This is a block diagram of the visual feedback module of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Adaptive learning systems for robotic object grasping based on visual and tactile feedback, such as Figure 1 and Figure 2 As shown, it includes:

[0041] The data acquisition module is used to acquire adaptive control data for the robot;

[0042] The adaptive control data of the robot is acquired. The adaptive control data of the robot includes the robot's visual data, tactile data and joint-related data. The joint-related data of the robot includes the robot's joint angle control data, the thermal sensitivity coefficient of each joint and the reference temperature of each joint in the calibration environment.

[0043] It should be further explained that the data acquisition module uses the high-resolution RGB or multispectral camera mounted on the robot to collect image sequences of the environment in which the grasped object is located in real time. The vision sensor uses its built-in lens distortion correction algorithm to preprocess the original image to obtain a corrected real scene image. After image preprocessing, the image data is converted into visual information data containing the object's pose (position and orientation) in three-dimensional space, the object's feature size, the object's surface friction coefficient, and the object's material-related features (such as material color, texture, and shape features). The visual information data is used by a deep learning model to detect the surface temperature feature area of ​​the thermochromic object and output candidate temperature values.

[0044] Tactile sensors (hereinafter referred to as "tactile sensors"), which are force / torque sensors and pressure sensors installed at the fingertips or joints of the robot's gripper, collect tactile information in real time when the robot comes into contact with an object; the tactile sensors record the initial contact force when they first come into contact with the object; at the same time, they continuously monitor whether the object has a tendency to slide during the grasping process and output a measure of the sliding tendency; the tactile signals collected by the above sensors are filtered and normalized to generate tactile data.

[0045] Read the real-time control data of each joint of the robot to obtain joint angle control data: the current expected control angle value of each joint or the actual unprocessed angle value read; the thermal sensitivity coefficient of each joint: obtained through offline calibration experiments or factory calibration parameters. Each joint has a corresponding thermal sensitivity coefficient, which represents the sensitivity of the joint to the angle deviation caused by temperature changes; the reference temperature of each joint in the calibration environment. The joint temperature measured in the standard calibration environment (such as room temperature 20℃) is used to compare with the surface temperature of the grasped object in the thermal compensation formula, and then calculate the thermal compensation amount.

[0046] The joint adjustment module is used to obtain the joint compensation amount of the robot based on the robot's adaptive control data, and to adjust the robot's grasping motion to obtain the final joint adjustment degree.

[0047] Based on the robot's visual data, the surface temperature and calibration reference temperature of the thermochromic object being grasped are obtained. Based on the surface temperature of the thermochromic object and the robot's joint-related data, a thermal compensation formula is used to obtain the robot's joint compensation amount. The robot's grasping motion is then adjusted based on this joint compensation amount to obtain the final joint adjustment degree. The thermal compensation formula is as follows:

[0048]

[0049] In the formula, Δθ i This represents the angle compensation amount for the i-th joint of the robot. Ti is the thermal sensitivity coefficient of the i-th joint of the robot; T0 is the reference temperature of the robot joint under the calibration environment; Ti is the thermal sensitivity coefficient of the i-th joint of the robot. obj The surface temperature of the thermochromic object.

[0050] It should be further explained that under extreme temperatures (especially low temperatures), the robot's robotic arm joints and linkages will experience thermal contraction. When grasping thermochromic objects, the object's temperature is not necessarily the same as the surrounding air temperature: if the object has just been taken out of a low-temperature environment (such as a refrigerator or cold storage), its surface temperature is much lower than the surrounding room temperature; if the object has just been heated (such as in an oven or on a heating plate), its surface temperature is higher than room temperature. Due to heat conduction between the object and the robot's end effector, when the robotic finger comes into contact with the object, the robot's end effector (such as the gripper or end flange) will experience heat transfer in a short time due to the object's temperature, causing a momentary change in the local temperature of the joints, resulting in local thermal expansion or contraction. The joint compensation amount of the robot is obtained through a thermal compensation formula, and the final joint adjustment degree is obtained based on the joint compensation amount. When the robot performs the grasping action, the joints will move according to the corrected final joint adjustment degree, thereby compensating for the thermal deformation error caused by the high temperature of the thermochromic object and ensuring the accuracy of the grasping trajectory.

[0051] The robot acquires original images of the target thermochromic object within the grasping area using a high-resolution RGB or multispectral camera. The original images are then enhanced and corrected to obtain pre-processed images. A pre-trained deep learning network is used on the pre-processed images to obtain the surface temperature of the thermochromic object. The calibration reference temperature of the object is retrieved from internal storage or a cloud database by identifying QR codes, RFID tags, or externally visible markings carried by the object in the image.

[0052] It should be further explained that when the robot is looking down at the grasping area, its end effector or the platform above it is pre-installed with a high-resolution RGB camera or multispectral camera (hereinafter collectively referred to as a vision sensor). During the grasping preparation stage, the vision sensor automatically focuses and exposes to acquire the raw image data of the area to be grasped. The raw image is an unprocessed pixel-level color or multispectral image sequence, which contains the color and spectral characteristics of thermochromic objects under different spectral bands (visible light, near-infrared, and infrared). The image is then preprocessed with color correction, geometric correction, and noise reduction and enhancement, and input into a pre-trained deep learning network. The pre-trained deep learning network is a multi-channel convolutional neural network (referred to as a temperature estimation network). This network has been trained offline on a large dataset of thermochromic object images labeled with real temperature values. It can map the input preprocessed image into a two-dimensional thermogram or scalar value of the object's surface temperature, and output the overall surface temperature of the thermochromic object through a fully connected layer. The calibration reference temperature of the object is retrieved from internal storage or a cloud database by identifying the QR code, RFID tag, or external visible mark carried by the object in the image.

[0053] The visual feedback module is used to obtain a rough estimate of the grasping force based on the robot's visual data using a visual feedback formula.

[0054] The robot obtains image data of the target object using its visual data, and processes the image data to obtain visual information data, which includes the object's feature dimensions, surface friction coefficient, and material stiffness. Based on this visual information data, a coarse estimate of the grasping force is obtained using a visual feedback formula, where the visual feedback formula is:

[0055]

[0056] In the formula, F V D is a rough estimate of the ability to grasp visual information. obj μ is the characteristic dimension of the object. vis k is the surface friction coefficient of the object. vis K represents the material stiffness of the object. size K fric K stiff These are the weighting coefficients for characteristic size, surface friction coefficient, and material stiffness, respectively; ε is the adjustment parameter; G(k vis ) is the stiffness influence function for obtaining the material stiffness influence factor.

[0057] It needs to be further explained that D objTo determine the feature size of the object, in the preprocessed image, a deep learning object detection network (such as YOLO or Mask-R-CNN) trained offline is invoked to locate the bounding box or segmentation mask of the target object in the image. Based on the camera calibration parameters, the bounding box or segmentation result in the image coordinate system is mapped to the actual three-dimensional space to obtain the feature size information of the object.

[0058] μ vis To determine the surface friction coefficient of an object, the local texture features of the preprocessed image (such as gray-level co-occurrence matrix statistical features or local binary pattern features) and the color distribution of the object's surface are used to predict the surface friction coefficient through a pre-trained regression model. The extracted visual features are directly input into the regression model to obtain the surface friction coefficient of the target object.

[0059] k vis To determine the material stiffness of an object, features of the object's gloss, texture details, and color uniformity are extracted from the pre-processed image. These visual features are then input into a pre-trained classification or regression network, which outputs a corresponding material stiffness.

[0060] Acquire stiffness-related data of the object, which includes a first stiffness threshold, a second stiffness threshold, and a reference value for the maximum indentation stiffness that the material can withstand without damage. The first stiffness threshold is the lower limit of stiffness at which deformation differences can be ignored; the second stiffness threshold is the upper limit at which caution is required before entering the fragile high stiffness range. Based on the stiffness-related data, use the stiffness influence function to obtain the material stiffness influence factor.

[0061] It should be further explained that after identifying the object to be grasped, the object identifier is obtained through visual recognition or RFID / barcode retrieval. The identifier is then transmitted to a local or cloud database query interface to query the stiffness-related data of the corresponding object. The database pre-stores the lower limit of stiffness, upper limit of stiffness, and maximum indentation reference value of common materials under different working conditions.

[0062] The stiffness influence function is:

[0063]

[0064] In the formula, G(k) vis The stiffness influence function is used to obtain the material stiffness influence factor; k thresh1 k thresh2 These are the first stiffness threshold and the second stiffness threshold, respectively; k vis k represents the material stiffness of the object. max_safe ρ is the reference value for maximum indentation stiffness; ρ and δ are adjustment parameter values.

[0065] It should be further explained that in the low stiffness range, the deformation of the material under external force is negligible and there is no risk of breakage. Therefore, the stiffness influence factor is set to the minimum value. In the intermediate stiffness range, when the stiffness of the object is between the first and second stiffness thresholds, it indicates that the material begins to show significant elastic deformation but has not reached a fragile state. At this time, the influence of stiffness on the gripping force gradually increases and needs to be dynamically adjusted. In the high stiffness range, the material exhibits high stiffness or fragile characteristics. If the gripping force is increased further, the object will break or produce permanent indentations. Therefore, the gripping force needs to be reduced.

[0066] The tactile feedback module is used to obtain the corrective force of tactile feedback based on the robot's tactile data using a tactile feedback formula;

[0067] Based on the robot's tactile data, the initial contact force and sliding tendency measure of the robot are obtained, and the tactile feedback correction force is obtained using the tactile feedback formula, where the tactile feedback formula is:

[0068] F T =K slip *s slip +K init *f init ;

[0069] In the formula, F T For the corrective force of tactile feedback; s slip For measuring the tendency to slide; f init K represents the initial contact force. slip K init These are the weighting coefficients for the sliding tendency measure and the initial contact force, respectively.

[0070] It needs to be further explained that f init Initial contact force is defined as the instantaneous normal force or resultant force measured when the robot's gripper fingertips first come into contact with the object's surface. This value reflects the magnitude of the force exerted by the robot when it comes into contact with the object at the beginning of the grasping process.

[0071] s slip The sliding tendency measure is defined as a dimensionless index calculated by a sliding detection algorithm when a small relative displacement occurs on the surface of an object during contact, or when a tactile sensor detects a micro-disturbance signal indicating a tendency to slide. Specifically, the sliding tendency measure is obtained in the following two ways: Torque difference method: When the gripper grasps the object, if the sensor detects a sudden increase in shear force or torque, it indicates that the object has a sliding tendency. The rate of change of shear force is captured by sampling and compared with a preset threshold to obtain the sliding tendency measure; Micro-motion detection method: A micro-vibration sensing diaphragm is integrated into the tactile sensor. When the object slides slightly, the diaphragm vibrates slightly. The amplitude of the micro-motion signal is extracted by spectrum analysis and normalized to obtain the sliding tendency measure.

[0072] K slip K init The weighting coefficients for sliding tendency and initial contact force are respectively used. Through extensive experimental data collection on gripping tests of objects of different materials and shapes, K is determined using least-squares fitting or machine learning optimization algorithms (such as linear regression and Bayesian optimization). slip K init The optimal range of values.

[0073] The temperature gain module is used to obtain the temperature influence factor based on the surface temperature and the calibration reference temperature of the thermochromic object using the temperature gain function.

[0074] The temperature gain function is:

[0075] g(T obj )=1+γ|T obj ―T ref |;

[0076] In the formula, g(T) obj T is the temperature gain function used to obtain the temperature influence factor; obj T is the surface temperature of a thermochromic object. ref γ is the calibration reference temperature for thermochromic objects; γ is the temperature gain sensitivity coefficient.

[0077] It should be further explained that γ is the temperature gain sensitivity coefficient. In the offline calibration stage, for a series of thermochromic object samples (such as thermochromic materials with different formulations), the relationship between the success rate of grasping and the temperature difference is measured by a temperature test bench, and a suitable γ value is obtained by fitting. In the online stage, combined with machine learning algorithms (such as incremental minimum mean square or online gradient descent), γ is fine-tuned according to the actual grasping success rate and object temperature difference data, so that it can adapt to the differences in mechanical properties of different thermochromic materials.

[0078] The grasping adjustment module is used to obtain the robot's grasping force using a grasping force model formula based on the coarse grasping force estimate of the visual information, the correction force of the tactile feedback, and the temperature influence factor.

[0079] The formula for the gripping force model is:

[0080] F = g(T) obj )*[α*F V +β*F T ], F min ≤F≤F max ;

[0081] In the formula, F represents the robot's grasping force; g(T) obj F is the temperature gain function for obtaining the temperature influence factor;V F is a rough estimate of the ability to grasp visual information. T For the corrective force of tactile feedback; F min F represents the lower limit of the robot's grasping force. max α represents the upper limit of the robot's grasping force; α and β are the fusion weight coefficients of visual and tactile feedback, and α+β=1.

[0082] The final grasping module grasps the thermochromic object based on the final joint adjustment and the robot's grasping force.

[0083] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A machine physical object grasping adaptive learning system based on visual and haptic feedback, characterized in that, The method comprises the following steps: a data acquisition module for acquiring adaptive control data of the robot; a joint adjustment module for obtaining joint compensation of the robot according to the adaptive control data of the robot, and adjusting the grasping movement of the robot to obtain a final joint adjustment degree; a visual feedback module for obtaining a rough grasping force estimation value of visual information according to visual data of the robot using a visual feedback formula; a tactile feedback module for obtaining a correction force of tactile feedback according to tactile data of the robot using a tactile feedback formula; a temperature gain module for obtaining a temperature influence factor using a temperature gain function according to the surface temperature of the thermochromic object and the calibration reference temperature of the thermochromic object; a grasping adjustment module for obtaining the grasping force of the robot using a grasping force model formula according to the rough grasping force estimation value of the visual information, the correction force of the tactile feedback, and the temperature influence factor; a final grasping module for grasping the thermochromic object according to the final joint adjustment degree and the grasping force of the robot; The joint adjustment module is used for obtaining joint compensation of the robot according to the adaptive control data of the robot, and adjusting the grasping movement of the robot to obtain a final joint adjustment degree, comprising: obtaining the surface temperature of the grasped thermochromic object and the calibration reference temperature of the thermochromic object according to the visual data of the robot, obtaining the joint compensation of the robot using a thermal compensation formula according to the surface temperature of the thermochromic object and the joint-related data of the robot, adjusting the grasping movement of the robot according to the joint compensation of the robot to obtain a final joint adjustment degree, wherein the thermal compensation formula is: ； In the formula, is an angle compensation amount of a first joint of the robot; is an angle compensation amount of a first joint of the robot; is a thermal sensitivity coefficient of a first joint of the robot; is a thermal sensitivity coefficient of a first joint of the robot; is a reference temperature of a joint of the robot in a calibration environment; is a surface temperature of a thermochromic object; The temperature gain module is used for obtaining a temperature influence factor using a temperature gain function according to the surface temperature of the thermochromic object and the calibration reference temperature of the thermochromic object, comprising: wherein the temperature gain function is: ； wherein temperature gain function for the temperature influencing factor; surface temperature of the thermochromic object; calibrated reference temperature of the thermochromic object; temperature gain sensitivity coefficient; The grasping adjustment module is used for obtaining the grasping force of the robot using a grasping force model formula according to the rough grasping force estimation value of the visual information, the correction force of the tactile feedback, and the temperature influence factor, comprising: wherein the grasping force model formula is: ； wherein, is the robot grasping force; is a temperature gain function to obtain a temperature influence factor; is a coarse grasping force estimate of visual information; is a correction force of haptic feedback; is a lower limit of the robot grasping force; is an upper limit of the robot grasping force; is a fusion weight coefficient of visual and haptic feedback, and . 2.The visual and haptic feedback based robotic object grasping adaptive learning system of claim 1, wherein, The data acquisition module is used for acquiring adaptive control data of the robot, comprising: acquiring adaptive control data of the robot, wherein the adaptive control data of the robot includes visual data, tactile data, and joint-related data of the robot, and the joint-related data of the robot includes joint angle control data of the robot, thermal sensitivity coefficients of each joint, and reference temperatures of each joint in a calibration environment. 3.The visual and tactile feedback based robotic object grasping adaptive learning system of claim 1, wherein, The method comprises the following steps: acquiring the surface temperature of the grabbed thermochromic object and the calibration reference temperature of the thermochromic object according to the visual data of the robot, including: acquiring the original image of the target thermochromic object in the area to be grabbed by the high-resolution RGB camera or the multispectral camera carried by the robot, performing image enhancement and correction on the original image to obtain the preprocessed image, and using the pre-trained deep learning network on the preprocessed image to obtain the surface temperature of the thermochromic object; and acquiring the calibration reference temperature of the object from the internal storage or the cloud database by identifying the two-dimensional code, RFID tag or external visible mark carried by the object in the image.

4. The visual and haptic feedback based robotic object grasping adaptive learning system of claim 1, wherein, The visual feedback module is configured to use a visual feedback formula to obtain a rough grasp force estimation value of visual information according to the visual data of the robot, including: obtaining image data of the target object through the visual data of the robot, and obtaining visual information data by processing the image data, wherein the visual information data includes characteristic dimensions of the object, a surface friction coefficient of the object and a material stiffness of the object; and using the visual feedback formula to obtain the rough grasp force estimation value of the visual information according to the visual information data, wherein the visual feedback formula is as follows: ； wherein, is a coarse graspability estimate of the visual information; is a characteristic dimension of the object; is a surface friction coefficient of the object; is a material stiffness of the object; are weight coefficients of the characteristic dimension, the surface friction coefficient and the material stiffness, respectively; is an adjustment parameter; is a stiffness influence function for obtaining the material stiffness influence factor.

5. The visual and haptic feedback based robotic object grasping adaptive learning system according to claim 4, wherein, The stiffness data acquisition unit is configured to acquire stiffness-related data of the object, including: acquiring the stiffness-related data of the object, wherein the stiffness-related data includes a first stiffness threshold, a second stiffness threshold and a maximum indentation stiffness reference value that the material can withstand without being damaged, the first stiffness threshold is a lower limit of stiffness at which the difference in deformation can be ignored, and the second stiffness threshold is an upper limit at which the material enters a fragile high-stiffness state and needs to be cautious; and using a stiffness influence function to obtain a material stiffness influence factor from the stiffness factor acquisition unit according to the stiffness-related data.

6. The visual and haptic feedback based robotic object grasping adaptive learning system according to claim 5, wherein, The stiffness factor acquisition unit is configured to use a stiffness influence function to obtain a material stiffness influence factor according to the stiffness-related data, including: wherein the stiffness influence function is as follows: ； In the formula, a stiffness influence function for obtaining a material stiffness influence factor; respectively a first stiffness threshold and a second stiffness threshold; a material stiffness of the object; a maximum denting stiffness reference value; an adjustment parameter value.

7. The visual and haptic feedback based adaptive learning system for robot object grasping according to claim 1, wherein, The haptic feedback module is configured to use a haptic feedback formula to obtain a correction force of haptic feedback according to the haptic data of the robot, including: obtaining an initial contact force and a sliding tendency measure of the robot according to the haptic data of the robot, and using the haptic feedback formula to obtain the correction force of haptic feedback, wherein the haptic feedback formula is as follows: ； wherein is a correction force for haptic feedback; is a measure of the tendency to slide; is an initial contact force; are weight coefficients for the measure of the tendency to slide and the initial contact force, respectively.

Images