Visual-tactile sensor and tactile reconstruction method

Abstract

The application relates to the technical field of robot perception, and provides a visual-tactile sensor and a tactile reconstruction method.The visual-tactile sensor comprises a partitioned elastic contact layer, an illumination control module and an image acquisition module; the partitioned elastic contact layer comprises a high-sensitivity zone, a transition zone and a high-load-bearing zone; the elastic modulus of the high-sensitivity zone, the transition zone and the high-load-bearing zone increases in turn; the illumination control module is used for providing illumination light, and the image acquisition module is used for acquiring a deformation image of the partitioned elastic contact layer under the illumination light.The application divides the elastic contact layer into the high-sensitivity zone, the transition zone and the high-load-bearing zone, and adopts differentiated design of gradient elastic modulus for the zones, so that different zones can accurately perceive and reflect different degrees of contact force through deformation, thereby giving consideration to sensitivity and range, showing high sensitivity in different stress zones, significantly reducing the probability of deformation saturation, and ensuring the accuracy of the deformation image.

Description

Technical Field

[0001] This application relates to the field of robot perception technology, specifically to a visual-tactile sensor and a tactile reconstruction method. Background Technology

[0002] During robot grasping and manipulation, the end effector needs to accurately perceive the contact state with the object being grasped in order to achieve stable grasping and precise manipulation. Existing robot systems mainly rely on external vision sensors to obtain the pose information of the object, but after contact occurs, pure visual information is difficult to reflect the distribution of contact force, changes in contact shape, and sliding trends.

[0003] To address these issues, researchers have proposed various tactile sensor solutions, such as resistive, capacitive, and piezoelectric tactile sensors. These sensors are typically arranged in arrays and can measure localized pressure changes, but they suffer from complex wiring, limited spatial resolution, difficulty in simultaneously sensing normal and shear forces, and are susceptible to environmental interference and material aging during long-term use.

[0004] In recent years, vision-based tactile sensing technologies have emerged. These technologies typically involve placing an imaging device under an elastic medium and indirectly acquiring tactile information by capturing images of the deformation of the elastic medium under contact.

[0005] However, existing solutions mostly use a uniform elastic medium. Although it can achieve high-resolution tactile perception in experimental environments, in actual robot grasping applications, the sensitivity and range of the uniform elastic medium are mutually constrained. In different force ranges, it is easy to have insufficient sensitivity or deformation saturation, which makes it difficult to guarantee the accuracy of deformation images. Summary of the Invention

[0006] This application provides a visual-tactile sensor and a tactile reconstruction method to solve the technical problem that in actual robot grasping applications, the sensitivity and range of a uniform elastic medium are mutually constrained, and insufficient sensitivity or deformation saturation is likely to occur in different force ranges, making it difficult to guarantee the accuracy of deformation images.

[0007] In a first aspect, embodiments of this application provide a visual-tactile sensor, including: a partitioned elastic contact layer, a light control module, and an image acquisition module; The partitioned elastic contact layer includes a high-sensitivity zone, a transition zone, and a high-load-bearing zone; The elastic modulus of the high-sensitivity region, the transition region, and the high-load-bearing region increases sequentially. The illumination control module and the image acquisition module are located on the side of the partitioned elastic contact layer away from the contact surface. The illumination control module is used to provide illumination light, and the image acquisition module is used to acquire deformation images of the partitioned elastic contact layer under the illumination light.

[0008] In one embodiment, it further includes: an encoded visual marker layer; The coded visual marker layer is attached to the surface of the partitioned elastic contact layer on the side away from the contact surface; The encoded visual marker layer includes a first morphological region covered by a first morphological marker, a second morphological region covered by a second morphological marker, and a third morphological region covered by a third morphological marker. The first morphological region is aligned with the high-sensitivity region to sense the deformation characteristics of the high-sensitivity region; The second morphological region is aligned with the high load-bearing region and is used to sense the deformation characteristics of the high load-bearing region; The third morphological region is aligned with the transition region and is used to sense the deformation characteristics of the transition region.

[0009] In one embodiment, it further includes: a buffer support layer; The buffer support layer is attached to the surface of the coded visual marker layer on the side away from the partitioned elastic contact layer; The illumination control module and the image acquisition module are embedded within the buffer support layer.

[0010] In one embodiment, it further includes: a light-shielding contact layer; The light-shielding contact layer is attached to the contact surface of the partitioned elastic contact layer; The side of the light-shielding contact layer away from the contact surface of the partitioned elastic contact layer has a micro-textured structure and is coated with elastic reflective powder.

[0011] Secondly, embodiments of this application provide a tactile reconstruction method based on the first aspect, including: Acquire continuous multi-frame deformation images captured by the visual-tactile sensor; Based on the continuous multi-frame deformation images, tactile features are extracted; Based on the aforementioned tactile features, tactile information is reconstructed.

[0012] In one embodiment, extracting tactile features based on the continuous multi-frame deformation images includes: For any two consecutive deformed images in the aforementioned multi-frame deformed images, identify the morphological marker information in the two consecutive deformed images; the morphological marker information includes the ID, initial coordinates, and morphological features of the morphological marker. Based on the morphological marker information, the morphological region to which the morphological marker belongs is determined; For the same shape marker with the same ID in two consecutive deformed images, based on the current coordinates of the shape marker and the calibration parameters of the image acquisition module, the pixel displacement of the shape marker on the two consecutive deformed images is converted into physical displacement to obtain a global tactile deformation displacement field. Based on the encoding algorithm adapted to the morphological region, the morphological marking information and the global tactile deformation displacement field are partitioned and encoded to obtain the tactile features corresponding to each region in the partitioned elastic contact layer.

[0013] In one embodiment, obtaining the global tactile deformation displacement field includes: When the displacement of the morphological markers is in a stable state in the continuous multi-frame deformation images, the current frame deformation image is used as the reference image for calculating the global tactile deformation displacement field with the next frame deformation image, and the reference image is dynamically updated.

[0014] In one embodiment, reconstructing the tactile information based on the tactile features includes: When the displacement of the morphological marker is stable in the continuous multi-frame deformation images, sample features matching the tactile features are obtained from a preset pattern library; the preset pattern library is established based on the mapping relationship between the sample features and their corresponding tactile deformation patterns. Based on the tactile deformation pattern corresponding to the sample feature, determine the current tactile deformation pattern of the tactile feature; Based on the tactile parameter reconstruction model that matches the current tactile deformation pattern, the tactile information is constrained and solved to obtain the reconstructed tactile information.

[0015] In one embodiment, the reconstruction of the tactile information includes: The pre-stored gripper structure deformation mapping model is invoked, and the pseudo displacement caused by the gripper structure deformation is obtained based on the current gripping force, current opening and closing angle and current coordinates of the shape marker. Based on the pseudo-displacement, the displacement field in the reconstructed tactile information is corrected.

[0016] In one embodiment, the reconstruction of the tactile information includes: Obtain multi-dimensional indicators for gripper stability assessment; If all the multi-dimensional indicators meet the preset conditions, then the gripper's grasping state is determined to be stable. If any dimension indicator does not meet the preset conditions, it is determined that the gripper's grasping state is unstable, and the gripper's adaptive action is triggered.

[0017] The visual-tactile sensor provided in this application includes a partitioned elastic contact layer, an illumination control module, and an image acquisition module. The partitioned elastic contact layer comprises a high-sensitivity zone, a transition zone, and a high-load-bearing zone, with the elastic modulus increasing sequentially from these zones. The illumination control module and the image acquisition module are located on the side of the partitioned elastic contact layer away from the contact surface. The illumination control module provides illumination, and the image acquisition module acquires deformation images of the partitioned elastic contact layer under illumination. In this application, by dividing the elastic contact layer into a high-sensitivity zone, a transition zone, and a high-load-bearing zone, and employing a differentiated design with gradient elastic modulus for each zone, different zones can accurately sense and reflect different degrees of contact force through deformation. This balances sensitivity and measurement range, exhibiting high sensitivity across different force ranges and significantly reducing the probability of deformation saturation, thus ensuring the accuracy of the deformation images. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of the visual-tactile sensor provided in the embodiments of this application; Figure 2 This is one of the flowcharts illustrating the tactile reconstruction method provided in the embodiments of this application; Figure 3 This is a second schematic flowchart of the tactile reconstruction method provided in the embodiments of this application; Figure 4 This is the third schematic flowchart of the tactile reconstruction method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the calibration of the gripper structure deformation mapping model provided in the embodiments of this application; Figure 6 This is the fourth flowchart illustrating the tactile reconstruction method provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Attached image description: 1-Partitioned elastic contact layer; 2-Light control module; 3-Image acquisition module; 31-Protective plate; 4-Encoded visual marker layer; 41-First morphological area; 42-Second morphological area; 43-Third morphological area; 5-Buffer support layer; 6-Light-shielding contact layer; 7-Sealed outer shell. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] It should be noted that in the description of the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, or integral connections; they can be mechanical connections or electrical connections; they can be direct connections or indirect connections through an intermediate medium; and they can be internal connections between two elements. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0023] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0024] Figure 1 This is a schematic diagram of the structure of the visual-tactile sensor provided in an embodiment of this application. (Refer to...) Figure 1 This application provides a visual-touch sensor, which may include: a partitioned elastic contact layer 1, a light control module 2, and an image acquisition module 3; The partitioned elastic contact layer 1 includes a high-sensitivity zone, a transition zone, and a high-load-bearing zone; The elastic modulus increases sequentially in the high-sensitivity zone, the transition zone, and the high-load-bearing zone. The illumination control module 2 and the image acquisition module 3 are located on the side of the partitioned elastic contact layer 1 away from the contact surface. The illumination control module 2 is used to provide illumination light, and the image acquisition module 3 is used to acquire deformation images of the partitioned elastic contact layer 1 under illumination light.

[0025] In this embodiment, the visual-tactile sensor can be mechanically fixedly connected to the robot's end effector and directly contact the object being grasped.

[0026] Among them, the partitioned elastic contact layer 1 uses a polydimethylsiloxane composite material with a gradient elastic modulus. The elastic modulus is adjusted by adding different proportions of reinforcing agents (such as fumed silica). Specifically: For the high-sensitivity area, i.e. Figure 1 The A region, located in the center and edge of the contact surface of the partitioned elastic contact layer 1, can have a relatively small elastic modulus and thickness. The specific values ​​can be set according to actual needs and are not limited here. In this embodiment, the elastic modulus can be set to be greater than 1 kPa and less than 5 kPa, and the thickness can be set to be greater than 0.8 mm and less than 1.2 mm. It is used to sense small contact forces less than or equal to 5 Newtons, such as the contact force when grasping thin plastic sheets, paper, etc. For high load-bearing areas, i.e. Figure 1 The B region, located at the edge of the contact surface of the partitioned elastic contact layer 1, can have a relatively large elastic modulus and thickness. The specific values ​​can be set according to actual needs and are not limited here. In this embodiment, the elastic modulus can be set to be greater than 50 kPa and less than 100 kPa, and the thickness can be set to be greater than 2 mm and less than 3 mm. It is used to sense larger contact forces greater than 5 Newtons and less than 50 Newtons, such as the contact force when a metal block is clamped, to prevent overall deformation saturation. For the transition zone, i.e. Figure 1 The elastic modulus and thickness of the material in region C, which is located between region A and region B, can be moderate. The specific values ​​can be set according to actual needs and are not limited here. In this embodiment, the elastic modulus can be set to be greater than 20 kPa and less than 30 kPa, and the thickness can be set to be greater than 1.5 mm and less than 2 mm, in order to alleviate the abrupt deformation between region A and region B and avoid stress concentration.

[0027] By designing different elastic moduli and thicknesses for different regions of the partitioned elastic contact layer 1, the sensitivity and load-bearing capacity of each region to contact deformation are differentiated, so that the tactile response is dominated by different regions in different contact force ranges.

[0028] It should be noted that the material of the partitioned elastic contact layer 1 can also be a silicone and polyurethane composite material, or a hollow elastomer, such as a micro-airbag array, to achieve a gradient elastic modulus. Furthermore, the partitioned elastic contact layer 1 can be simplified to two regions, namely a high-sensitivity region and a high-load-bearing region, or expanded to four regions, i.e., adding an ultra-sensitive region, and setting the elastic modulus of the material in this region to less than or equal to 1 kPa, for sensing even smaller contact forces.

[0029] Furthermore: The illumination control module 2 can use multiple LED array light sources with wavelengths matching the visible light wavelength. The power of each light source is adapted to the lighting requirements, and the brightness can be adjusted by pulse width modulation. In this embodiment, four LED array light sources can be used with a wavelength of 530 nanometers.

[0030] The image acquisition module 3 may include at least one camera. In this embodiment, two cameras may be used, and a protective plate 31 may be set on the outside of the camera. The focal length of the lens may be adapted to the imaging requirements. In addition, it may use a high-speed data transmission interface, the transmission rate may meet the requirements of real-time image transmission, and the data transmission delay may be controlled within a preset range. Furthermore, a single camera and a wide-angle lens with a field of view greater than or equal to 120 degrees may be used to replace the dual cameras, or a 3D camera, such as a structured light camera, may be used to acquire depth information simultaneously.

[0031] Furthermore, in this embodiment, the visual-tactile sensor can be integrated into parallel grippers, serial grippers, flexible grippers, or other end effectors to adapt to different robot grasping scenarios. It can also be modularly designed according to different types of end effectors, such as grippers and suction cups. The installation interface between the visual-tactile sensor and various types of end effectors is compatible with industry standards.

[0032] The visual-tactile sensor provided in this embodiment includes a partitioned elastic contact layer, an illumination control module, and an image acquisition module. The partitioned elastic contact layer includes a high-sensitivity zone, a transition zone, and a high-load-bearing zone, with the elastic modulus increasing sequentially from these zones. The illumination control module and the image acquisition module are located on the side of the partitioned elastic contact layer away from the contact surface. The illumination control module provides illumination, and the image acquisition module acquires deformation images of the partitioned elastic contact layer under illumination. In this embodiment, by dividing the elastic contact layer into a high-sensitivity zone, a transition zone, and a high-load-bearing zone, and employing a differentiated design with gradient elastic modulus for each zone, different zones can accurately sense and reflect different degrees of contact force through deformation. This balances sensitivity and measurement range, exhibiting high sensitivity across different force ranges and significantly reducing the probability of deformation saturation, thus ensuring the accuracy of the deformation images.

[0033] Reference Figure 1In one embodiment, the visual-tactile sensor may further include: an encoded visual marker layer 4; The coded visual marker layer 4 is attached to the surface of the partitioned elastic contact layer 1 on the side away from the contact surface; The encoding visual marker layer 4 includes a first morphological region 41 covered by a first morphological marker, a second morphological region 42 covered by a second morphological marker, and a third morphological region 43 covered by a third morphological marker. The first morphological region 41 is aligned with the high-sensitivity region and is used to sense the deformation characteristics of the high-sensitivity region; The second morphological region 42 is aligned with the high load-bearing region and is used to sense the deformation characteristics of the high load-bearing region. The third morphological region 43 is aligned with the transition region and is used to sense the deformation characteristics of the transition region.

[0034] The coded visual marker layer 4 can be obtained by bearing morphological markers on a transparent acrylic substrate with a thickness between 0.3 mm and 0.5 mm. Specifically: For the first morphological region 41 that matches the high-sensitivity region, the first morphological marker inside it can be designed as a two-dimensional coded micro-marker with error correction and identity encoding capabilities, i.e. Figure 1 The diameter of each circular marker, the encoding capacity of the high-sensitivity area, and the marker density can be set according to actual needs and are not limited here. In this embodiment, the diameter of each circular marker can be set to 50 micrometers to 100 micrometers, and the marker density in the area can be set to 300 to 500 per square centimeter. For the second morphological region 42, which matches the high load-bearing area, the internal second morphological markers can be designed as square dot matrix macro markers with stronger resistance to deformation interference, i.e. Figure 1 The square markers in the text can have their side length, high-capacity area coding capacity, and marker density set according to actual needs, and are not limited here. In this embodiment, the side length of each square marker can be set to 200 micrometers to 500 micrometers, and the marker density within the area can be set to 50 to 100 per square centimeter. For the third morphological region 43 that matches the transition region, the third morphological marker inside it can be designed as an elliptical gradient marker that adapts to the gradient transformation characteristics of the transition region, i.e. Figure 1 The elliptical markers in the diagram can have their major and minor axes, the encoding capacity of the transition area, and the marker density set according to actual needs, and are not limited here. In this embodiment, the major axis of each elliptical marker can be set to 200 micrometers, the minor axis can be set to 100 micrometers, and the marker density within the area can be set to 150 to 250 markers per square centimeter.

[0035] Furthermore, each morphological mark can be printed with high-temperature resistant and anti-aging fluorescent ink, with the fluorescence wavelength adapted to the light source wavelength of the illumination control module to improve imaging contrast. For example, fluorescent ink with a wavelength of 520 nm to 550 nm can be selected.

[0036] It should be noted that, for the shape of the marker, triangles, rhombuses, etc. can be used instead of circular or square markers; for the arrangement of the markers on the coded visual marker layer 4, i.e. the encoding method, one-dimensional barcodes or dot matrix encoding can also be used; for the fluorescence characteristics of the markers, infrared fluorescent markers can also be used, such as fluorescent markers with a wavelength of 850nm, which can be collected with an infrared camera and are suitable for strong light interference scenarios.

[0037] Furthermore, the settings for the illumination control module 2 and the image acquisition module 3 also need to be adapted to the coded visual marker layer 4. Specifically: For the illumination control module 2, its core objectives are to adapt to the imaging of morphological markers and eliminate reflective interference. The light source parameters can be flexibly selected according to the characteristics of the morphological markers. In addition, its light source can adopt an oblique layout with an angle of about 30 degrees with the coded visual marker layer 4 to avoid direct reflection. Each group of light sources is equipped with a light-focusing or light-diffusing component, such as a light-diffusing plate, to achieve uniform light distribution. The illumination intensity error in the morphological marker area must be less than or equal to 5%.

[0038] For the image acquisition module 3, its camera frame rate must be no less than the minimum frame rate required for tactile dynamic perception, and its resolution must meet the accuracy requirements for morphological mark recognition. It can be used with a microscopic teleconverter to improve the recognition capability of morphological marks. In addition, its optical axis and the coded visual mark layer 4 must maintain a preset angle, such as perpendicular, and the imaging range must cover the entire morphological mark area. The overlap rate of adjacent camera imaging must be greater than or equal to 10% to ensure that there are no visual blind spots.

[0039] It should be noted that for low-speed capture scenarios, the image acquisition module 3 can use a low frame rate camera to reduce costs, while for high-speed scenarios, the frame rate can be increased to a higher range.

[0040] In this embodiment, the coded visual marker layer uses a dual feature design of morphology and coding to differentiate the markers in the morphological regions that are adapted to different regions of the partitioned elastic contact layer. This enables differentiated perception of the deformation characteristics of different regions of the partitioned elastic contact layer. Furthermore, it can combine the marker ID to couple the marker identity information with the region semantic information, which helps in subsequent marker identity recognition and region semantic differentiation, and improves the accuracy of displacement extraction and anti-interference.

[0041] Reference Figure 1 In one embodiment, the visual-touch sensor may further include: a buffer support layer 5; The buffer support layer 5 is attached to the surface of the coded visual marker layer 4 on the side away from the partitioned elastic contact layer; The illumination control module 2 and the image acquisition module 3 are embedded inside the buffer support layer 5.

[0042] Among them, the buffer support layer 5 can be a hollow structure filled with high-hardness silicone, which allows the partitioned elastic contact layer 1 to produce controllable deformation under the action of contact force, while limiting excessive nonlinear deformation, such as wrinkles and tears; the overall structure remains rigid to ensure the flatness of the installation of the coded visual mark layer 4, and the installation error is controlled within a preset threshold, such as less than or equal to 0.1 mm.

[0043] This embodiment, by designing a buffer support layer, can balance the deformation freedom and structural stability of the partitioned elastic contact layer, and can also protect the lighting control module and the image acquisition module.

[0044] Reference Figure 1 In one embodiment, the visual-touch sensor may further include: a light-shielding contact layer 6; The light-shielding contact layer 6 is bonded to the contact surface of the partitioned elastic contact layer 1; The side of the light-shielding contact layer 6 away from the contact surface of the partitioned elastic contact layer 1 has a micro-textured structure and is coated with elastic reflective powder.

[0045] The microtexture structure can be set according to actual needs, and is not limited here. In this embodiment, a rhomboid anti-slip texture with a side length of 10 micrometers to 20 micrometers can be used.

[0046] In addition, the elastic reflective powder can be aluminum powder or titanium dioxide.

[0047] This embodiment, by setting a light-shielding contact layer, can, on the one hand, increase the friction coefficient between the sensor and the grasped object through the micro-texture structure, while reducing the impact of contact slippage on perception; on the other hand, it can block external light from entering through the reflective coating, providing a clean background for the camera inside the sensor.

[0048] Reference Figure 1 In one embodiment, the visual-touch sensor may further include: a sealed housing 7; The sealed housing 7 can be a rigid, wear-resistant, one-piece molded housing, such as aluminum alloy, with a surface treated for corrosion protection, such as anodizing, and a protection level that meets the requirements of industrial applications, such as a waterproof rating of IP65. Furthermore, it has reserved an image acquisition module 3 mounting slot and a light control module 2 mounting position inside to adapt to the installation requirements of the corresponding modules. The inner wall of the outer shell is provided with a light-absorbing structure, such as black light-absorbing velvet, to avoid stray light reflection interfering with imaging.

[0049] Furthermore, after the image acquisition module 3 and the illumination control module 2 are installed into the corresponding mounting slots and mounting positions of the sealed housing 7, high-hardness silicone is filled into the sealed housing to form a buffer support layer 5, so that the image acquisition module 3 and the illumination control module 2 can be smoothly installed into the sealed housing 7 and embedded inside the buffer support layer 5.

[0050] This embodiment provides an installation location for the image acquisition module's illumination and control module by setting a sealed outer shell, provides filling space for the buffer support layer, protects the sensor as a whole through material treatment, and prevents interference with the sensor's internal imaging by setting a light-absorbing structure.

[0051] Figure 2 This is one of the flowcharts illustrating the tactile reconstruction method provided in this application. (Refer to...) Figure 2 This application provides a tactile reconstruction method, which may include: Step 201: Acquire continuous multi-frame deformation images captured by the visual-tactile sensor; Step 202: Extract tactile features based on continuous multi-frame deformation images; Step 203: Reconstruct tactile information based on tactile features.

[0052] In step 201, during the grasping process of the robot end effector, the image acquisition module in the visual-tactile sensor will continuously acquire image sequences of the encoded visual marker layer according to a preset frame rate. These image sequences are continuous multi-frame deformation images that characterize the deformation characteristics of the partitioned elastic contact layer under contact force by the displacement changes of each morphological marker in the encoded visual marker layer under contact force.

[0053] During the process of the image acquisition module acquiring multiple consecutive frames of deformed images, the illumination control module can dynamically adjust the illumination intensity in sync with the acquisition frame rate.

[0054] Furthermore, after obtaining multiple consecutive frames of deformed images, they can be preprocessed, such as image denoising, brightness normalization, distortion correction, etc., which are not limited here.

[0055] In step 202, tactile features can be extracted from the preprocessed multi-frame deformation images. During this process, contact scene signals can also be fed back to the lighting control module so that the lighting control module can adjust the lighting intensity according to different contact scenes, such as reducing brightness in strong light environments and increasing brightness in weak light environments.

[0056] In step 203, based on tactile features, various important information that can be used to reconstruct tactile information is obtained, thereby realizing the reconstruction of tactile information based on this important information.

[0057] This embodiment acquires continuous multi-frame deformation images characterizing the deformation properties of the partitioned elastic contact layer under contact force. Since the partitioned elastic contact layer is divided into a high-sensitivity zone, a transition zone, and a high-load-bearing zone, and each zone adopts a differentiated design with gradient elastic modulus, different zones can accurately sense and reflect different degrees of contact force through deformation, thus balancing sensitivity and range. It exhibits high sensitivity in different force ranges and significantly reduces the probability of deformation saturation, ensuring the accuracy of continuous multi-frame deformation images. Tactile features are extracted from the continuous multi-frame deformation images, and important information that can be used to reconstruct tactile information is obtained based on these tactile features to reconstruct tactile information, thereby simultaneously improving the reconstruction accuracy.

[0058] Figure 3 This is the second schematic flowchart of the tactile reconstruction method provided in the embodiments of this application. (Refer to...) Figure 3 In one embodiment, step 202 may include: Step 301: For any two consecutive deformed images in a series of multiple deformed images, identify the morphological marker information in the two consecutive deformed images; The morphological labeling information includes the morphological label's ID, initial coordinates, and morphological features; Step 302: Based on the morphological marker information, determine the morphological region to which the morphological marker belongs; Step 303: For the shape marker with the same ID in the two consecutive deformed images, based on the current coordinates of the shape marker and the calibration parameters of the image acquisition module, convert the pixel displacement of the shape marker on the two consecutive deformed images into physical displacement to obtain the global tactile deformation displacement field. Step 304: Based on the encoding algorithm adapted to the morphological region, the morphological marking information and the global tactile deformation displacement field are partitioned and encoded to obtain the tactile features corresponding to each region in the partitioned elastic contact layer.

[0059] In steps 301 to 302, the two consecutive deformation images can be the current frame deformation image and the previous frame deformation image. Then, the ID, initial coordinates and morphological features of each morphological marker in the two deformation images are identified. Based on this information, the morphological region to which each morphological marker belongs can be identified, and then the region corresponding to the partitioned elastic contact layer, namely the high-sensitivity region, the transition region and the high-load region, can be identified, thus realizing the dual recognition of marker identity and region semantics.

[0060] In step 303, for the shape marker with the same ID in the two deformed images, a sub-pixel-level positioning algorithm can be used to locate its center coordinates as the current coordinates. The positioning accuracy needs to meet the displacement measurement requirements, for example, less than or equal to 0.01 pixels. Then, combined with camera calibration parameters, such as pixel equivalent, the pixel displacement of the shape marker on the two deformed images is converted into physical displacement to obtain the global tactile deformation displacement field. Here, pixel equivalent refers to the actual length of each pixel in the physical world, which can be set according to actual needs. It is not limited here. In this embodiment, the pixel equivalent can be set to 5 micrometers per pixel.

[0061] In step 304, for different morphological regions, a corresponding encoding algorithm is used to encode the morphological marker information and tactile deformation displacement field in the corresponding region, so as to obtain the tactile features corresponding to the high-sensitivity region, the transition region and the high-load region.

[0062] In this embodiment, the morphological region to which each morphological marker belongs is determined by the morphological marker information in two consecutive frames of deformation images. At the same time, a sub-pixel-level positioning algorithm is used to achieve accurate positioning of the morphological marker and high-precision extraction of displacement. Then, an encoding algorithm adapted to the morphological region is used to encode the morphological marker and tactile deformation displacement field of the corresponding region, so as to achieve accurate extraction of tactile features of each region of the partitioned elastic contact layer. Among them, it has been proven in practice that the displacement extraction accuracy of the morphological marker can reach 0.1 micrometers for the high-sensitivity region and 1 micrometer for the high-load-bearing region. The anti-interference performance can be improved by 50% compared with the existing solution.

[0063] In one embodiment, step 303 may be followed by: When the displacement of the morphological markers is in a stable state in multiple consecutive deformed images, the current frame deformed image is used as the reference image for calculating the global tactile deformation displacement field with the next frame deformed image, and the reference image is dynamically updated.

[0064] The displacement of morphological markers in a series of deformed images can be determined to be in a stable state based on the following method: When the average displacement of all morphological markers in a series of consecutive deformation images with a preset number of frames is less than or equal to a preset stability threshold, it is determined that the displacement of the morphological markers in the series of consecutive deformation images is in a stable state, that is, the robot end effector grasping is in a stable state, such as the object being fully gripped and without relative slippage. The preset number of frames and the preset stability threshold can be set according to actual needs and are not limited here. In this embodiment, the preset number of frames can be set to 3 frames and the preset stability threshold can be set to 5 micrometers.

[0065] Furthermore, the current frame deformation image is used as the reference image for calculating the global tactile deformation displacement field with the next frame deformation image, and the current coordinates of each morphological marker in the reference image are stored. Subsequent tactile reconstruction is based on these coordinates to calculate the tactile deformation displacement field. Furthermore, if during subsequent grasping, it is detected that the average displacement of all morphological markers in a series of consecutive deformed images of a preset number of frames is greater than a preset stability threshold, such as when object slippage or gripper posture adjustment occurs, the stability determination is restarted. When the displacement of the morphological markers in the series of consecutive deformed images is re-determined to be in a stable state, the reference image is updated to achieve self-updating of the deformation reference state during the tactile reconstruction process.

[0066] In existing technologies, the deformation image acquired by the sensor in its initial unloaded reference state, i.e., under no stress, is usually used as the reference image for deformation field calculation. However, during the continuous grasping process of the robot's end effector, the sensor may undergo structural deformation, posture changes, and material creep and aging, causing the reference image in its initial unloaded reference state to drift, thus rendering the reference for deformation field calculation invalid.

[0067] In this embodiment, the current frame deformation image under the displacement stabilization state is selected as the reference image and dynamically updated. This can compensate for the reference drift caused by structural deformation, attitude change, material creep, and aging of the sensor. Practice has proven that the drift of tactile parameters can be controlled within 2%. Furthermore, in this embodiment, the reference images under the displacement stabilization state and the initial no-load reference state can coexist in parallel, respectively serving short-term dynamic deformation calculation and long-term drift compensation. Their switching and updating do not affect the established tactile deformation mode determination results.

[0068] Figure 4 This is the third schematic flowchart of the tactile reconstruction method provided in the embodiments of this application. (Refer to...) Figure 4 In one embodiment, step 203 may include: Step 401: When the displacement of the shape marker is in a stable state in multiple consecutive frames of deformed images, obtain sample features that match the tactile features from the preset pattern library. The preset pattern library is established based on the mapping relationship between sample features and their corresponding tactile deformation patterns; Step 402: Determine the current tactile deformation pattern of the tactile features based on the tactile deformation patterns corresponding to the sample features; Step 403: Based on the tactile parameter reconstruction model that matches the current tactile deformation pattern, perform constraint solving on the tactile information to obtain the reconstructed tactile information.

[0069] In steps 401 to 402, it can be determined that the displacement of the morphological marker in the continuous multi-frame deformed images is in a stable state based on the same method described above.

[0070] The preset mode library can be built based on simulation data, experimental data, or any combination of the two; no restrictions are imposed here.

[0071] Furthermore, sample features that match the tactile features can be obtained by calculating the similarity between the tactile features and each sample feature in the preset pattern library. The similarity calculation method is a classification algorithm based on distance metric, such as an improved KNN algorithm, or other arbitrary probability models or learning models, such as lightweight neural networks, such as MobileNet, or support vector machines, to further optimize the matching time.

[0072] Taking the distance-based classification algorithm as an example, the Euclidean distance between the tactile feature and each sample feature in the preset pattern library is calculated. The top 3 sample features with the smallest Euclidean distance to the tactile feature are selected, and the final matching sample feature is determined by voting according to weights. Then, the tactile deformation pattern corresponding to the sample feature is obtained and used as the current tactile deformation pattern of the tactile feature. The matching time can be controlled within a preset range, such as less than or equal to 0.5 milliseconds.

[0073] In step 403, the selection principle for the tactile parameter reconstruction model is to match the main physical characteristics of the current tactile deformation mode, rather than being limited to a specific model form. For example: 1. When the current tactile deformation mode is point contact or edge contact, Hertzian contact theory model can be used to assess the normal contact force. The constraint solution is performed, and the model formula is as follows: ; in, The contact stiffness coefficient is determined by the material parameters of the corresponding region of the partitioned elastic contact layer. The normal deformation is calculated from the normal displacement of the shape marker.

[0074] 2. Given that the current tactile deformation mode is a surface contact mode, an elastic mechanical plane stress model can be used to assess the average normal pressure. The constraint solution is performed, and the model formula is as follows: ; in, This represents the elastic modulus of the corresponding region in the partitioned elastic contact layer. The in-plane strain is calculated from the displacement field gradient of the corresponding region of the partitioned elastic contact layer.

[0075] 3. Given that the current tactile deformation mode is a sliding precursor mode, a shear force-displacement coupling model can be used to address the shear force. The constraint solution is performed, and the model formula is as follows: ; in, The coefficient of friction, The viscous damping coefficient is... is the shear strain rate.

[0076] It should be noted that the precursor to slip refers to the state in which an object is about to slip but has not yet produced obvious displacement, which is manifested as a sudden change in local shear strain rate.

[0077] When all three tactile deformation modes exist simultaneously in a certain area of ​​the partitioned elastic contact layer, a multi-dimensional tactile information set can be obtained. .

[0078] It should be noted that data-driven deep learning models, such as CNN-LSTM models, can also be used to replace tactile parameter reconstruction models, improving the reconstruction accuracy of complex contact states through training with a large number of samples.

[0079] This embodiment uses simulation data and / or experimental data to construct a preset pattern library, achieving comprehensive coverage of sample features. A similarity matching algorithm is used to obtain sample features matching tactile features and their corresponding tactile deformation patterns, resulting in short matching time. The current tactile deformation pattern then constrains the modeling logic to reconstruct tactile information, effectively improving the robustness of complex contact states. By employing differentiated parameterized modeling methods for different tactile deformation patterns, high-precision reconstruction of multi-dimensional tactile information is achieved. Comparative experiments verify that the accuracy of slip precursor pattern recognition can reach a high level, and the misjudgment rate during contact pattern abrupt changes can be controlled within a low range. In contact pattern abrupt change scenarios, compared to a single continuous regression modeling method, the pattern-constrained reconstruction strategy in this embodiment exhibits higher stability and robustness. Existing continuous regression models have a higher misjudgment rate in contact pattern abrupt change scenarios, with the specific value greatly affected by the model and application scenario.

[0080] In one embodiment, step 203 may be followed by: The pre-stored gripper structure deformation mapping model is invoked. Based on the current gripping force, current opening and closing angle, and current coordinates of the shape marker, the pseudo displacement caused by the gripper structure deformation is obtained. Based on the pseudo displacement, the displacement field in the reconstructed tactile information is corrected.

[0081] That is, the current clamping force, current opening and closing angle of the gripper, and the current coordinates of the tactile feature corresponding shape markers are input into the pre-stored gripper structure deformation mapping model to obtain the pseudo displacement caused by the gripper structure deformation output by the model. This pseudo displacement refers to the marker displacement that is unrelated to the contact with the object caused by the movement or elastic deformation of the gripper structure itself (such as slight bending during the opening and closing process). In this way, the displacement field in the reconstructed tactile information can be subtracted from the pseudo displacement to obtain the pure contact deformation displacement field, so as to eliminate the interference of gripper structure deformation.

[0082] It should be noted that the pseudo-displacement calculation in this embodiment can run independently of the tactile reconstruction process to perform state monitoring and structural health assessment of the gripper.

[0083] Furthermore, the deformation mapping model of the gripper structure can be calibrated beforehand. The gripper can hold the visual-tactile sensor of this application and press the calibration fixture, while a high-precision displacement measurement device tracks the gripper movement and collects the gripper displacement information. Combined with the deformation images collected by the visual-tactile sensor, the deformation mapping model of the gripper structure is calibrated. The measurement accuracy of the high-precision displacement measurement device must meet the gripper deformation detection requirements, and the sampling frequency must be adapted to the dynamic characteristics of the gripper movement. In this embodiment, the high-precision displacement measurement device can be a laser tracker with a measurement accuracy of not less than ±0.1 micrometers and a sampling frequency of 500 Hz. Figure 5 As shown.

[0084] It should be noted that nonlinear models such as BP neural network models can also be used to replace the multinomial regression model of the gripper structure deformation mapping model in this embodiment, in order to adapt to complex gripper structures, such as the nonlinear deformation of flexible grippers. In addition, when the end effector is of other types besides grippers, a similar method can be used to calibrate the adapted structural deformation mapping model and calculate the pseudo displacement accordingly, so as to correct the displacement field in the reconstructed tactile information, or simply to perform state monitoring and structural health assessment of the end effector.

[0085] In this embodiment, pseudo-displacement is calculated based on a high-precision calibrated gripper structure deformation mapping model, and the reconstructed tactile information is corrected accordingly. This enables precise compensation for non-contact deformation, effectively eliminates non-contact deformation interference, and improves the accuracy of tactile reconstruction. After correction, the reconstruction error can be controlled within a range of less than or equal to 3%, while existing schemes that do not consider this interference typically have an error of more than 12%.

[0086] In one embodiment, step 203 may be followed by: The system acquires multi-dimensional indicators for evaluating the stability of the gripper. If all multi-dimensional indicators meet the preset conditions, the gripper's grasping state is determined to be stable. If any single indicator does not meet the preset conditions, the gripper's grasping state is determined to be unstable, and the gripper's adaptive action is triggered.

[0087] The multi-dimensional indicators and their preset conditions can be obtained according to actual needs, and are not limited here. In this embodiment, the multi-dimensional indicators and their preset conditions may include: 1. Normal force change rate: The normal force fluctuation amplitude corresponding to multiple consecutive frames of deformation images is less than or equal to a preset threshold, such as less than or equal to 5%, indicating that the clamping force is stable; 2. Shear force ratio: The ratio of shear force to normal force is less than or equal to the critical friction ratio, for example, ≤0.8 times the coefficient of friction, indicating no risk of slippage; 3. Displacement field fluctuation: The average displacement fluctuation of the global morphological marker is less than or equal to a preset threshold, for example, less than or equal to 2 micrometers, indicating that there is no sudden change in the contact state; 4. Centroid displacement of the contact area: If the centroid displacement of the contact area corresponding to multiple consecutive frames of deformation images is less than or equal to a preset threshold, such as less than or equal to 3 micrometers, it indicates that the contact position is stable. 5. Probability of slip precursors: If the detection probability of slip precursor features is less than or equal to a preset threshold, such as less than or equal to 10%, it indicates that the grasping posture is stable.

[0088] If all the above indicators meet their preset conditions, the gripper's grasping state is determined to be stable. If any indicator does not meet its preset conditions, the gripper's grasping state is determined to be unstable. An unstable warning signal is then output to the robot's main controller through the robot interface module to trigger adaptive actions such as gripper force adjustment and posture calibration.

[0089] This embodiment adapts to the closed-loop control requirements of robots by employing a multi-dimensional index-based stability judgment mechanism and real-time feedback control logic. The real-time feedback mechanism features low latency, less than or equal to 2 milliseconds. In typical experimental scenarios, the stability judgment accuracy reaches a high level, meeting the practical application needs of industrial grasping. It can be directly integrated with robot control algorithms, significantly improving the success rate of grasping complex scenarios, such as curved parts and thin objects. During repeated grasping operations, the system stability remains at a high level without performance degradation.

[0090] Reference Figure 6 In one embodiment, the overall process of the tactile reconstruction method of this application can be briefly described as follows: 1. Acquisition and preprocessing of continuous multi-frame deformed images to obtain preprocessed continuous multi-frame deformed images; 2. Based on the preprocessed multi-frame deformation images, extract tactile features, including the ID of the morphological marker, initial coordinates, morphological features, and tactile deformation displacement field; 3. Online freezing of tactile reference state, that is, displacement stability judgment based on morphological markers, using the current frame deformation image as the reference image for calculating the global tactile deformation displacement field with the next frame deformation image, and dynamically updating the reference image to obtain the updated reference state. 4. Based on a preset pattern library, perform tactile deformation pattern matching on tactile features, including point / edge contact patterns, surface contact patterns, and sliding precursor patterns; 5. Based on the pattern matching tactile parameter reconstruction model, multi-dimensional tactile information is reconstructed to obtain a multi-dimensional tactile information set; 6. Based on the current gripping force, current opening and closing angle of the gripper, and the current coordinates of the shape marker, calculate the pseudo-displacement to compensate for the deformation of the gripper structure and obtain the corrected tactile reconstruction information. 7. Based on multi-dimensional indicators, the gripper's grasping stability is judged and feedback is provided. The stability signal and the corrected tactile reconstruction information are output to the robot's main controller through the robot interface module, achieving complete closed-loop control.

[0091] It should be noted that the tactile reconstruction method in this embodiment can be encapsulated as a software development kit, supporting calls from mainstream programming languages ​​such as C++ and Python, and is also compatible with mainstream robot control systems such as ROS (Robot Operating System), Fanuc Focas (FANUC Open Computer Numerical Control API Specifications), and KUKA connect.

[0092] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 7 As shown, the electronic device may include a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call a computer program stored in the memory 730 to execute steps of the tactile reconstruction method, such as: Acquire continuous multi-frame deformation images captured by the visual-tactile sensor; Based on the continuous multi-frame deformation images, tactile features are extracted; Based on the aforementioned tactile features, tactile information is reconstructed.

[0093] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the tactile reconstruction method provided in the above embodiments, such as including: Acquire continuous multi-frame deformation images captured by the visual-tactile sensor; Based on the continuous multi-frame deformation images, tactile features are extracted; Based on the aforementioned tactile features, tactile information is reconstructed.

[0095] On the other hand, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, the computer program being used to cause a processor to execute the steps of the tactile reconstruction methods provided in the above embodiments, for example including: Acquire continuous multi-frame deformation images captured by the visual-tactile sensor; Based on the continuous multi-frame deformation images, tactile features are extracted; Based on the aforementioned tactile features, tactile information is reconstructed.

[0096] The non-transitory computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A visual-tactile sensor, characterized in that, include: Partitioned elastic contact layer, illumination control module, and image acquisition module; The partitioned elastic contact layer includes a high-sensitivity zone, a transition zone, and a high-load-bearing zone; The elastic modulus of the high-sensitivity region, the transition region, and the high-load-bearing region increases sequentially. The illumination control module and the image acquisition module are located on the side of the partitioned elastic contact layer away from the contact surface. The illumination control module is used to provide illumination light, and the image acquisition module is used to acquire deformation images of the partitioned elastic contact layer under the illumination light.

2. The visual-tactile sensor according to claim 1, characterized in that, Also includes: Encoding visual marker layer; The coded visual marker layer is attached to the surface of the partitioned elastic contact layer on the side away from the contact surface; The encoded visual marker layer includes a first morphological region covered by a first morphological marker, a second morphological region covered by a second morphological marker, and a third morphological region covered by a third morphological marker. The first morphological region is aligned with the high-sensitivity region to sense the deformation characteristics of the high-sensitivity region; The second morphological region is aligned with the high load-bearing region and is used to sense the deformation characteristics of the high load-bearing region; The third morphological region is aligned with the transition region and is used to sense the deformation characteristics of the transition region.

3. The visual-tactile sensor according to claim 2, characterized in that, Also includes: Buffer support layer; The buffer support layer is attached to the surface of the coded visual marker layer on the side away from the partitioned elastic contact layer; The illumination control module and the image acquisition module are embedded within the buffer support layer.

4. The visual-tactile sensor according to claim 1, characterized in that, Also includes: Light-shielding contact layer; The light-shielding contact layer is attached to the contact surface of the partitioned elastic contact layer; The side of the light-shielding contact layer away from the contact surface of the partitioned elastic contact layer has a micro-textured structure and is coated with elastic reflective powder.

5. A tactile reconstruction method based on the visual-tactile sensor according to any one of claims 1 to 4, characterized in that, include: Acquire continuous multi-frame deformation images captured by the visual-tactile sensor; Based on the continuous multi-frame deformation images, tactile features are extracted; Based on the aforementioned tactile features, tactile information is reconstructed.

6. The tactile reconstruction method according to claim 5, characterized in that, The extraction of tactile features based on the continuous multi-frame deformation images includes: For any two consecutive deformed images in the aforementioned multi-frame deformed images, identify the morphological marker information in the two consecutive deformed images; the morphological marker information includes the ID, initial coordinates, and morphological features of the morphological marker. Based on the morphological marker information, the morphological region to which the morphological marker belongs is determined; For the same shape marker with the same ID in two consecutive deformed images, based on the current coordinates of the shape marker and the calibration parameters of the image acquisition module, the pixel displacement of the shape marker on the two consecutive deformed images is converted into physical displacement to obtain a global tactile deformation displacement field. Based on the encoding algorithm adapted to the morphological region, the morphological marking information and the global tactile deformation displacement field are partitioned and encoded to obtain the tactile features corresponding to each region in the partitioned elastic contact layer.

7. The tactile reconstruction method according to claim 6, characterized in that, After obtaining the global tactile deformation displacement field, the following is included: When the displacement of the morphological markers is in a stable state in the continuous multi-frame deformation images, the current frame deformation image is used as the reference image for calculating the global tactile deformation displacement field with the next frame deformation image, and the reference image is dynamically updated.

8. The tactile reconstruction method according to claim 5, characterized in that, The reconstruction of tactile information based on the tactile features includes: When the displacement of the shape marker is in a stable state in the continuous multi-frame deformation images, sample features that match the tactile features are obtained from a preset pattern library; the preset pattern library is established based on the mapping relationship between the sample features and their corresponding tactile deformation patterns. Based on the tactile deformation pattern corresponding to the sample feature, determine the current tactile deformation pattern of the tactile feature; Based on the tactile parameter reconstruction model that matches the current tactile deformation pattern, the tactile information is constrained and solved to obtain the reconstructed tactile information.

9. The tactile reconstruction method according to claim 5, characterized in that, After reconstructing the tactile information, the following is included: The pre-stored gripper structure deformation mapping model is invoked, and the pseudo displacement caused by the gripper structure deformation is obtained based on the current gripping force, current opening and closing angle and current coordinates of the shape marker. Based on the pseudo-displacement, the displacement field in the reconstructed tactile information is corrected.

10. The tactile reconstruction method according to claim 5, characterized in that, After reconstructing the tactile information, the following is included: Obtain multi-dimensional indicators for gripper stability assessment; If all the multi-dimensional indicators meet the preset conditions, then the gripper's grasping state is determined to be stable. If any dimension indicator does not meet the preset conditions, it is determined that the gripper's grasping state is unstable, and the gripper's adaptive action is triggered.

Images