A tactile sensor, preparation method and point cloud reconstruction method

Abstract

The present invention provides a tactile sensor, a preparation method and a point cloud reconstruction method. The tactile sensor includes a binocular camera, a sensor frame and a gel module. The contact surface of the tactile sensor is a curved surface. The sensor frame includes a sensor support frame and a sensor lighting circuit. The sensor lighting circuit is composed of two circuit boards, and each circuit board is provided with a plurality of lighting lamp beads. The gel module includes a gel layer and a coating. Marking points are evenly distributed on the outer surface of the gel layer. When contacting an object, the gel layer produces geometric deformation, resulting in displacement of the marking points on the outer surface of the gel layer. The three-dimensional position of the marking points is calculated by the displacement of the marking points, so that tactile information can be extracted from the three-dimensional point cloud. The defects of traditional tactile sensors that cannot perceive three-dimensional geometric information, high production cost, poor application convenience, narrow application range and low precision are overcome, and the perception and dexterous operation of robots are facilitated.

Description

Technical Field

[0001] The present invention relates to the field of tactile sensing technology, and in particular to a tactile sensor, a preparation method and a point cloud reconstruction method. Background Art

[0002] Touch is used to perceive the hardness, shape and texture of an object. Tactile signals are crucial for robots to perceive and operate the environment. Among them, three-dimensional geometric information plays a very important role in the robot's object recognition, object grasping and dexterous manipulation tasks.

[0003] At present, among the tactile sensors based on various conduction principles, visual tactile sensors are widely used due to their high spatial resolution, low cost and high sensitivity. However, visual tactile sensors lack three-dimensional geometric perception capabilities and require complex encoding of tactile signals, which makes their application inconvenient.

[0004] Furthermore, the GelSight sensor, which is widely used in visual-tactile sensors, uses a photometric stereo algorithm to reconstruct the three-dimensional geometric deformation of the contact surface. However, this type of sensor requires a complex lighting system and specific calibration parts for color calibration, which brings difficulties to the manufacturing of the sensor and the improvement of its accuracy. In addition, the application of a photometric stereo algorithm makes it difficult to expand the contact surface of this type of sensor from a flat or slightly curved surface to a three-dimensional geometric surface.

[0005] Therefore, in the field of tactile sensing, the design and production of tactile sensors that can obtain three-dimensional geometric perception, are simple to manufacture, low-cost, and have a wide range of applications has become an urgent problem to be solved. Summary of the invention

[0006] The present invention provides a tactile sensor, a preparation method and a point cloud reconstruction method, which are used to solve the defects of the prior art tactile sensors that cannot perceive three-dimensional geometric information, have high production costs, poor application convenience, narrow application range and low precision.

[0007] The present invention provides a tactile sensor, comprising a binocular camera, a sensor frame, and a gel module, wherein the contact surface of the tactile sensor is a curved surface, the sensor frame is connected to the binocular camera and is mounted on the periphery of the binocular camera, and the gel module is connected to the sensor frame and is sleeved on the top of the sensor frame;

[0008] The sensor frame includes a sensor support frame and a sensor lighting circuit. The sensor lighting circuit is a ring structure. The sensor lighting circuit is composed of two circuit boards, and each circuit board is provided with a plurality of lighting lamp beads.

[0009] The gel module comprises a gel layer and a coating, the gel layer wraps a gel support plate, the outer surface of the gel layer is a curved surface, and the gel support plate is a hemispherical structure;

[0010] The outer surface of the gel layer is evenly distributed with marking points, the marking points are composed of a central marking point and an outer circle marking points, and the outer circle marking points are evenly distributed on a plurality of concentric circles with the central marking point as the center; the coating is attached to the outer surface of the gel layer.

[0011] The present invention also provides a method for preparing a tactile sensor, comprising:

[0012] Calibrate the camera intrinsic parameters of the binocular camera and the position relationship between the left camera and the right camera in the binocular camera, wherein the camera intrinsic parameters include at least one of the camera size, camera viewing angle, baseline distance, distortion condition, shooting frame rate, and imaging size;

[0013] Perform three-dimensional printing based on photosensitive resin to obtain a sensor support frame, obtain a sensor lighting circuit based on splicing two circuit boards, and assemble a sensor frame based on the sensor support frame and the sensor lighting circuit;

[0014] Prepare a gel layer based on a gel mold and a gel support plate, inject pigment into pits on the outer surface of the gel layer, dry to obtain marking points distributed on the outer surface of the gel layer, spray a coating liquid on the outer surface of the gel layer with the marking points, dry to obtain a coating, and assemble a gel module based on the gel layer and the coating, wherein the coating liquid is obtained by mixing a silicone adhesive, a gray silicone pigment and a diluent;

[0015] Calibrate the position relationship between the sensor frame coordinate system and the left camera coordinate system based on the binary code, wherein the sensor frame coordinate system is constructed with the sensor frame as the origin, and the left camera coordinate system is constructed with the left camera in the binocular camera as the origin;

[0016] The binocular camera, the sensor frame and the gel module are assembled to obtain the tactile sensor.

[0017] According to a method for preparing a tactile sensor provided by the present invention, the gel mold comprises a bottom gel mold and an upper gel mold, and the gel mold and the gel support plate are obtained by three-dimensional printing through a transparent photosensitive resin;

[0018] The method of preparing the gel layer based on the gel mold and the gel support plate comprises:

[0019] Selecting a transparent two-component silica gel, mixing the first component and the second component of the two-component silica gel to obtain a silica gel mixed liquid;

[0020] Pour the silica gel mixture into the bottom gel mold, place the gel support plate on the bottom gel mold with the silica gel mixture, and pour the silica gel mixture into the gel support plate until it covers the gel support plate;

[0021] The upper gel mold is placed on the gel support plate, and the bottom gel mold, the gel support plate and the upper gel mold are fixed with fasteners to obtain a fixing part, and the gel layer is taken out after the silica gel mixture in the fixing part is left to solidify.

[0022] According to a method for preparing a tactile sensor provided by the present invention, the position and posture relationship between the sensor frame coordinate system and the left camera coordinate system is calibrated based on binary code, including:

[0023] Performing three-dimensional printing based on an opaque photosensitive resin to obtain a three-dimensional printed part, pasting a binary code on the three-dimensional printed part, installing the three-dimensional printed part on the sensor frame, and determining a relative position between the three-dimensional printed part and a sensor frame coordinate system;

[0024] Shooting the binary code on the three-dimensional printed part by the left camera to obtain a binary image, and determining the relative position and posture between the three-dimensional printed part and the left camera coordinate system based on the binary image;

[0025] Based on the relative posture between the three-dimensional print and the sensor frame coordinate system, and the relative posture between the three-dimensional print and the left camera coordinate system, the posture relationship between the sensor frame coordinate system and the left camera coordinate system is determined.

[0026] The present invention also provides a point cloud reconstruction method based on a tactile sensor, comprising:

[0027] Obtaining a left-eye image and a right-eye image captured by a binocular camera of a tactile sensor;

[0028] Matching the left target mark point in the left image with the right target mark point in the right image, and determining the pixel coordinates of the mark points corresponding to the two-dimensional mark points to be reconstructed in the left image and the right image based on the matching relationship between the two target mark points obtained by matching, wherein the mark points include the left two-dimensional mark points and the right two-dimensional mark points;

[0029] Based on the pixel coordinates of the marking point, determine the pseudo point corresponding to the two-dimensional marking point in the gel layer of the tactile sensor and the coordinates of the pseudo point in the gel layer coordinate system; based on the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system and the inner spherical radius of the gel layer, determine the left eye incident light and the right eye incident light; the pseudo point is the intersection of the straight line where the left eye refracted light corresponding to the left eye incident light is located and the straight line where the right eye refracted light corresponding to the right eye incident light is located;

[0030] Based on the left-eye incident light and the right-eye incident light, and the posture transformation matrix between the gel layer coordinate system and the sensor frame coordinate system, a three-dimensional point cloud of the two-dimensional marking point in the sensor frame coordinate system is determined.

[0031] According to the present invention, a point cloud reconstruction method based on a tactile sensor is provided, wherein the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system, and the inner spherical radius of the gel layer, are used to determine the left eye incident light and the right eye incident light, including:

[0032] Based on the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system, and the radius of the inner spherical surface of the gel layer, determine the coordinates of the refraction point of the left eye refracted light and the refraction point of the right eye refracted light in the gel layer coordinate system, wherein the refraction point is the intersection of the corresponding refracted light and the inner spherical surface of the gel layer;

[0033] Based on the coordinates of the refraction point in the gel layer coordinate system, the coordinates of the optical center of the binocular camera in the gel layer coordinate system, the refractive index of the medium in the gel layer, the refractive index of air, and the inner spherical radius of the gel layer, determine the left eye incident light and the right eye incident light corresponding to the left eye refracted light and the right eye refracted light respectively.

[0034] According to the present invention, a point cloud reconstruction method based on a tactile sensor is provided, wherein based on the pixel coordinates of the marking point, a pseudo point corresponding to the two-dimensional marking point in the gel layer of the tactile sensor and the coordinates of the pseudo point in the gel layer coordinate system are determined, including:

[0035] Based on the pixel coordinates of the marking point, determining a pseudo point corresponding to the two-dimensional marking point in the gel layer of the tactile sensor, and the coordinates of the pseudo point in the left camera coordinate system;

[0036] Based on the coordinates of the pseudo point in the left camera coordinate system and the pose transformation matrix between the gel layer coordinate system and the left camera coordinate system, the coordinates of the pseudo point in the gel layer coordinate system are determined.

[0037] According to the present invention, a point cloud reconstruction method based on a tactile sensor is provided, wherein the matching of the left target points in the left image and the right target points in the right image comprises:

[0038] In the case where the left-eye image and the right-eye image are first frame images captured by the binocular camera, the left-eye image and the right-eye image are corrected, and a left-eye polar coordinate system and a right-eye polar coordinate system are established based on the corrected left-eye image and the right-eye image;

[0039] Based on the polar coordinates of the left target mark point in the left eye polar coordinate system and the polar coordinates of the right target mark point in the right eye polar coordinate system, sort the mark points to obtain the sequence number of the left target mark point and the sequence number of the right target mark point;

[0040] The left target point and the right target point are matched based on the sequence number of the left target point and the sequence number of the right target point.

[0041] According to the present invention, a point cloud reconstruction method based on a tactile sensor is provided, wherein the marking points are sorted based on the polar coordinates of the left target mark points in the left eye polar coordinate system and the polar coordinates of the right target mark points in the right eye polar coordinate system to obtain the sequence numbers of the left target mark points and the sequence numbers of the right target mark points, including:

[0042] Based on the polar diameter in the polar coordinates of the left target mark point, the marked points are grouped to obtain the left target mark point in each polar diameter circle, and based on the polar diameter in the polar coordinates of the right target mark point, the target mark points are grouped to obtain the right target mark point in each polar diameter circle;

[0043] Based on the polar angle in the polar coordinates of the left target point, the left target points in the same polar circle are sorted to obtain the serial number of the left target point, and based on the polar angle in the polar coordinates of the right target point, the right target points in the same polar circle are sorted to obtain the serial number of the right target point.

[0044] According to the present invention, a point cloud reconstruction method based on a tactile sensor is provided, wherein the matching of the left target points in the left image and the right target points in the right image comprises:

[0045] In the case that the left-eye image and the right-eye image are not the first frames of images captured by the binocular camera, based on the previous left-eye image and the previous right-eye image, the left target point in the left-eye image and the right target point in the right-eye image are tracked, and the matching relationship between the two target points is determined.

[0046] The tactile sensor, preparation method and point cloud reconstruction method provided by the present invention include a binocular camera, a sensor frame and a gel module. When contacting an object, the gel layer undergoes geometric deformation, causing the marking points attached to the outer surface of the gel layer to be displaced. The three-dimensional position of the marking points can be calculated through the displacement of the marking points, thereby extracting tactile information from the three-dimensional point cloud. This overcomes the defects of traditional tactile sensors that cannot perceive three-dimensional geometric information, have high production costs, poor application convenience, narrow application range and low precision, and provides assistance for the perception and dexterous operation of robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 is a schematic structural diagram of a tactile sensor provided by the present invention;

[0049] Figure 2 It is a schematic diagram of a process for preparing a tactile sensor provided by the present invention;

[0050] Figure 3 is a schematic diagram of a gel mold and a gel support plate provided by the present invention;

[0051] Figure 4 is a schematic diagram of the preparation process of the gel layer provided by the present invention;

[0052] Figure 5 is a schematic diagram of a binary code provided by the present invention;

[0053] Figure 6 It is a flow chart of a point cloud reconstruction method based on a tactile sensor provided by the present invention;

[0054] Figure 7 is an abstract diagram of the point cloud reconstruction process provided by the present invention;

[0055] Figure 8 It is a structural schematic diagram of the manufacturing device of the tactile sensor provided by the present invention

[0056] Fig. 9 is a structural schematic diagram of a point cloud reconstruction device based on a tactile sensor provided by the present invention;

[0057] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention.

[0058] Reference numerals:

[0059] 110: binocular camera; 120: sensor frame; 121: sensor support frame; 122: sensor lighting circuit; 130: gel module; 131: gel layer; 1311: gel support plate; 1312: marking point; 132 coating. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Touch is used to sense the hardness, shape and texture of an object. Tactile signals are crucial for robots to perceive and operate the environment. Among them, three-dimensional geometric information can play a key role in machine tasks such as object recognition, object grasping and dexterous manipulation.

[0062] At present, among the tactile sensors based on various conduction principles, visual tactile sensors are widely used due to their high spatial resolution, low cost and high sensitivity. Most visual tactile sensors capture the physical deformation of the elastic body through a camera, and interpret the tactile information through the two-dimensional movement of the texture or marking point caused by the deformation of the elastic body. However, this type of visual tactile sensor lacks three-dimensional geometric perception capabilities. In short, it requires complex encoding of the collected tactile signals before it can be used for subsequent recognition tasks or operation tasks.

[0063] As a visual tactile sensor with a wide range of applications, the GelSight sensor can use the photometric stereo algorithm to accurately reconstruct the three-dimensional geometric deformation of the contact surface. However, the photometric stereo algorithm has the defect of large reconstruction errors on the edges of the contact surface, and this type of sensor also requires a complex lighting system and specific calibration parts for color calibration, all of which make it extremely difficult to manufacture and improve the reconstruction accuracy of this type of sensor. In addition, the strict requirements of the photometric stereo algorithm for lighting will make it difficult for the contact surface of this type of sensor to expand from a flat or slightly curved surface to a three-dimensional geometric surface, such as a hemispherical shape or a shape similar to a human fingertip.

[0064] Compared with flat sensors, curved sensors can sense contact in a wider range, which is very important for the dexterous operation of robots, especially in the multi-finger operation of robots. Among curved sensors, GelSight-like fingertip sensors approximate the curved contact surface into multiple planes for calibration, but the sensor still has problems such as complex light source, difficult calibration, and low accuracy; DenseTact fingertip sensors use monocular tactile images as input to the neural network to estimate the three-dimensional geometric shape, but monocular depth estimation is an ill-posed problem and difficult to solve; Soft-bubble sensors are composed of a depth camera and a latex film, which form a confined space. Inflating the confined space can cause the latex film to undergo elastic deformation, and the depth camera can obtain the three-dimensional point cloud of the surface. However, the sensor is large in size, high in cost, and has a small scope of application.

[0065] In addition, binocular vision systems are also used in visual tactile sensors to estimate the three-dimensional contact deformation of surfaces. However, in the binocular vision system of the tactile sensor, light passes through a transparent elastic body and air before entering the camera to form a tactile image. In addition, the use of traditional triangulation principles to reconstruct three-dimensional point clouds will inevitably lead to large errors. Currently, binocular vision-based sensors often ignore this problem, or use fitting methods to correct the error. However, the fitting method requires complex data acquisition programs and high-precision data acquisition equipment, and has poor generalization.

[0066] In this regard, the present invention provides a tactile sensor (GelStereo Palm sensor) with a curved contact surface, capable of sensing three-dimensional geometric information, simple to manufacture, low cost, and a wide range of applications, which can provide assistance for the perception and dexterous operation of robots. Figure 1 is a schematic diagram of the structure of the tactile sensor provided by the present invention, such as Figure 1 As shown, the tactile sensor includes a binocular camera 110, a sensor frame 120, and a gel module 130. The contact surface of the tactile sensor is a curved surface. The sensor frame 120 is connected to the binocular camera 110 and is mounted on the periphery of the binocular camera 110. The gel module 130 is connected to the sensor frame 120 and is sleeved on the top of the sensor frame 120.

[0067] The sensor frame 120 includes a sensor support frame 121 and a sensor lighting circuit 122. The sensor lighting circuit 122 is a ring structure. The sensor lighting circuit 122 is composed of two circuit boards, and each circuit board is provided with a plurality of lighting lamp beads.

[0068] The gel module 130 includes a gel layer 131 and a coating layer 132. The gel layer 131 contains a gel support plate 1311. The outer surface of the gel layer 130 is a curved surface, and the gel support plate 1311 is a hemispherical structure.

[0069] Marking points 1312 are evenly distributed on the outer surface of the gel layer 131. The marking points 1312 are composed of a central marking point and an outer circle marking point. The outer circle marking points are evenly distributed on a plurality of concentric circles with the central marking point as the center. The coating 132 is attached to the outer surface of the gel layer.

[0070] Specifically, the tactile sensor in the embodiment of the present invention has a curved contact surface, in other words, its contact surface is a curved surface, which includes a binocular camera 110, a sensor frame 120 and a gel module 130, wherein the binocular camera 110 includes a left camera and a right camera, and the types of the left camera and the right camera can be set accordingly according to actual needs. However, when selecting the camera, it is necessary to ensure that the camera intrinsic parameters of the left camera and the right camera meet the preset requirements. The preset requirements here are the requirements set for the camera size, camera angle of view, baseline distance, distortion condition, shooting frame rate, imaging size, etc. in the camera intrinsic parameters. For example, the camera size is required to be no more than 40×30×20mm, the camera angle of view is not less than 120°, the baseline distance is not more than 11.5mm, etc.

[0071] Among them, the sensor frame 120 is connected to the binocular camera 110 and is mounted on the periphery of the binocular camera 110. It is composed of a sensor support frame 121 and a sensor lighting circuit 122. The sensor support frame 121 is a printed part, which can be obtained by three-dimensional printing. In the embodiment of the present invention, in order to ensure the accuracy of the sensor and reduce the interference of external light, black photosensitive resin can be selected for three-dimensional printing, so as to obtain a black sensor support frame 121. The sensor support frame 121 can eliminate the influence of external light on the imaging of the tactile sensor to a great extent.

[0072] The sensor lighting circuit 122 is an annular structure, which is composed of two circuit boards. The shapes and compositions of the two circuit boards may be consistent or inconsistent, and the embodiment of the present invention does not specifically limit this. When consistent, the two circuit boards are both semi-annular structures, and the annular sensor lighting circuit 122 can be spliced ​​by splicing two semi-annular circuit boards.

[0073] There are multiple lighting beads distributed on each circuit board. To ensure uniform illumination, the lighting beads can be evenly distributed and connected in series. The distribution spacing between the lighting beads, the illumination angle of the lighting beads, the lighting power, the lamp bead model, the number of lamp beads, etc. can be set according to actual needs. For example, the illumination angle can be 30 degrees, the lighting power can be 0.2W, the lamp bead model can be 2835LED lamp beads, and the distribution spacing can be less than 2mm (about 2mm).

[0074] Furthermore, to ensure uniformity of illumination, an adjustment circuit may be connected in an embodiment of the present invention to control the brightness of the lamp beads through the adjustment circuit, thereby achieving uniform illumination, that is, the two circuit boards in the sensor lighting circuit 122 are connected to a voltage divider circuit board with an adjustable resistor, and the adjustable resistor is adjusted to change the brightness of the lamp beads. In other words, the lighting power of the lighting lamp beads is adjusted by changing the resistance value of the adjustable resistor, thereby controlling the brightness of the lamp beads.

[0075] Among them, the gel module 130 is connected to the sensor frame 120 and is mounted on the top of the sensor frame 120, and includes a gel layer 131 and a coating 132; wherein the gel layer 131 can be prepared by a gel mold and a gel support plate 1311, and the gel layer is made of silicone, and the gel support plate 1311 is wrapped in the gel layer 131. In other words, the gel support plate 1311 is wrapped inside the silicone for preparing the gel layer 131.

[0076] Here, the gel mold and the gel support plate 1311 are both printed parts, which can be obtained by three-dimensional printing of transparent photosensitive resin, are hemispherical, and their outer surface is a curved surface. In addition, marking points 1312 are evenly distributed on the outer surface of the gel layer 131, and the marking points 1312 are composed of a central marking point and an outer circle marking point. The outer circle marking points are evenly distributed on a plurality of concentric circles with the central marking point as the center. The number of concentric circles is determined according to the size of the gel layer 131, and the number of outer circle marking points distributed on each concentric circle can be set according to actual needs. Preferably, the number of concentric circles in the embodiment of the present invention is 11, and the number of outer circle marking points on each concentric circle from the inside to the outside with the central marking point as the center is 6, 12, 18, 24, 30, 36, 42, 48, 54, 60 and 66 respectively.

[0077] The marking points 1312 are made of colored pigment (such as black acrylic paint). The pigment is injected into a plurality of evenly distributed pits on the outer surface of the gel layer 131, and evenly distributed marking points 1312 are obtained after drying.

[0078] When contacting an object, the gel layer 131 undergoes geometric deformation, and the marking point 1312 attached to the outer surface of the gel layer 131 is displaced. The three-dimensional position of the marking point 1312 can be calculated through the displacement of the marking point 1312, so that tactile information can be extracted from the three-dimensional point cloud.

[0079] The coating 132 is attached to the outer surface of the gel layer 131 with the marking point 1312, and its manufacturing material is a coating liquid, which is sprayed on the outer surface of the gel layer 131 with the marking point 1312 and left to stand to obtain the coating 132. The coating liquid is obtained by mixing a silicone adhesive, a gray silicone pigment and a diluent in a certain ratio (such as 1:1:3).

[0080] The tactile sensor in the embodiment of the present invention has a curved contact surface, which can capture the geometric deformation of the elastic body on the contact surface, thereby realizing the perception of three-dimensional geometric information. In addition, the tactile sensor is simple to manufacture, low in cost, convenient to use and has a wide range of applications.

[0081] The tactile sensor provided by the present invention includes a binocular camera, a sensor frame and a gel module. When contacting an object, the gel layer produces geometric deformation, causing the marking points attached to the outer surface of the gel layer to be displaced. The three-dimensional position of the marking points can be calculated through the displacement of the marking points, so that tactile information can be extracted from the three-dimensional point cloud. This overcomes the defects of traditional tactile sensors that cannot perceive three-dimensional geometric information, have high production costs, poor application convenience, narrow application range and low precision, and provides assistance for the perception and dexterous operation of robots.

[0082] The present invention also provides a method for preparing a tactile sensor. Figure 2 is a schematic diagram of a process for preparing a tactile sensor provided by the present invention, such as Figure 2 As shown, the method includes:

[0083] Step 210, calibrating the camera intrinsic parameters of the binocular camera and the position relationship between the left camera and the right camera in the binocular camera, the camera intrinsic parameters including at least one of the camera size, camera viewing angle, baseline distance, distortion condition, shooting frame rate, and imaging size;

[0084] Step 220, performing three-dimensional printing based on photosensitive resin to obtain a sensor support frame, obtaining a sensor lighting circuit based on splicing two circuit boards, and assembling a sensor frame based on the sensor support frame and the sensor lighting circuit;

[0085] Step 230, preparing a gel layer based on the gel mold and the gel support plate, injecting pigment into the pits on the outer surface of the gel layer, drying to obtain marking points distributed on the outer surface of the gel layer, spraying a coating liquid on the outer surface of the gel layer with the marking points, drying to obtain a coating, and forming a gel module based on the gel layer and the coating, wherein the coating liquid is obtained by mixing a silicone adhesive, a gray silicone pigment and a diluent;

[0086] Step 240, calibrating the position and posture relationship between the sensor frame coordinate system and the left camera coordinate system based on the binary code, the sensor frame coordinate system is constructed with the sensor frame as the origin, and the left camera coordinate system is constructed with the left camera in the binocular camera as the origin;

[0087] Step 250, assemble the binocular camera, the sensor frame and the gel module to obtain a tactile sensor.

[0088] Specifically, the design idea of ​​the tactile sensor (GelStereo Palm sensor) is to convert tactile information into geometric deformation of the elastic body, so as to prepare for the subsequent three-dimensional reconstruction of the elastic contact surface. It consists of three parts: a binocular camera, a sensor frame and a gel module. The overall preparation process includes the following steps:

[0089] First, execute step 210, and perform parameter and posture calibration; wherein parameter calibration is to calibrate the camera internal parameters of the binocular camera, and posture calibration is to calibrate the posture relationship between the left camera and the right camera in the binocular camera;

[0090] Here, the camera intrinsic parameters may be one or more of the camera size, camera viewing angle, baseline distance, distortion condition, shooting frame rate, and imaging size of the left camera and the right camera in the binocular camera. The camera intrinsic parameters may be set accordingly according to the application scenario, actual situation, task requirements, etc.

[0091] Specifically in the embodiment of the present invention, the camera size of the binocular camera should not exceed 40×30×20mm, the camera viewing angle should not be less than 120°, the baseline distance should not be greater than 11.5mm, no distortion should be generated, the shooting frame rate should not be lower than 30FPS, and the imaging size should not be less than 1080P.

[0092] Preferably, in the camera intrinsic parameters of the binocular camera in the embodiment of the present invention, the camera size is 39×28 mm, the camera viewing angle is 120°, the baseline distance is 11.5 mm, no distortion is generated, the shooting frame rate is 30 FPS, and the imaging size is 1080P.

[0093] The posture calibration process is to determine the relative posture of the left camera and the right camera in the binocular camera, and determine the posture relationship between the left camera and the right camera based on the relative posture, and calibrate the posture relationship between the two, thus completing the calibration process in the preparation of the tactile sensor.

[0094] Then, step 220 is executed to prepare the sensor frame. The specific process may be to first perform three-dimensional printing using photosensitive resin to obtain the sensor support frame. In the process of manufacturing the sensor support frame, in order to ensure the manufacturing accuracy, the photosensitive resin may be black. The black sensor support frame obtained by three-dimensional printing with black photosensitive resin can effectively prevent the influence of external light on the imaging of the tactile sensor. Then, the sensor lighting circuit can be obtained by splicing two circuit boards. In the embodiment of the present invention, in order to adapt to the sensor support frame, the sensor lighting circuit can be a ring structure, and the corresponding two circuit boards can be regular semi-rings or other structures. The embodiment of the present invention does not specifically limit this. Thereafter, the sensor frame can be formed based on the two.

[0095] Among them, there are multiple lighting beads distributed on each circuit board. To ensure uniform illumination, the lighting beads can be evenly distributed and connected in series. The distribution spacing between the lighting beads on each circuit board, the illumination angle of the lighting beads, the lighting power, the lamp bead model, the number of lamp beads, etc. can be set accordingly according to the application scenario, actual needs, etc. As a preference, the two circuit boards selected in the embodiment of the present invention have the same specifications and are both semi-annular structures. Each circuit board is connected in series with 12 evenly distributed 2835LED lamp beads with an illumination angle of 30 degrees and a lighting power of 0.2W, and the distribution spacing between the lighting beads is about 2mm.

[0096] Furthermore, to ensure uniformity of illumination, an adjustment circuit may be connected in an embodiment of the present invention to control the brightness of the lamp beads through the adjustment circuit, thereby achieving uniform illumination, that is, the two circuit boards in the sensor lighting circuit are connected to a voltage divider circuit board with an adjustable resistor, and the adjustable resistor is adjusted to change the brightness of the lamp beads. In other words, the lighting power of the lighting beads is adjusted by changing the resistance value of the adjustable resistor, thereby controlling the brightness of the lamp beads.

[0097] Then, step 230 is performed to prepare a gel module. The specific preparation process may be as follows: first, a gel layer may be prepared by using a gel mold and a gel support plate. The gel mold here is a two-piece mold, including an upper gel mold and a lower gel mold. Figure 3 Schematic diagram of the gel mold and gel support plate provided by the present invention, such as Figure 3 As shown, the left side is the upper gel mold, the right side is the lower gel mold; the middle is the gel support plate, all three are printed parts, and are hemispherical, through which the gel layer can be prepared;

[0098] The outer surface of the prepared gel layer is provided with a plurality of pits, and pigments are injected into the pits. After static drying, a plurality of marking points distributed on the outer surface of the gel layer can be obtained. After that, a coating liquid can be sprayed on the outer surface of the gel layer. After drying, a coating can be obtained. The coating is attached to the outer surface of the gel layer. The coating liquid here is a mixture of silicone adhesive, gray silicone pigment and diluent. In short, the prepared gel mold includes a gel layer and a coating attached to the outer surface of the gel layer, and a plurality of marking points are distributed on the gel layer.

[0099] Specifically, in the embodiment of the present invention, the preparation process of the marking point can be: first, use a syringe to load diluted colored acrylic paint (such as black acrylic paint), and then inject the paint into multiple pits with a diameter of 0.3mm and a depth of 0.6mm evenly distributed on the outer surface of the gel layer, and let it stand and dry to obtain multiple evenly distributed marking points. The marking points here can be divided into central marking points and outer circle marking points, wherein the outer circle marking points are evenly distributed on multiple concentric circles with the central marking point as the center, and the number of concentric circles is determined according to the size of the gel layer, and the number of outer circle marking points distributed on each concentric circle can be set according to actual needs. Preferably, the number of concentric circles in the embodiment of the present invention is 11, and the number of outer circle marking points on each concentric circle from the inside to the outside with the central marking point as the center is 6, 12, 18, 24, 30, 36, 42, 48, 54, 60 and 66 respectively.

[0100] The coating preparation process can be as follows: first, the silicone adhesive, gray silicone pigment and diluent are mixed in a certain proportion (such as 1:1:3) to obtain a coating liquid, and then the coating liquid can be sprayed on the outer surface of the gel layer with the marked points, and the coating can be obtained by standing and drying.

[0101] Thereafter, step 240 is executed to calibrate the pose relationship between the sensor frame coordinate system and the left camera coordinate system. Here, the sensor frame coordinate system is a coordinate system constructed with the sensor frame as the origin, and the left camera coordinate system is a coordinate system constructed with the optical center of the left camera as the origin. Here, the pose calibration process between the two coordinate systems can be performed according to binary code, that is, the pose relationship between the sensor frame coordinate system and the left camera coordinate system can be determined with the help of binary code (ArUco code), and pose calibration can be performed.

[0102] Finally, step 250 is performed to integrate the components prepared through the above steps, that is, the binocular camera, the sensor frame and the gel module are assembled by fasteners (such as screws and nuts) to obtain a tactile sensor.

[0103] When the prepared tactile sensor touches an object, the gel layer undergoes geometric deformation, and the marking points attached to the outer surface of the gel layer are displaced. The three-dimensional position of the marking points can be calculated through the displacement of the marking points, thereby realizing the extraction of tactile information from the three-dimensional point cloud.

[0104] The preparation method of the tactile sensor provided by the present invention can ensure that the prepared tactile sensor has a curved contact surface, can sense three-dimensional geometric information, has extremely high precision, and the preparation process is low-cost and simple to operate. The prepared tactile sensor has a wide range of applications, especially in robot scenarios, which is crucial for perception and dexterous operation tasks.

[0105] Based on the above embodiment, the gel mold includes a bottom gel mold and an upper gel mold, and the gel mold and the gel support plate are obtained by three-dimensional printing through transparent photosensitive resin;

[0106] In step 230, a gel layer is prepared based on the gel mold and the gel support plate, including:

[0107] Select transparent two-component silicone rubber, mix the first component and the second component of the two-component silicone rubber in a ratio of 1:1 to obtain a silicone rubber mixed liquid;

[0108] Pour the silica gel mixture into the bottom gel mold, place the gel support plate on the bottom gel mold with the silica gel mixture, and pour the silica gel mixture into the gel support plate until it covers the gel support plate;

[0109] The upper gel mold is placed on the gel support plate, and the bottom gel mold, the gel support plate and the upper gel mold are fixed with fasteners to obtain a fixture, and the gel layer is taken out after the silica gel mixture in the fixture is allowed to stand and solidify.

[0110] Specifically, in step 230, before preparing the gel layer through the gel mold and the gel support plate, it is first necessary to prepare the gel mold and the gel support plate. The gel mold and the gel support plate here can be made by three-dimensional printing of transparent photosensitive resin. The finished gel mold and gel support plate are hemispherical, and their outer surfaces are curved. Among them, the gel mold can be divided into a bottom gel mold and an upper gel mold, and the process of preparing the gel layer through the bottom gel mold, the upper gel mold and the gel support plate can specifically include the following steps:

[0111] First, a material for making the gel layer needs to be selected. Here, a transparent two-component silicone can be selected. Then, the first component and the second component of the two-component silicone can be mixed in a certain ratio to obtain a silicone mixed liquid. Specifically, in the embodiment of the present invention, a two-component silicone with a hardness of 45 Shore 00 is selected, and the first component and the second component of the silicone are mixed in a ratio of 1:1 to obtain a silicone mixed liquid.

[0112] Then, the gel layer can be prepared. Figure 4 is a schematic diagram of the preparation process of the gel layer provided by the present invention, such as Figure 4As shown, the silica gel mixture can be poured into the bottom gel mold in the gel mold until it covers one-third of the height of the bottom gel mold, and then the gel support plate is placed on the bottom gel mold containing the silica gel mixture, and then the silica gel mixture is continued to be poured into the gel support plate until it covers two-thirds of the height of the gel support plate, and then the upper gel mold in the gel mold can be placed on the gel support plate, and finally fasteners (screws and nuts) are used to fix the bottom gel mold, the gel support plate and the upper gel mold to obtain a fixing part, and the gel layer can be taken out after the silica gel mixture in the fixing part is left to solidify. It is worth noting that the gel support plate is wrapped in the gel layer, that is, the prepared gel layer is wrapped with the gel support plate.

[0113] Based on the above embodiment, step 240 includes:

[0114] Perform three-dimensional printing based on an opaque photosensitive resin to obtain a three-dimensional printed part, paste a binary code on the three-dimensional printed part, install the three-dimensional printed part on a sensor frame, and determine the relative position between the three-dimensional printed part and the sensor frame coordinate system;

[0115] The left camera is used to capture the binary code on the three-dimensional printed part to obtain a binary image, and the relative position and posture between the three-dimensional printed part and the left camera coordinate system are determined based on the binary image;

[0116] Based on the relative pose between the three-dimensional print and the sensor frame coordinate system, and the relative pose between the three-dimensional print and the left camera coordinate system, the pose relationship between the sensor frame coordinate system and the left camera coordinate system is determined.

[0117] Specifically, in step 240, the process of calibrating the position and posture relationship between the sensor frame coordinate system and the left camera coordinate system by using binary code may include the following steps:

[0118] First, determine the binary code (ArUco code), which is a square code consisting of only black and white. Figure 5 is a schematic diagram of the binary code provided by the present invention, such as Figure 5 As shown, in the embodiment of the present invention, the black part is replaced by stripes for the convenience of display, and then the binary code can be pasted on the three-dimensional print. The three-dimensional print here can be obtained by three-dimensional printing with non-transparent photosensitive resin (white photosensitive resin). The shape of the three-dimensional print can be set according to the actual situation, for example, it can be square, round, etc., and then the three-dimensional print is installed on the sensor frame;

[0119] Then, the left camera in the binocular camera can be used to capture the binary code to obtain a binary image, and then the binary image can be used for calculation to determine the position and posture of the three-dimensional printed part relative to the left camera, that is, to calculate the relative position and posture between the three-dimensional printed part and the left camera coordinate system;

[0120] At the same time, the pose of the 3D printed part relative to the sensor frame, that is, the relative pose between the 3D printed part and the sensor frame coordinate system, can be calculated through the 3D models of the 3D printed part and the sensor frame;

[0121] After that, the posture relationship between the sensor frame coordinate system and the left camera coordinate system can be converted through the relative posture between the three-dimensional print and the sensor frame coordinate system, as well as the relative posture between the three-dimensional print and the left camera coordinate system. That is, the posture relationship can be converted based on the relative posture between the three-dimensional print and the sensor frame coordinate system, as well as the relative posture between the three-dimensional print and the left camera coordinate system, to obtain the posture relationship between the sensor frame coordinate system and the left camera coordinate system.

[0122] The present invention also provides a point cloud reconstruction method based on a tactile sensor. Figure 6 is a flow chart of a point cloud reconstruction method based on a tactile sensor provided by the present invention, such as Figure 6 As shown, the method includes:

[0123] Step 610, obtaining a left-eye image and a right-eye image captured by a binocular camera of the tactile sensor;

[0124] Step 620, matching the left target mark point in the left image with the right target mark point in the right image, and determining the pixel coordinates of the mark points corresponding to the two-dimensional mark points to be reconstructed in the left image and the right image based on the matching relationship between the two target mark points obtained by matching, wherein the mark points include the left two-dimensional mark points and the right two-dimensional mark points;

[0125] Step 630, based on the pixel coordinates of the marking point, determine the pseudo point corresponding to the two-dimensional marking point in the gel layer of the tactile sensor and the coordinates of the pseudo point in the gel layer coordinate system, and determine the left eye incident light and the right eye incident light based on the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system and the inner spherical radius of the gel layer; the pseudo point is the intersection of the straight line where the left eye refracted light corresponding to the left eye incident light is located and the straight line where the right eye refracted light corresponding to the right eye incident light is located;

[0126] Step 640, based on the left eye incident light and the right eye incident light, and the pose transformation matrix between the gel layer coordinate system and the sensor frame coordinate system, determine the three-dimensional point cloud of the two-dimensional marker point in the sensor frame coordinate system.

[0127] Specifically, before performing point cloud reconstruction on the elastic contact surface of the tactile sensor, it is first necessary to obtain the tactile image captured by the binocular camera of the tactile sensor, where the tactile image can be divided into a left-eye image and a right-eye image; and after obtaining the left-eye image and the right-eye image, it is necessary to detect the marking points thereon to obtain the left target points in the left-eye image and the right target points in the right-eye image, that is, based on the left-eye image and the right-eye image, perform marking point detection to obtain the left target points in the left-eye image and the right target points in the right-eye image;

[0128] Among them, for the marker point detection process of the left eye image and the right eye image, specifically, the marker point detection can be performed by using an image preprocessing method and a spot detection method, that is, a circular mask is used to retain the effective marker point area in the left eye image and the right eye image captured by the binocular camera when the tactile sensor is not in contact, and then two Gaussian kernels of different sizes can be used to perform Gaussian filtering on the left eye image and the right eye image respectively to obtain two filtering results, and the two filtering results are subtracted to remove background noise; the sizes of the two Gaussian kernels here can be selected accordingly according to actual conditions, for example, they can be 3×3 and 51×51;

[0129] Then, the left and right images after removing the background noise can be binarized to obtain the preprocessed image. Finally, the blob detection tool of OpenCV can be used to perform blob detection on the preprocessed image to obtain the center position of the blob from the preprocessed image. Finally, each marking point in the left image, i.e., the left target mark point, and each marking point in the right image, i.e., the right target mark point, can be obtained.

[0130] After obtaining the marked points in the left / right image, the left target points in the left image and the right target points in the right image can be matched to determine the matching relationship between the two target points. Specifically, image judgment can be first performed, that is, whether the left image and the right image are the first frame images (first frame images) captured by the binocular camera. Further, when the left image and the right image are the first frame images captured by the binocular camera, the left target points in the left image and the right target points in the right image can be grouped and sorted respectively to determine the sequence numbers of each left target point and each right target point, and the left target points and the right target points are matched according to the sequence numbers, that is, the two marked points with the same sequence numbers in the left target points and the right target points are matched, so that the matching relationship between the left target points and the right target points, that is, the matching relationship between the two target points, can be obtained.

[0131] Correspondingly, in the case where the left-eye image and the right-eye image are not the first frame images captured by the binocular camera, marker point tracking can be performed, that is, the marker points in the left-eye image and the right-eye image are tracked through the previous and next frame images captured by the binocular camera, and the matching relationship between the two is maintained accordingly.

[0132] Furthermore, after obtaining the matching relationship between the two target marking points, it is necessary to use this matching relationship to determine the left-eye two-dimensional marking point and the right-eye two-dimensional marking point corresponding to the two-dimensional marking point to be reconstructed in the left-eye image and the right-eye image respectively, and determine the pixel coordinates of the two. Based on the pixel coordinates of the left-eye two-dimensional marking point and the pixel coordinates of the right-eye two-dimensional marking point, the triangulation principle can be used to calculate the pseudo point corresponding to the two-dimensional marking point to be reconstructed in the gel layer of the tactile sensor, and determine the coordinates of the pseudo point in the gel layer coordinate system. In short, the pseudo point is calculated by the triangulation principle without considering the refraction of light. In essence, it is the intersection of the straight line where the left-eye refracted light corresponding to the left-eye incident light is located and the straight line where the right-eye refracted light corresponding to the right-eye incident light is located.

[0133] After that, it is necessary to calculate the left eye incident light and the right eye incident light with the help of the coordinates of the pseudo point in the gel layer coordinate system. Specifically, first, the intersection of the left eye refracted light and the refraction interface, as well as the intersection of the right eye refracted light and the refraction interface, are determined through the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system, as well as the inner spherical radius of the gel layer. At the same time, the coordinates of the two intersection points in the gel layer coordinate system need to be determined. It is worth noting that the intersection point here can be called the refraction point, and the refraction interface is the inner spherical surface of the gel layer. Then, with the help of the law of refraction, based on the coordinates of the refraction point and the optical center of the binocular camera in the gel layer coordinate system, as well as the inner spherical radius of the gel layer, the left eye incident light corresponding to the left eye refracted light and the right eye incident light corresponding to the right eye refracted light can be calculated.

[0134] After that, the three-dimensional point cloud of the two-dimensional marker point in the sensor frame coordinate system can be solved with the help of the left-eye incident light and the right-eye incident light, as well as the pose transformation matrix between the gel layer coordinate system and the sensor frame coordinate system. Specifically, the point closest to the left-eye incident light and the right-eye incident light is calculated. This point is the required three-dimensional point. Since the coordinates of the point are relative to the gel layer coordinate system at this time, the coordinates in the gel layer coordinate system need to be converted into coordinates in the sensor frame coordinate system through the pose transformation matrix between the gel layer coordinate system and the sensor frame coordinate system. In this way, the three-dimensional point cloud of the two-dimensional marker point to be reconstructed in the sensor frame coordinate system is obtained.

[0135] The point cloud reconstruction method based on the tactile sensor provided by the present invention determines the pixel coordinates of the marking points corresponding to the two-dimensional marking points to be reconstructed in the left eye image and the right eye image according to the matching relationship between the left target marking points and the right target marking points, and determines the pseudo points corresponding to the two-dimensional marking points in the gel layer of the tactile sensor and the coordinates of the pseudo points in the gel layer coordinate system according to the pixel coordinates of the marking points, and determines the left eye incident light and the right eye incident light according to the coordinates of the pseudo points and the optical center of the binocular camera in the gel layer coordinate system and the inner spherical radius of the gel layer; determines the three-dimensional point cloud of the two-dimensional marking points in the sensor frame coordinate system according to the left eye incident light and the right eye incident light and the posture transformation matrix between the gel layer coordinate system and the sensor frame coordinate system, overcomes the defect of large point cloud reconstruction error in traditional schemes, can solve the problem of difficulty in point cloud reconstruction in binocular vision caused by refraction of light through multiple media without relying on complex calibration processes and high-precision calibration equipment, can perform point cloud reconstruction in real time, and at the same time, achieves dual improvement in point cloud reconstruction accuracy and efficiency.

[0136] Based on the above embodiment, in step 630, based on the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system, and the inner spherical radius of the gel layer, the left eye incident light and the right eye incident light are determined, including:

[0137] Based on the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system, and the radius of the inner sphere of the gel layer, determine the coordinates of the refraction point of the left eye refracted light and the refraction point of the right eye refracted light in the gel layer coordinate system, the refraction point being the intersection of the corresponding refracted light and the inner sphere of the gel layer;

[0138] Based on the coordinates of the refraction point in the gel layer coordinate system, the coordinates of the optical center of the binocular camera in the gel layer coordinate system, the refractive index of the medium in the gel layer, the refractive index of air, and the inner spherical radius of the gel layer, determine the left eye incident light and the right eye incident light corresponding to the left eye refracted light and the right eye refracted light respectively.

[0139] Specifically, in step 630, the process of determining the left eye incident light and the right eye incident light according to the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system and the inner spherical radius of the gel layer may specifically include the following steps:

[0140] First, the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system, and the radius of the inner sphere of the gel layer, can be used to determine the refraction points of the left-eye refracted light and the right-eye refracted light. Specifically, since the pseudo point is the intersection of the left-eye refracted light and the right-eye refracted light, the refraction point can be solved by the line between the pseudo point and the optical center of the binocular camera, and the equation of the refraction interface (inner sphere), that is, the line between the pseudo point and the optical center of the binocular camera can be calculated by using the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system. Since the radius of the inner sphere of the gel layer is known, the equation of the inner sphere can be directly determined. By combining the equation of the straight line of the connecting line and the equation of the inner sphere, the coordinates of the two intersection points of the connecting line and the inner sphere, that is, the coordinates of the refraction point of the left-eye refracted light in the gel layer coordinate system, and the coordinates of the refraction point of the right-eye refracted light in the gel layer coordinate system, can be directly obtained.

[0141] Then, with the help of the law of refraction, the left eye incident light and the right eye incident light can be calculated. Specifically, based on the coordinates of the refraction point in the gel layer coordinate system, the coordinates of the optical center of the binocular camera in the gel layer coordinate system, the refractive index of the medium in the gel layer, the refractive index of air, and the inner spherical radius of the gel layer, the law of refraction stipulates that when the refractive index of the medium is fixed, the ratio of the sine of the incident angle to the sine of the refraction angle is a constant, and the law of refraction stipulates that the refracted light is located in the plane determined by the incident light and the normal of the refraction interface, the left eye incident light and the right eye incident light corresponding to the left eye refracted light and the right eye refracted light are calculated respectively.

[0142] Based on the above embodiment, in step 620, based on the pixel coordinates of the marking point, determining the pseudo point corresponding to the two-dimensional marking point in the gel layer of the tactile sensor and the coordinates of the pseudo point in the gel layer coordinate system includes:

[0143] Based on the pixel coordinates of the marker point, determine the pseudo point corresponding to the two-dimensional marker point in the gel layer of the tactile sensor, and the coordinates of the pseudo point in the left camera coordinate system;

[0144] Based on the coordinates of the pseudo point in the left camera coordinate system and the pose transformation matrix between the gel layer coordinate system and the left camera coordinate system, the coordinates of the pseudo point in the gel layer coordinate system are determined.

[0145] Specifically, in step 620, the process of determining the pseudo point corresponding to the two-dimensional marking point in the gel layer of the tactile sensor and the coordinates of the pseudo point in the gel layer coordinate system through the pixel coordinates of the marking point may include the following two steps:

[0146] Firstly, the pixel coordinates of the left-eye 2D marker point and the pixel coordinates of the right-eye 2D marker point can be used as references to calculate the pseudo point corresponding to the 2D marker point to be reconstructed in the gel layer of the tactile sensor and the coordinates of the pseudo point in the left camera coordinate system through the triangulation principle;

[0147] Then, the coordinates of the pseudo point in the left camera coordinate system need to be converted into the coordinates of the pseudo point in the gel layer coordinate system. That is, the pose conversion matrix between the gel layer coordinate system and the left camera coordinate system can be used to solve the coordinates of the pseudo point in the gel layer coordinate system.

[0148] Based on the above embodiments, Figure 7 is an abstract diagram of the point cloud reconstruction process provided by the present invention, such as Figure 7 As shown, {H} is the coordinate system of the hemispherical gel layer (gel layer coordinate system), where the center of the sphere is H O is the origin of {H}, Z axis H Z passes through the center mark point on the sphere; {C} is the left camera coordinate system; {S} is the sensor frame coordinate system.

[0149] H L(l x , l y , l z )and H R(r x , r y , r z ) are the coordinates of the optical center of the left camera and the optical center of the right camera under {H} respectively. H P(p x , p y , p z ) is a point on the contact surface of the gel layer, i.e., a contact point of the contact surface of the tactile sensor; H P′(p′ x , p′ y , p′ z )for H P corresponds to a pseudo point in the gel layer of the tactile sensor, which can be calculated by the triangulation principle without considering the refraction of light.

[0150] H A(a x , a y , a z )and H B(b x , b y , b z ) is the intersection point of the light entering the left camera and the right camera with the inner sphere, that is, the refraction point of the left eye refracted light and the refraction point of the right eye refracted light. and They are the normals at points HA and HB on the refractive interface (inner sphere) respectively. and are the incident light for the left eye and the incident light for the right eye, respectively. and are the refracted light of the left eye and the refracted light of the right eye respectively. β and α are the incident angle and refraction angle respectively; r is the inner spherical radius of the gel layer; n1 and n2 are the refractive indices of air and silicone mixture respectively.

[0151] H p′ is a straight line H AL and H The intersection of BR. According to the law of refraction, the refracted light is located in the plane determined by the incident light and the normal of the refractive interface, that is, Located by and In the plane determined by, it can be known that H P, H A, H L, H O is coplanar, and the same logic can be proved H P, H B, H R, H O is coplanar. H L, H R, r, n1, n2 and the camera intrinsic parameters of the binocular camera and the pose relationship between the binocular cameras (pose transformation matrix), the 3D point is reconstructed from the pixel coordinates of the left and right 2D markers in the left and right images. H The process of P may include the following steps:

[0152] First, the pseudo point can be calculated based on the pixel coordinates of the left 2D marker point and the pixel coordinates of the right 2D marker point using the triangulation principle. H Coordinates of P′ in the left camera coordinate system C P′(p′ x，c , p′ y，c , p′ z，c ), and with the help of the pose transformation matrix between the gel layer coordinate system and the left camera coordinate system, the pseudo point coordinates in the left camera coordinate system are transformed into the pseudo point coordinates in the gel layer coordinate system, thereby obtaining H P′(p′ x , p′ y , p′ z ), the calculation formula is as follows:

[0153]

[0154] In the formula, represents the pose transformation matrix (homogeneous pose transformation matrix) between {H} and {C}, [p′ x , p′ y , p′ z , 1] T and [p′x，c , p′ y，c , p′ z，c , 1] T Respectively represent [p′ x , p′ y , p′ z , 1] and [p′ x，c , p′ y，c , p′ z，c , the transpose of 1].

[0155] Then, the coordinates of the pseudo point in the gel layer coordinate system can be used to calculate the intersection of the refracted light and the refractive interface, that is, the coordinates of the refraction point of the left eye refracted light and the right eye refracted light in the gel layer coordinate system. Specifically, the line between the pseudo point and the optical center of the binocular camera is first calculated. Since the equation of the inner sphere is known, the equation of the straight line of the connecting line and the equation of the inner sphere can be combined to obtain the two intersection points.

[0156] In this embodiment of the invention, the two intersection points are H A and H B, due to H Point A is located at H above the XOY plane, and H A is H The intersection of LP′ and the inner sphere, so H A(a x , a y , a z ) can be calculated by the following formula:

[0157]

[0158] Similarly, we can obtain H B(b x , b y , b z ).

[0159] Then, the left eye incident light and the right eye incident light can be calculated by the law of refraction. Specifically, let the unit direction vector of the left eye incident light be The unit direction vector of the incident light from the right eye is flat H The unit normal vector of LOP is because H P, H A, H L, H Oh, H P′ is coplanar, so the unit direction vector can be determined With the unit normal vector vertical, so we can get and in You can do this by H L(l x , l y , l z )and H A(a x , a y , a z ) is substituted into the equation of plane HLOP to obtain;

[0160] At the same time, the law of refraction indicates that when the refractive index is fixed, the ratio of the sine of the incident angle to the sine of the refraction angle is a constant, so it can be determined According to the trigonometric identities and the dot product operation of vectors, we can get as well as and

[0161] Combining the above formulas, we can get Similarly, we can obtain

[0162] After that, the point closest to the left eye incident light and the right eye incident light can be calculated. This point is the desired three-dimensional point. Specifically, the least squares method can be used to calculate the point closest to the two incident light rays.

[0163] In the embodiment of the present invention, the point closest to the incident light of the left eye and the incident light of the right eye is the intersection of the two, that is, H P(p x , p y , p z )for H PA and H The intersection of PB is:

[0164]

[0165] Where, t a and t b They are and

[0166] The above H The expression of P is converted into matrix form:

[0167]

[0168] This is an overdetermined system of equations with three equations and two unknowns, so it can be solved by the least squares method:

[0169]

[0170]

[0171] set up However, due to errors in the actual solution process, the incident light from the left eye and the incident light from the right eye are very likely not to intersect, resulting in H P a ≠ H P b ,therefore H P can be calculated by the following formula:

[0172]

[0173] Finally, the pose transformation matrix between the gel layer coordinate system and the sensor frame coordinate system can be used Obtain H The coordinates of P under {S} s P can also be called the three-dimensional point cloud of the two-dimensional marker points in the sensor frame coordinate system.

[0174] Based on the above embodiment, in step 620, matching the left target mark in the left image with the right target mark in the right image includes:

[0175] When the left-eye image and the right-eye image are first-frame images captured by the binocular camera, the left-eye image and the right-eye image are corrected, and a left-eye polar coordinate system and a right-eye polar coordinate system are established based on the corrected left-eye image and the right-eye image;

[0176] Based on the polar coordinates of the left target mark in the left eye polar coordinate system and the polar coordinates of the right target mark in the right eye polar coordinate system, the mark points are sorted to obtain the sequence number of the left target mark and the sequence number of the right target mark;

[0177] Based on the sequence number of the left target point and the sequence number of the right target point, the left target point and the right target point are matched.

[0178] Specifically, in step 620, the process of matching the left target point in the left image with the right target point in the right image may include the following steps:

[0179] First, determine whether the left-eye image and the right-eye image are the first frame images captured by the binocular camera, and when the left-eye image and the right-eye image are the first frame images captured by the binocular camera, use perspective transformation to correct the left-eye image and the right-eye image so that the concentric circles are perfect circles, thereby obtaining the corrected left-eye image and the right-eye image. Specifically, the perspective transformation matrix can be calculated through the position of the central marking point and the positions of the marking points on the four diagonals of the outermost circle, and then the perspective transformation tool is used to perform perspective transformation on the left-eye image and the right-eye image, thereby obtaining the images after perspective transformation, that is, the corrected left-eye image and the right-eye image;

[0180] Then, the left eye polar coordinate system and the right eye polar coordinate system can be established with the central marking point in the corrected left eye image and the right eye image as the origin. In the polar coordinate system, the position of any marking point i can be expressed as (ρ i ,θ i ), where ρ i is the polar diameter, indicating the distance from the marked point i to the pole; θ i is the polar angle, which represents the angle between the line from the marked point i to the pole and the polar axis;

[0181] Subsequently, the polar coordinates of the left target mark point in the left eye polar coordinate system and the polar coordinates of the right target mark point in the right eye polar coordinate system can be used as references to sort the left target mark points and the right target mark points respectively to determine the serial numbers of the left target mark points and the right target mark points. Specifically, the left target mark points can be sorted in the order of polar diameter first and then polar angle, that is, the mark points can be sorted in the order of polar diameter from small to large or from large to small, and for the mark points with the same polar diameter, that is, the mark points on the same polar diameter circle, they can be sorted in the order of polar angle from small to large or from large to small. Finally, the serial numbers of all the left target mark points can be obtained, and the right target mark points can be sorted in the same way to obtain the serial numbers of the right target mark points;

[0182] After that, the left target point and the right target point can be matched with reference to their serial numbers. Specifically, the two marking points with the same serial numbers in the left target point and the right target point can be matched to determine the matching relationship between the left target point and the right target point, that is, the matching relationship between the two target points.

[0183] Based on the above embodiment, in step 620, based on the polar coordinates of the left target mark in the left eye polar coordinate system and the polar coordinates of the right target mark in the right eye polar coordinate system, the mark points are sorted to obtain the sequence numbers of the left target mark and the right target mark, including:

[0184] The marking points are grouped based on the polar diameter in the polar coordinates of the left target mark point to obtain the left target mark point in each polar diameter circle, and the target mark points are grouped based on the polar diameter in the polar coordinates of the right target mark point to obtain the right target mark point in each polar diameter circle;

[0185] Based on the polar angle in the polar coordinates of the left target point, the left target points in the same polar circle are sorted to obtain the serial number of the left target point, and based on the polar angle in the polar coordinates of the right target point, the right target points in the same polar circle are sorted to obtain the serial number of the right target point.

[0186] Specifically, in the above process, the marking points are sorted according to the polar coordinates of the left target mark points in the left eye polar coordinate system and the polar coordinates of the right target mark points in the right eye polar coordinate system to obtain the sequence numbers of the left target mark points and the right target mark points. The process can be divided into two steps, namely:

[0187] First, the left target mark point and the right target mark point can be grouped respectively based on the polar diameter in the polar coordinates of the left target mark point and the polar diameter in the polar coordinates of the right target mark point, so as to obtain the left target mark point in each polar diameter circle and the right target mark point in each polar diameter circle. Specifically, the marking points can be grouped with reference to the polar diameter in the polar coordinates of the left target mark point to obtain the left target mark point in each polar diameter circle, that is, the left target mark points are grouped in the order of polar diameter from small to large or from large to small, so as to obtain the left target mark point in each polar diameter circle. At the same time, the right target mark points can be grouped in the order of polar diameter in the polar coordinates of the right target mark point from small to large or from large to small, so as to obtain the right target mark point in each polar diameter circle.

[0188] Subsequently, the left target points in the same polar circle and the right target points in the same polar circle can be sorted according to the polar angle in the polar coordinates of the left target point and the polar angle in the polar coordinates of the right target point, respectively, to obtain the serial numbers of the left target points and the right target points, that is, the left target points in the same polar circle can be sorted with reference to the polar angle in the polar coordinates of the left target point, and the right target points in the same polar circle can be sorted with reference to the polar angle in the polar coordinates of the right target point. Specifically, for the marking points with the same polar diameter, that is, the marking points in the same polar circle, they can be sorted in the order of polar angle from small to large or from large to small to determine the serial numbers of each marking point in the polar circle. After sorting the marking points in each polar circle, the serial numbers of all left target points can be obtained to obtain the serial numbers of all right target points.

[0189] Based on the above embodiment, in step 620, matching the left target mark in the left image with the right target mark in the right image includes:

[0190] In the case that the left eye image and the right eye image are not the first frame images captured by the binocular camera, based on the previous left eye image and the previous right eye image, the left target point in the left eye image and the right target point in the right eye image are tracked, and the matching relationship between the two target points is determined.

[0191] Specifically, in step 620, the process of matching the left target mark in the left eye image and the right target mark in the right eye image can also be to track the marked points when it is determined that the left eye image and the right eye image are not the first frame images captured by the binocular camera, that is, the nearest neighbor matching algorithm can be used to match the marked points in the previous and next frame images, so as to achieve tracking of the marked points. Specifically, the last frame of binocular images captured by the binocular camera, that is, the last frame of the left eye image and the last frame of the right eye image, are used to track the left target mark in the left eye image and the right target mark in the right eye image, and determine the matching relationship between the two, that is, the matching relationship between the two target marks can be maintained by tracking the marked points.

[0192] The following is a description of a device for preparing a tactile sensor provided by the present invention. The device for preparing a tactile sensor described below and the method for preparing a tactile sensor described above can refer to each other.

[0193] Figure 8 is a schematic diagram of the structure of the tactile sensor manufacturing device provided by the present invention, such as Figure 8 As shown, the device comprises:

[0194] A parameter calibration unit 810 is used to calibrate the camera internal parameters of the binocular camera and the position relationship between the left camera and the right camera in the binocular camera, wherein the camera internal parameters include at least one of the camera size, camera viewing angle, baseline distance, distortion condition, shooting frame rate, and imaging size;

[0195] The frame preparation unit 820 is used to perform three-dimensional printing based on photosensitive resin to obtain a sensor support frame, obtain a ring-shaped sensor lighting circuit based on two semi-ring-shaped circuit boards, and assemble a sensor frame based on the sensor support frame and the sensor lighting circuit;

[0196] The gel preparation unit 830 is used to prepare a gel layer based on a gel mold and a gel support plate, inject pigment into pits on the outer surface of the gel layer, dry to obtain marking points distributed on the outer surface of the gel layer, spray a coating liquid on the outer surface of the gel layer with the marking points, dry to obtain a coating, and form a gel module based on the gel layer and the coating, wherein the coating liquid is obtained by mixing a silicone adhesive, a gray silicone pigment and a diluent;

[0197] A posture calibration unit 840, used for calibrating the posture relationship between the sensor frame coordinate system and the left camera coordinate system based on binary code, wherein the sensor frame coordinate system is constructed with the sensor frame as the origin, and the left camera coordinate system is constructed with the left camera in the binocular camera as the origin;

[0198] The component integration unit 850 is used to assemble the binocular camera, the sensor frame and the gel module to obtain the tactile sensor.

[0199] The tactile sensor preparation device provided by the present invention can ensure that the prepared tactile sensor has a curved contact surface, can sense three-dimensional geometric information, has extremely high precision, and the preparation process is low-cost and simple to operate. The prepared tactile sensor has a wide range of applications, especially in robot scenarios, which is crucial for perception and dexterous operation tasks.

[0200] Based on the above embodiment, the gel mold includes a bottom gel mold and an upper gel mold, and the gel mold and the gel support plate are obtained by three-dimensional printing through transparent photosensitive resin; the gel preparation unit 830 is used for:

[0201] Selecting a transparent two-component silica gel, mixing the first component and the second component of the two-component silica gel to obtain a silica gel mixed liquid;

[0202] Pour the silica gel mixture into the bottom gel mold, place the gel support plate on the bottom gel mold with the silica gel mixture, and pour the silica gel mixture into the gel support plate until it covers the gel support plate;

[0203] The upper gel mold is placed on the gel support plate, and the bottom gel mold, the gel support plate and the upper gel mold are fixed with fasteners to obtain a fixing part, and the gel layer is taken out after the silica gel mixture in the fixing part is left to solidify.

[0204] Based on the above embodiment, the posture calibration unit 840 is used for:

[0205] Performing three-dimensional printing based on an opaque photosensitive resin to obtain a three-dimensional printed part, pasting a binary code on the three-dimensional printed part, installing the three-dimensional printed part on the sensor frame, and determining a relative position between the three-dimensional printed part and a sensor frame coordinate system;

[0206] Shooting the binary code on the three-dimensional printed part by the left camera to obtain a binary image, and determining the relative position and posture between the three-dimensional printed part and the left camera coordinate system based on the binary image;

[0207] Based on the relative posture between the three-dimensional print and the sensor frame coordinate system, and the relative posture between the three-dimensional print and the left camera coordinate system, the posture relationship between the sensor frame coordinate system and the left camera coordinate system is determined.

[0208] The point cloud reconstruction device based on a tactile sensor provided by the present invention is described below. The point cloud reconstruction device based on a tactile sensor described below and the point cloud reconstruction method based on a tactile sensor described above can refer to each other.

[0209] Fig. 9 is a schematic diagram of the structure of a point cloud reconstruction device based on a tactile sensor provided by the present invention, such as Fig. 9 As shown, the device comprises:

[0210] An image acquisition unit 910 is used to acquire a left-eye image and a right-eye image captured by a binocular camera of a tactile sensor;

[0211] The point cloud reconstruction unit 920 is used to match the left target mark point in the left image with the right target mark point in the right image, and determine the pixel coordinates of the mark points corresponding to the two-dimensional mark points to be reconstructed in the left image and the right image based on the matching relationship between the two target mark points obtained by matching, wherein the mark points include the left two-dimensional mark points and the right two-dimensional mark points;

[0212] Based on the pixel coordinates of the marking point, determine the pseudo point corresponding to the two-dimensional marking point in the gel layer of the tactile sensor and the coordinates of the pseudo point in the gel layer coordinate system; based on the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system and the inner spherical radius of the gel layer, determine the left eye incident light and the right eye incident light; the pseudo point is the intersection of the straight line where the left eye refracted light corresponding to the left eye incident light is located and the straight line where the right eye refracted light corresponding to the right eye incident light is located;

[0213] Based on the left-eye incident light and the right-eye incident light, and the posture transformation matrix between the gel layer coordinate system and the sensor frame coordinate system, a three-dimensional point cloud of the two-dimensional marking point in the sensor frame coordinate system is determined.

[0214] The point cloud reconstruction device based on the tactile sensor provided by the present invention determines the pixel coordinates of the marking points corresponding to the two-dimensional marking points to be reconstructed in the left eye image and the right eye image according to the matching relationship between the left target marking points and the right target marking points, and determines the pseudo points corresponding to the two-dimensional marking points in the gel layer of the tactile sensor and the coordinates of the pseudo points in the gel layer coordinate system according to the pixel coordinates of the marking points, and determines the left eye incident light and the right eye incident light according to the coordinates of the pseudo points and the optical center of the binocular camera in the gel layer coordinate system and the inner spherical radius of the gel layer; determines the three-dimensional point cloud of the two-dimensional marking points in the sensor frame coordinate system according to the left eye incident light and the right eye incident light and the posture transformation matrix between the gel layer coordinate system and the sensor frame coordinate system, overcomes the defect of large point cloud reconstruction error in traditional schemes, can solve the problem of difficulty in point cloud reconstruction in binocular vision caused by refraction of light through multiple media without relying on complex calibration processes and high-precision calibration equipment, can perform point cloud reconstruction in real time, and at the same time, achieves double improvement in point cloud reconstruction accuracy and efficiency.

[0215] Based on the above embodiment, the point cloud reconstruction unit 920 is used for:

[0216] Based on the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system, and the radius of the inner spherical surface of the gel layer, determine the coordinates of the refraction point of the left eye refracted light and the refraction point of the right eye refracted light in the gel layer coordinate system, wherein the refraction point is the intersection of the corresponding refracted light and the inner spherical surface of the gel layer;

[0217] Based on the coordinates of the refraction point in the gel layer coordinate system, the coordinates of the optical center of the binocular camera in the gel layer coordinate system, the refractive index of the medium in the gel layer, the refractive index of air, and the inner spherical radius of the gel layer, determine the left eye incident light and the right eye incident light corresponding to the left eye refracted light and the right eye refracted light respectively.

[0218] Based on the above embodiment, the point cloud reconstruction unit 920 is used for:

[0219] Based on the pixel coordinates of the marking point, determining a pseudo point corresponding to the two-dimensional marking point in the gel layer of the tactile sensor, and the coordinates of the pseudo point in the left camera coordinate system;

[0220] Based on the coordinates of the pseudo point in the left camera coordinate system and the pose transformation matrix between the gel layer coordinate system and the left camera coordinate system, the coordinates of the pseudo point in the gel layer coordinate system are determined.

[0221] Based on the above embodiment, the point cloud reconstruction unit 920 is used for:

[0222] In the case where the left-eye image and the right-eye image are first frame images captured by the binocular camera, the left-eye image and the right-eye image are corrected, and a left-eye polar coordinate system and a right-eye polar coordinate system are established based on the corrected left-eye image and the right-eye image;

[0223] Based on the polar coordinates of the left target mark point in the left eye polar coordinate system and the polar coordinates of the right target mark point in the right eye polar coordinate system, sort the mark points to obtain the sequence number of the left target mark point and the sequence number of the right target mark point;

[0224] The left target point and the right target point are matched based on the sequence number of the left target point and the sequence number of the right target point.

[0225] Based on the above embodiment, the point cloud reconstruction unit 920 is used for:

[0226] Based on the polar diameter in the polar coordinates of the left target mark point, the marked points are grouped to obtain the left target mark point in each polar diameter circle, and based on the polar diameter in the polar coordinates of the right target mark point, the target mark points are grouped to obtain the right target mark point in each polar diameter circle;

[0227] Based on the polar angle in the polar coordinates of the left target point, the left target points in the same polar circle are sorted to obtain the serial number of the left target point, and based on the polar angle in the polar coordinates of the right target point, the right target points in the same polar circle are sorted to obtain the serial number of the right target point.

[0228] Based on the above embodiment, the point cloud reconstruction unit 920 is used for:

[0229] In the case that the left-eye image and the right-eye image are not the first frames of images captured by the binocular camera, based on the previous left-eye image and the previous right-eye image, the left target point in the left-eye image and the right target point in the right-eye image are tracked, and the matching relationship between the two target points is determined.

[0230] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the method for preparing a tactile sensor or the method for reconstructing a point cloud based on a tactile sensor, wherein the method for preparing a tactile sensor includes: calibrating the camera internal parameters of a binocular camera and the posture relationship between the left camera and the right camera in the binocular camera, wherein the camera internal parameters include at least one of the camera size, camera viewing angle, baseline distance, distortion condition, shooting frame rate and imaging size; performing three-dimensional printing based on photosensitive resin to obtain a sensor support frame, obtaining a ring-shaped sensor lighting circuit based on two semi-circular circuit boards, and assembling a sensor frame based on the sensor support frame and the sensor lighting circuit; preparing a gel mold based on a gel support plate The invention relates to a method for tactile sensing of a tactile sensor comprising: firstly, applying a coating liquid to the outer surface of the gel layer, and then injecting a pigment into the pits on the outer surface of the gel layer, and drying the pits to obtain marking points distributed on the outer surface of the gel layer, and then drying the coating liquid, and then forming a gel module based on the gel layer and the coating, wherein the coating liquid is obtained by mixing a silicone adhesive, a gray silicone pigment and a diluent; calibrating the position relationship between the sensor frame coordinate system and the left camera coordinate system based on the binary code, wherein the sensor frame coordinate system is constructed with the sensor frame as the origin, and the left camera coordinate system is constructed with the left camera in the binocular camera as the origin; and assembling the binocular camera, the sensor frame and the gel module to obtain the tactile sensor.The point cloud reconstruction method based on the tactile sensor includes: obtaining a left-eye image and a right-eye image collected by a binocular camera of the tactile sensor; matching a left target mark in the left-eye image with a right target mark in the right-eye image, and determining the pixel coordinates of the mark points corresponding to the two-dimensional mark points to be reconstructed in the left-eye image and the right-eye image based on the matching relationship between the two-target mark points obtained by matching, wherein the mark points include a left-eye two-dimensional mark point and a right-eye two-dimensional mark point; based on the pixel coordinates of the mark points, determining the pseudo points corresponding to the two-dimensional mark points in the gel layer of the tactile sensor, and The coordinates of the pseudo point in the gel layer coordinate system are determined based on the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system and the inner spherical radius of the gel layer to determine the left eye incident light and the right eye incident light; the pseudo point is the intersection of the straight line where the left eye refracted light corresponding to the left eye incident light and the straight line where the right eye refracted light corresponding to the right eye incident light; based on the left eye incident light and the right eye incident light, and the posture transformation matrix between the gel layer coordinate system and the sensor frame coordinate system, the three-dimensional point cloud of the two-dimensional marker point in the sensor frame coordinate system is determined.

[0231] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0232] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the preparation method of the tactile sensor or the point cloud reconstruction method based on the tactile sensor provided by the above methods, wherein the preparation method of the tactile sensor includes: calibrating the camera intrinsic parameters of the binocular camera and the posture relationship between the left camera and the right camera in the binocular camera, the camera intrinsic parameters including at least one of the camera size, camera viewing angle, baseline distance, distortion condition, shooting frame rate, and imaging size; performing three-dimensional printing based on photosensitive resin to obtain a sensor support frame, splicing two semi-circular circuit boards to obtain a ring-shaped sensor lighting circuit, and based on the sensor support A sensor frame is formed by using a supporting frame and the sensor lighting circuit; a gel layer is prepared based on a gel mold and a gel supporting plate, a pigment is injected into pits on the outer surface of the gel layer, and marking points distributed on the outer surface of the gel layer are obtained by drying, a coating liquid is sprayed on the outer surface of the gel layer with the marking points, and a coating is obtained by drying, and a gel module is formed based on the gel layer and the coating, wherein the coating liquid is obtained by mixing a silicone adhesive, a gray silicone pigment and a diluent; the position relationship between the sensor frame coordinate system and the left camera coordinate system is calibrated based on a binary code, the sensor frame coordinate system is constructed with the sensor frame as the origin, and the left camera coordinate system is constructed with the left camera in the binocular camera as the origin; the binocular camera, the sensor frame and the gel module are assembled to obtain the tactile sensor. The point cloud reconstruction method based on the tactile sensor includes: obtaining a left-eye image and a right-eye image collected by a binocular camera of the tactile sensor; matching a left target mark in the left-eye image with a right target mark in the right-eye image, and determining the pixel coordinates of the mark points corresponding to the two-dimensional mark points to be reconstructed in the left-eye image and the right-eye image based on the matching relationship between the two-target mark points obtained by matching, wherein the mark points include a left-eye two-dimensional mark point and a right-eye two-dimensional mark point; based on the pixel coordinates of the mark points, determining the pseudo points corresponding to the two-dimensional mark points in the gel layer of the tactile sensor, and The coordinates of the pseudo point in the gel layer coordinate system are determined based on the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system and the inner spherical radius of the gel layer to determine the left eye incident light and the right eye incident light; the pseudo point is the intersection of the straight line where the left eye refracted light corresponding to the left eye incident light and the straight line where the right eye refracted light corresponding to the right eye incident light; based on the left eye incident light and the right eye incident light, and the posture transformation matrix between the gel layer coordinate system and the sensor frame coordinate system, the three-dimensional point cloud of the two-dimensional marker point in the sensor frame coordinate system is determined.

[0233] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for preparing a tactile sensor or the point cloud reconstruction method based on the tactile sensor provided by the above-mentioned methods, wherein the method for preparing a tactile sensor includes: calibrating the camera intrinsic parameters of a binocular camera, and the posture relationship between a left camera and a right camera in the binocular camera, the camera intrinsic parameters including at least one of camera size, camera viewing angle, baseline distance, distortion condition, shooting frame rate, and imaging size; performing three-dimensional printing based on photosensitive resin to obtain a sensor support frame, splicing two semi-circular circuit boards to obtain a ring-shaped sensor lighting circuit, and assembling a sensor based on the sensor support frame and the sensor lighting circuit. A frame; preparing a gel layer based on a gel mold and a gel support plate, injecting a pigment into pits on the outer surface of the gel layer, drying to obtain marking points distributed on the outer surface of the gel layer, spraying a coating liquid on the outer surface of the gel layer with the marking points, drying to obtain a coating, and constructing a gel module based on the gel layer and the coating, wherein the coating liquid is obtained by mixing a silicone adhesive, a gray silicone pigment and a diluent; calibrating the position relationship between the sensor frame coordinate system and the left camera coordinate system based on a binary code, wherein the sensor frame coordinate system is constructed with the sensor frame as the origin, and the left camera coordinate system is constructed with the left camera in the binocular camera as the origin; assembling the binocular camera, the sensor frame and the gel module to obtain the tactile sensor. The point cloud reconstruction method based on the tactile sensor includes: obtaining a left-eye image and a right-eye image collected by a binocular camera of the tactile sensor; matching a left target mark in the left-eye image with a right target mark in the right-eye image, and determining the pixel coordinates of the mark points corresponding to the two-dimensional mark points to be reconstructed in the left-eye image and the right-eye image based on the matching relationship between the two-target mark points obtained by matching, wherein the mark points include a left-eye two-dimensional mark point and a right-eye two-dimensional mark point; based on the pixel coordinates of the mark points, determining the pseudo points corresponding to the two-dimensional mark points in the gel layer of the tactile sensor, and The coordinates of the pseudo point in the gel layer coordinate system are determined based on the coordinates of the pseudo point and the optical center of the binocular camera in the gel layer coordinate system and the inner spherical radius of the gel layer to determine the left eye incident light and the right eye incident light; the pseudo point is the intersection of the straight line where the left eye refracted light corresponding to the left eye incident light and the straight line where the right eye refracted light corresponding to the right eye incident light; based on the left eye incident light and the right eye incident light, and the posture transformation matrix between the gel layer coordinate system and the sensor frame coordinate system, the three-dimensional point cloud of the two-dimensional marker point in the sensor frame coordinate system is determined.

[0234] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0235] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0236] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for preparing a tactile sensor, characterized in that: include: Calibrate the camera intrinsic parameters of the binocular camera and the position relationship between the left camera and the right camera in the binocular camera, wherein the camera intrinsic parameters include at least one of the camera size, camera viewing angle, baseline distance, distortion condition, shooting frame rate, and imaging size; Performing three-dimensional printing based on photosensitive resin to obtain a sensor support frame, splicing two semi-circular circuit boards to obtain a ring-shaped sensor lighting circuit, and assembling a sensor frame based on the sensor support frame and the sensor lighting circuit; Prepare a gel layer based on a gel mold and a gel support plate, inject pigment into pits on the outer surface of the gel layer, dry to obtain marking points distributed on the outer surface of the gel layer, spray a coating liquid on the outer surface of the gel layer with the marking points, dry to obtain a coating, and assemble a gel module based on the gel layer and the coating, wherein the coating liquid is obtained by mixing a silicone adhesive, a gray silicone pigment and a diluent; Calibrate the position relationship between the sensor frame coordinate system and the left camera coordinate system based on the binary code, wherein the sensor frame coordinate system is constructed with the sensor frame as the origin, and the left camera coordinate system is constructed with the left camera in the binocular camera as the origin; Assembling the binocular camera, the sensor frame and the gel module to obtain the tactile sensor; The method of calibrating the position and posture relationship between the sensor frame coordinate system and the left camera coordinate system based on the binary code includes: Performing three-dimensional printing based on an opaque photosensitive resin to obtain a three-dimensional printed part, pasting a binary code on the three-dimensional printed part, installing the three-dimensional printed part on the sensor frame, and determining a relative position between the three-dimensional printed part and a sensor frame coordinate system; Shooting the binary code on the three-dimensional printed part by the left camera to obtain a binary image, and determining the relative position and posture between the three-dimensional printed part and the left camera coordinate system based on the binary image; Based on the relative posture between the three-dimensional print and the sensor frame coordinate system, and the relative posture between the three-dimensional print and the left camera coordinate system, the posture relationship between the sensor frame coordinate system and the left camera coordinate system is determined.

2. The method for preparing a tactile sensor according to claim 1, characterized in that: The gel mold comprises a bottom gel mold and an upper gel mold, and the gel mold and the gel support plate are obtained by three-dimensional printing through transparent photosensitive resin; The method of preparing the gel layer based on the gel mold and the gel support plate comprises: Selecting a transparent two-component silica gel, mixing the first component and the second component of the two-component silica gel to obtain a silica gel mixed liquid; Pour the silica gel mixture into the bottom gel mold, place the gel support plate on the bottom gel mold with the silica gel mixture, and pour the silica gel mixture into the gel support plate until it covers the gel support plate; The upper gel mold is placed on the gel support plate, and the bottom gel mold, the gel support plate and the upper gel mold are fixed with fasteners to obtain a fixing part, and the gel layer is taken out after the silica gel mixture in the fixing part is left to solidify.

Images