A multifunctional sensor integrating force sense and visual touch sense and its measurement method

Abstract

The present invention discloses a multifunctional sensor integrating force sense and visual touch sense and a measurement method thereof, wherein the sensor comprises an elastic structure, an optical fiber, a camera, a housing, a reflector, a light emitting diode, a contact body, a depth reconstruction and a three-dimensional force measurement module; the elastic structure comprises an upper and lower platform and a plurality of springs vertically fixed between the upper and lower platforms; the optical fiber is consistent in number with the springs, and is arranged inside the springs one by one, and the two ends are fixed to the upper and lower platforms and are in a taut suspended state; the upper platform and the reflector are inlaid and connected; the contact body is fixed to the inner side of the top boss of the upper platform, and the top contact surface is just exposed at the open position of the top of the housing; the camera is fixed to the lower platform and the camera is aligned with the bottom center of the contact body; the depth reconstruction algorithm module is used to output a depth map according to the contact body image; and the three-dimensional force measurement algorithm module is used to calculate the three-dimensional force of the contact body according to the center wavelength drift of the optical fiber. The present invention can simultaneously obtain the force information and depth information of the contact sensor.

Description

Technical Field

[0001] The invention belongs to the technical field of mechanical measurement, and in particular relates to a multifunctional sensor integrating force sense and visual touch sense and a measurement method thereof. Background Art

[0002] Acupuncture is a systematic treatment method in traditional Chinese medicine, and the acupuncture technique determines its treatment effect to a certain extent. In order to meet the needs of the acupuncture robot to simultaneously perceive the interactive force and posture of the acupuncture needle during operation, it is necessary to design a visual-tactile finger that integrates direct multi-dimensional force measurement to achieve direct interactive force measurement, texture measurement, depth measurement and posture measurement, so as to provide a guarantee for further realizing safe and effective humanoid acupuncture. Summary of the invention

[0003] The present invention provides a multifunctional sensor integrating force sense and visual touch sense and a measuring method thereof, which can simultaneously obtain force information and depth information contacting the sensor, thereby improving the accuracy and dexterity of mechanical arm operation.

[0004] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0005] A multifunctional sensor integrating force sense and visual touch sense, in cylindrical shape, including: an elastic structure, an optical fiber, a camera, a housing, a reflector, a light emitting diode, a contact body, a depth reconstruction algorithm module and a three-dimensional force measurement algorithm module;

[0006] The shell is a cylindrical shell with circular openings at the top and bottom ends;

[0007] The elastic structure includes an upper platform, a lower platform and a plurality of springs vertically fixed between the upper and lower platforms, the upper and lower platforms are both cylindrical and have an outer diameter smaller than the inner diameter of the housing; the number of the optical fibers is consistent with the number of the springs, which are arranged inside the springs one by one and have both ends fixed to the upper and lower platforms and in a taut suspension state;

[0008] The upper platform is a hollow cylindrical structure, and an arc groove is arranged on the inner side of the upper end of the upper platform;

[0009] The outer side of the reflective plate is provided with an arc protrusion matching the arc groove of the upper platform so as to be inlaid and connected with the upper platform;

[0010] A boss is provided on the outer side surface of the top end of the upper platform to fix the cylindrical contact body on the inner side of the boss on the top end of the upper platform;

[0011] The elastic structure is fixed inside the shell, and the top contact surface of the contact body is just exposed at the open position of the top of the shell;

[0012] The light emitting diode is arranged on the inner side of the housing and is used to provide a light source for the contact body;

[0013] The camera is fixed on the lower platform, and the camera is aimed at the bottom center of the contact body to photograph the contact body;

[0014] The depth reconstruction algorithm module is used to output a depth map of the object on the contact body according to the contact body image taken by the camera;

[0015] The three-dimensional force measurement algorithm module is used to calculate the three-dimensional force load of the output contact body according to the central wavelength drift of the optical fiber.

[0016] Furthermore, the number of the springs and optical fibers is set to 3, and the three springs are evenly distributed on the circumference of the outer sides of the upper and lower platforms at an interval of 120°.

[0017] Furthermore, the spring is a spring with a hollow interior and a rectangular cross-section.

[0018] Furthermore, the circular opening at the top of the shell has a diameter smaller than the diameter of the shell; the contact body includes an elastic body and an acrylic plate; the acrylic plate is fixed to the bottom surface of the elastic body and serves as a support plate for the elastic body;

[0019] The elastomer is divided into two layers; the diameter of the upper layer of the elastomer is the same as the circular opening at the top of the shell, and is completely exposed on the surface of the sensor; the diameter of the lower layer of the elastomer is the same as the diameter of the acrylic plate and is larger than the diameter of the circular opening at the top of the shell, and is clamped inside the shell of the sensor.

[0020] Furthermore, a plurality of shell fixing holes are evenly arranged on the outer circumferential surface of the upper platform; and through holes are also correspondingly arranged on the shell at corresponding positions of the shell fixing holes, for using bolts to pass through the through holes on the shell and to be bolted to the shell fixing holes of the upper platform, so as to fix the elastic structure to the shell through the upper platform.

[0021] Furthermore, a recessed groove is provided on the inner side surface of the housing above the reflector for pasting and fixing the light emitting diode.

[0022] Furthermore, the number of the light emitting diodes is set to 3, which are red, green and blue light diodes, respectively, and are evenly distributed on the circumference of the inner side surface of the shell and above the reflector at an interval of 120°.

[0023] Furthermore, the three-dimensional force measurement algorithm module calculates the three-dimensional force load of the output contact body according to the central wavelength drift of the optical fiber. The specific calculation formula is:

[0024] ;

[0025] The center point of the lower platform is taken as the origin, and the position point of any optical fiber on the lower platform along the direction of the center point of the lower platform is taken as the positive direction of the x-axis. A three-dimensional rectangular coordinate system is established using the right-hand rule to obtain the directions of the y-axis and the z-axis.

[0026] is the three-dimensional force on the contact body, is the temperature change, is the central wavelength of the optical fiber, is the center wavelength drift of the three optical fibers, is the coefficient of thermal expansion, is the thermo-optical effect coefficient, is the photoelastic coefficient; are all intermediate variables.

[0027] , , ;

[0028] in, represents the equivalent moment of inertia of the elastic structure as a whole about the y-axis, represents the equivalent moment of inertia of the elastic structure as a whole about the x-axis, represents the Young's modulus of the elastic structural material, represents the height of the platform under the elastic structure, represents the height of the spring structure, Indicates the height of the upper platform, is the stiffness of the hollow spring with rectangular cross section, is the stiffness of the fiber Bragg grating.

[0029] Furthermore, the three-dimensional force measurement algorithm module has a pre-trained neural network-based three-dimensional force prediction model built in, which is used to predict the three-dimensional force load of the output contact body according to the central wavelength drift of the optical fiber.

[0030] Furthermore, the depth reconstruction algorithm module has a pre-trained neural network-based depth reconstruction model built in, which is used to reconstruct and output a depth map of an object on the contact body according to the color intensity of the contact body image taken by the camera.

[0031] The present invention also provides a method for measuring force and visual touch, using any of the above-mentioned multifunctional sensors integrating force and visual touch to simultaneously measure and output three-dimensional force loads and tactile images.

[0032] At present, most visual-tactile sensors estimate force based on the marker layer method, using finite elements and the movement of marker points to map force information. Due to the introduction of the marker layer, this method will inevitably affect important image features, which is not conducive to the depth reconstruction task of the sensor. In addition, the use of a camera to collect force data is an indirect measurement, which is difficult to calibrate and the sampling frequency is affected by the camera frame rate. The multifunctional sensor that integrates force perception and visual-tactile perception of the present invention can simultaneously obtain high-precision, high-response force information and high-quality texture information without affecting the texture imaging effect of the sensor, thereby simultaneously obtaining the posture and clamping force of an acupuncture needle or other object, and improving the accuracy and dexterity of the robot arm operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the assembly of the sensor described in the embodiment of the present invention.

[0034] Figure 2 Schematic diagram of some components of the sensor according to an embodiment of the present invention; wherein (a) is a front view of the elastic structure, (b) is a bottom view of the lower platform, and (c) is a three-dimensional view of the housing.

[0035] Figure 3 It is a schematic diagram of the dimensions of the rectangular spring structure of the sensor according to an embodiment of the present invention.

[0036] Figure 4 Schematic diagram of the deformation of the flexible hinge described in an embodiment of the present invention when subjected to axial force and vertical axis moment; (a) is the force analysis of the structure when the elastic structure is subjected to a load Fx in the positive direction of the x-axis, and (b) is the torsion analysis after simplifying the elastic structure into a cantilever beam.

[0037] Figure 5 It is a schematic diagram of collecting dynamic and kinematic multimodal parameters during acupuncture manipulation in an embodiment of the present invention; wherein (a) is a front view of the torsion analysis of each FBG after the elastic structure is twisted by a load in the positive direction of the x-axis, and (b) is a top view of the torsion analysis of each FBG after the elastic structure is twisted by a load in the positive direction of the x-axis.

[0038] Figure 6 Schematic diagram of the process of the acupuncture manipulation classification method described in an embodiment of the present invention; wherein (a) is a front view of the torsion analysis of each FBG after the elastic structure is twisted by the positive load in the y-axis direction; (b) is a top view of the torsion analysis of each FBG after the elastic structure is twisted by the positive load in the y-axis direction.

[0039] Figure 7 A schematic diagram of a depth reconstruction method according to an embodiment of the present invention.

[0040] In the figure: 1000-sensor, 100-elastic structure, 110-upper platform, 111-housing fixing hole, 120-lower platform, 121-camera mounting through hole, 130-rectangular cross-section spring, 200-camera, 300-housing, 310-fixing through hole, 320-camera wiring channel, 330-optical fiber lead-out channel, 340-light-emitting diode groove, 350-light-emitting diode power line channel, 400-fixing bolt, 500-reflector, 600-light-emitting diode, 700-contact body, 710-elastic body, 720-acrylic plate, 801-first optical fiber, 802-second optical fiber, 803-third optical fiber. DETAILED DESCRIPTION

[0041] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.

[0042] refer to Figure 1 , Figure 2 As shown, this embodiment provides a multifunctional sensor 1000 integrating force and visual touch, which is cylindrical and includes: an elastic structure 100, an optical fiber, a camera 200, a housing 300, a reflector 500, a light-emitting diode 600, a contact body 700, a depth reconstruction algorithm module and a three-dimensional force measurement algorithm module.

[0043] The shell 300 is a cylindrical shell with circular openings at the top and bottom, wherein the diameter of the circular opening at the top of the shell is smaller than the diameter of the shell 300, so as to expose the contact surface of the internal contact body 700 to receive external force load.

[0044] The elastic structure 100 includes an upper platform 110, a lower platform 120 and a plurality of springs 130 vertically fixed between the upper and lower platforms 110 and 120. The upper and lower platforms 110 and 120 are both cylindrical and have an outer diameter smaller than the inner diameter of the housing 300. The entire elastic structure 100 is made of aluminum alloy.

[0045] The upper platform 110 is a hollow cylindrical structure, and a circular arc groove is arranged on the inner side of the upper end of the upper platform 110. The outer side of the reflector 500 is provided with a circular arc protrusion matching the circular arc groove of the upper platform 110, so as to be embedded and connected with the upper platform 110.

[0046] A boss is disposed on the outer side surface of the top end of the upper platform 110 to fix the cylindrical contact body 700 on the inner side of the boss on the top end of the upper platform 110 .

[0047] The contact body 700 includes an elastic body 710 and an acrylic plate 720; the acrylic plate 720 is fixed to the bottom surface of the elastic body 710 and serves as a support plate for the elastic body 710. The elastic body 710 is divided into two layers. The diameter of the upper layer of the elastic body 710 is the same as the circular opening at the top of the shell, so that the contact surface of the upper layer of the elastic body 710 is completely exposed at the opening position at the top of the shell, and then the contact surface is exposed on the surface of the sensor 1000. The diameter of the lower layer of the elastic body 710 is the same as the diameter of the acrylic plate 720 and is slightly larger than the diameter of the circular opening at the top of the shell, so that the contact body 700 and the reflective plate 500 and the elastic structure 100 fixed together below are clamped as a whole inside the shell of the sensor 1000.

[0048] The spring 130 in this embodiment is a spring with a hollow interior and a rectangular cross section, and the number of springs 130 and optical fibers is set to 3. The three springs 130 are evenly distributed on the outer circumference of the upper and lower platforms 110 and 120 at intervals of 120°, and the three optical fibers are set one by one at the hollow position inside the spring 130, and the two ends of the optical fibers are fixed to the upper and lower platforms 110 and 120 and are in a taut suspension state. The optical fibers in this embodiment are all fiber Bragg gratings.

[0049] A plurality of through holes 121 are provided inside the circumference of the three springs 130 of the lower platform 120 for mounting the camera 200. Specifically, a plurality of bolts are passed through the through holes from the bottom of the lower platform 120 to be screwed to the camera 200, and the camera 200 is fixedly mounted above the lower platform 120 by bolts, so that the camera 200 is located inside the three springs and aligned with the bottom center of the contact body 700, so as to photograph the contact body 700.

[0050] The inner side of the housing 300 is provided with an annular groove 340 along the circumferential direction at a position above the reflector 500, for pasting and fixing the light emitting diode 600. In this embodiment, the number of the light emitting diodes 600 is set to 3, which are red, green and blue light diodes, respectively, and are evenly distributed in the annular groove 340 on the inner side of the housing 300 and above the reflector 500 at an interval of 120°. When the camera 200 captures an image on the contact body 700, the light emitting diode 600 provides the contact body 700 with a red, green and blue light source. In addition, a through hole 350 is provided at any position of the annular groove 340, which extends vertically downward to the bottom end of the housing 300 in the axial direction. The power lines of each diode are buried along the annular groove 340, and are concentrated at the position of the vertically downward through hole 350 and led downward to the bottom end of the housing 300.

[0051] After the entire elastic structure 100 is fixed to the reflector 500 , the contact body 700 , the camera 200 , the optical fiber and other components, it is placed and fixed into the inside of the housing 300 from the open position at the bottom of the housing 300 .

[0052] A plurality of shell fixing holes 111 are evenly arranged on the outer circumferential surface of the upper platform 110; the shell 300 is also provided with corresponding fixing through holes 310 at corresponding positions of the shell fixing holes 111, for using bolts 400 to pass through the through holes on the shell 300 and to be bolted to the shell fixing holes 111 of the upper platform 110, so as to fix the elastic structure 100 to the shell 300 through the upper platform 110.

[0053] The housing 300 is also provided with a camera cable channel 320 and an optical fiber lead-out channel 330 for leading the camera cable and optical fibers fixed inside the housing 300 to the outside of the sensor 1000 .

[0054] The depth reconstruction algorithm module is used to output a depth map of the object on the contact body according to the contact body image taken by the camera.

[0055] The three-dimensional force measurement algorithm module is used to calculate the three-dimensional force load of the output contact body according to the central wavelength drift of the optical fiber.

[0056] In this embodiment, the housing 300 is prepared from materials by a 3D printer, the reflector 500 is prepared from materials by a 3D printer, the elastic structure 100 is prepared from materials by a 3D printer, and the contact elastomer 710 is prepared from AB silicone.

[0057] In this embodiment, the three optical fibers are named as the first, second, and third rectangular cross-section springs in the counterclockwise direction, the internal optical fibers are named as the first, second, and third optical fibers in sequence, the center point of the lower platform 120 is represented as O, and the positions of the lower platform 120 fixing the first, second, and third optical fibers are represented as A, C, and E. Below, point O is taken as the origin, the direction from point E to point O is the positive direction of the x-axis, and the right-hand rule is used to establish a three-dimensional rectangular coordinate system.

[0058] When the contact body of the sensor 1000 of this embodiment contacts the object to be measured, the three-dimensional forces Fx, Fy and Fz exerted on the contact body can be calculated by converting the grating center wavelength offsets of the three optical fibers.

[0059] When the contact body 700 is subjected to an axial force Fz, the three optical fibers undergo deformations of equal magnitude and direction, and the central wavelengths of the gratings of the three optical fibers drift accordingly. The axial force Fz can be measured based on the sum of the central wavelength offsets of the three fiber gratings.

[0060] When the contact body 700 is subjected to a radial force Fx, the first optical fiber 801 and the second optical fiber 802 undergo deformations of equal magnitude and in the same direction, the third optical fiber 803 undergoes deformation in the opposite direction to the first optical fiber 801 and the second optical fiber 802, and the central wavelengths of the gratings of the three optical fibers undergo corresponding drifts. The radial force Fx can be measured by differential calculation of the central wavelength offsets of the three fiber gratings.

[0061] When the contact body 700 is subjected to a radial force Fy, the first optical fiber 801 and the second optical fiber 802 undergo deformations of equal magnitude and opposite directions, and the third optical fiber 803 is substantially not deformed. At this time, the grating center wavelengths of the first optical fiber 801 and the second optical fiber 802 undergo corresponding drifts. The radial force Fy can be measured by differential calculation of the grating center wavelength offsets of the first optical fiber 801 and the second optical fiber 802.

[0062] Since the first optical fiber 801, the second optical fiber 802, and the third optical fiber 803 undergo approximately linear deformation under the action of the three-dimensional force, by analyzing the mechanical model of the elastic structure 100, the deformation of each rectangular spring structure 130 under the action of the three-dimensional force can be obtained, and then the relationship matrix between the grating center wavelength offset and the three-dimensional force of the first optical fiber 801, the second optical fiber 802, and the third optical fiber 803 can be obtained. The grating center wavelength offset of the optical fiber is related to the temperature. This embodiment is carried out in an experimental environment with an approximately constant temperature, and the temperature is regarded as a constant.

[0063] The specific working principle of the three-dimensional force measurement of the sensor in this embodiment is as follows:

[0064] Assemble and fix the components of sensor 1000, see Figure 3 , the deformation of the sensor elastic structure 100 when subjected to the axial force Fz, combined with material mechanics, it can be known that the relationship between the force direction and the strain magnitude is:

[0065] ;

[0066] in is the deformation of the three rectangular cross-section springs 130 when subjected to the axial force Fz, is the stiffness of the hollow spring with rectangular cross section, is the stiffness of the fiber Bragg grating. For the stiffness of a hollow spring with a rectangular cross section, it can be expressed as:

[0067] ;

[0068] in Usually take the first three items, represents the shear modulus of the spring material, is the diameter of the spiral structure, is the effective number of coils of the spring, and Respectively represent the height and width of the cross section of the spiral structure.

[0069] See also Figure 4 (a), when the elastic structure 100 is subjected to a tangential force, the entire elastic structure 100 will bend, thereby causing the three optical fibers to be stretched or compressed. Since the upper platform 110 is approximately a solid cylindrical structure after the reflector 500 is installed, and the lower platform 120 is also approximately a solid cylindrical structure, its moment of inertia is obviously much larger than the three rectangular spring structures 130, so it can be approximately regarded as a rigid structure. When the sensor elastic structure 100 is subjected to a radial force Fx, the rectangular spring structure 130 is simplified to a cantilever beam, such as Figure 4 As shown in (b), the total deflection angle of the elastic structure 100 can be expressed as:

[0070] ;

[0071] in represents the equivalent moment of inertia of the elastic structure 100 as a whole about the y-axis, represents the Young's modulus of the material of the elastic structure 100, represents the height of the platform 120 under the elastic structure 100, represents the height of the spring structure 130, Indicates the height of the upper platform 110. Figure 5 (a) For the three optical fibers, the first optical fiber 801 and the second optical fiber 802 are compressed, and the third optical fiber 803 is stretched. Since the deformation is small, the deformation of the three optical fibers is approximately regarded as an arc structure and mapped to the neutral plane. The curvature after deformation is According to the geometric relationship:

[0072] ;

[0073] Assuming arc segment The radius of curvature is , each spring structure 130 is at a distance from the center of the entire elastic structure 100 ,like Figure 5 (b) shows the arc segment , and The radius of , and , the average strains of the three optical fibers can be expressed as:

[0074] ;

[0075] ;

[0076] ;

[0077] Similarly, see Figure 6 (a) and Figure 6 (b) When the sensor elastic structure 100 is subjected to a radial force Fy, the average strains of the three optical fibers can be expressed as:

[0078] ;

[0079] ;

[0080] ;

[0081] in, represents the equivalent moment of inertia of the elastic structure 100 as a whole with respect to the x-axis. Considering the relationship between the grating center wavelength offset of the optical fiber and the strain and temperature of the optical fiber, it can be obtained that:

[0082] ;

[0083] Assume that the central wavelength of the optical fiber is ,in, , , , is the temperature change, and are the thermal expansion coefficient and the thermo-optical effect coefficient, is the photoelastic coefficient.

[0084] Combining the above equations, we can obtain the grating center wavelength offset and three-dimensional force of the first optical fiber 801, the second optical fiber 802, and the third optical fiber 803. and temperature changes The relationship matrix:

[0085] ;

[0086] This embodiment is carried out in an experimental environment with a temperature approximately constant, and the temperature is regarded as a constant. is considered as 0. The matrix can be expressed as:

[0087] ;

[0088] Based on the central wavelength offset of the three optical fibers, combined with the pre-known central wavelength of the grating And the coefficient , the three-dimensional force can be obtained from the above relationship transformation matrix .

[0089] In other embodiments, the three-dimensional force measurement algorithm module can also use a trained neural network as a three-dimensional force measurement system to predict the three-dimensional force load of the output contact body according to the drift of the central wavelength of the optical fiber. Among them, the parameters of the neural network are obtained by training the force calibration data set: the drift of the central wavelength of the optical fiber under the action of a known three-dimensional force load is collected to construct a force calibration data set; the three-dimensional force load includes axial force and radial force, which are obtained by weights; the drift of the central wavelength of the optical fiber is used as input, and the three-dimensional force load is used as output to optimize the parameters of the neural network. The specific neural network can use an extreme learning machine ELM.

[0090] In this embodiment, the depth reconstruction algorithm module has a pre-trained neural network-based depth reconstruction model built in, which is used to reconstruct and output a depth map of the object on the contact body according to the color intensity of the contact body image taken by the camera, such as Figure 7 As shown. The calibration samples are constructed, and the depth reconstruction model is trained and used. Specifically, in this embodiment:

[0091] Step 1: Press an object of known actual size, such as a coin, on the sensor contact body. The conversion relationship between the number of image pixels obtained by the sensor and the actual corresponding number of millimeters can be calculated through the collected image and the actual size.

[0092] Step 2: Capture a non-contact image as a reference image, then press a known diameter on the sensor multiple times The collected image is first preprocessed, then the difference is calculated with the non-contact image, and finally the center of the contact circle in the tactile image is found through the difference image. With these parameters, the gradients Gx and Gy corresponding to each contact pixel can be calculated. The specific steps are:

[0093] (1) Assume that the coordinate value of each pixel point within the radius in the image is , first calculate the distance from each pixel within the radius to the center of the sphere :

[0094] ;

[0095] Then, according to the geometric model of the sphere, the spherical equation is used to calculate the height value of each pixel, the height of a point on the sphere It can be calculated by the following formula:

[0096] ;

[0097] (2) In order to calculate the gradient of the image, a simple difference operator is used to calculate the horizontal gradient:

[0098] ;

[0099] Through the convolution operation, the gradient can be calculated over the entire height map:

[0100] ;

[0101] in, Represents the convolution kernel in the horizontal direction, expressed as:

[0102] ;

[0103] Similarly, in the vertical direction:

[0104] ;

[0105] Through the convolution operation, the gradient can be calculated over the entire height map:

[0106] ;

[0107] in, Represents the convolution kernel in the vertical direction, expressed as:

[0108] ;

[0109] Step 3, according to the principle of photometric stereo, for the contact body 710 coated with a uniform reflective layer, there is a mapping relationship between the gradient of each contact pixel and the color intensity of the point. This mapping relationship is related to the reflection function of the contact body surface, which is usually a nonlinear function related to the gradient. This embodiment uses an MLP neural network to obtain the mapping between the gradient of each pixel and the light intensity through training. Since the light emitting diode 600 has three colors of red, green and blue, three channels are selected for mapping.

[0110] The MLP neural network adopts a structure with 3 hidden layers, each hidden layer has 32 neurons, and the activation function is tanh. The input is the position of the pixel point and the intensity values ​​of the three colors of red, green and blue at the corresponding position, and the output is the gradient Gx in the x-axis direction and the gradient Gy in the y-axis direction.

[0111] Step 4: After training, the gradient Gx and Gy corresponding to the point can be obtained by knowing the position of the pixel and the intensity values ​​of the red, green and blue colors at the point. The fast Poisson solver is used to obtain the depth corresponding to the point. The specific steps are:

[0112] (1) First, the problem to be solved for the Poisson equation can be expressed as ,in express The Laplace operator represents the second-order spatial derivative of the image, and and is the gradient field of a known image.

[0113] First, the Poisson equation is discretized to adapt to digital image processing. It can be regarded as a two-dimensional discrete function, and each pixel position in the image is expressed as :

[0114] ;

[0115] The discretized version of Poisson's equation is:

[0116] ;

[0117] Then, we calculate by gradient difference and The difference between , thus obtaining the source term of the Poisson equation:

[0118] ;

[0119] in:

[0120] ;

[0121] ;

[0122] (2) In order to efficiently solve the Poisson equation, the solver uses discrete sine transform (DST) to transform the problem into the frequency domain. For the image source term, the discrete sine transform can be expressed as:

[0123] ;

[0124] For two-dimensional discrete sine transform, the transform result needs to be orthogonally normalized, that is:

[0125] ;

[0126] in, and are the height and width of the image, and is the frequency index in the frequency domain.

[0127] Next, perform a secondary discrete sine transform on its transpose, and the whole process can be expressed as:

[0128] ;

[0129] ;

[0130] In the frequency domain, the process of solving Poisson's equation is very simple, by defining a denominator To adjust the source term, specifically:

[0131] ;

[0132] This denominator term reflects the frequency characteristics of the frequency domain and is usually used to adjust the impact of each frequency component in the frequency domain. Subsequently, the source term in the frequency domain is divided by the denominator term:

[0133] ;

[0134] (3) After completing the frequency domain processing, the inverse discrete sine transform (IDST) is used to convert the result back to the spatial domain. For the source term in the frequency domain, it can be expressed as:

[0135] ;

[0136] ;

[0137] These two inverse transforms restore the results in the frequency domain to the spatial domain to obtain the reconstructed depth map .

[0138] The above embodiments are preferred embodiments of the present invention. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present invention, these changes or improvements should fall within the scope of protection required by the present invention.

Claims

1. A multifunctional sensor integrating force sense and visual touch sense, characterized in that: It is cylindrical and includes: an elastic structure, an optical fiber, a camera, a housing, a reflector, a light-emitting diode, a contact body, a depth reconstruction algorithm module and a three-dimensional force measurement algorithm module; The shell is a cylindrical shell with circular openings at the top and bottom ends; The elastic structure includes an upper platform, a lower platform and a plurality of springs vertically fixed between the upper and lower platforms, the upper and lower platforms are both cylindrical and have an outer diameter smaller than the inner diameter of the housing; the number of the optical fibers is consistent with the number of the springs, which are arranged inside the springs one by one and have both ends fixed to the upper and lower platforms and in a taut suspension state; The upper platform is a hollow cylindrical structure, and an arc groove is arranged on the inner side of the upper end of the upper platform; The outer side of the reflective plate is provided with an arc protrusion matching the arc groove of the upper platform so as to be inlaid and connected with the upper platform; A boss is provided on the outer side surface of the top end of the upper platform to fix the cylindrical contact body on the inner side of the boss on the top end of the upper platform; The elastic structure is fixed inside the shell, and the top contact surface of the contact body is just exposed at the open position of the top of the shell; The light emitting diode is arranged on the inner side of the housing and is used to provide a light source for the contact body; The camera is fixed on the lower platform, and the camera is aimed at the bottom center of the contact body to photograph the contact body; The depth reconstruction algorithm module is used to output a depth map of the object on the contact body according to the contact body image taken by the camera; The three-dimensional force measurement algorithm module is used to calculate the three-dimensional force load of the output contact body according to the central wavelength drift of the optical fiber.

2. The multifunctional sensor integrating force sense and visual touch sense according to claim 1, characterized in that: The number of the springs and optical fibers is set to 3, and the three springs are evenly distributed on the outer circumference of the upper and lower platforms at an interval of 120 degrees.

3. The multifunctional sensor integrating force sense and visual touch sense according to claim 1, characterized in that: The spring is a spring with a hollow interior and a rectangular cross section.

4. The multifunctional sensor integrating force sense and visual touch sense according to claim 1, characterized in that: The circular opening at the top of the shell has a diameter smaller than the diameter of the shell; the contact body includes an elastic body and an acrylic plate; the acrylic plate is fixed to the bottom surface of the elastic body and serves as a support plate for the elastic body; The elastomer is divided into two layers; the diameter of the upper layer of the elastomer is the same as the circular opening at the top of the shell, and is completely exposed on the surface of the sensor; the diameter of the lower layer of the elastomer is the same as the diameter of the acrylic plate and is larger than the diameter of the circular opening at the top of the shell, and is clamped inside the shell of the sensor.

5. The multifunctional sensor integrating force sense and visual touch sense according to claim 1, characterized in that: A plurality of shell fixing holes are evenly arranged on the outer circumferential surface of the upper platform; through holes are also correspondingly arranged on the shell at corresponding positions of the shell fixing holes, for using bolts to pass through the through holes on the shell and to be bolted to the shell fixing holes of the upper platform, so as to fix the elastic structure to the shell through the upper platform.

6. The multifunctional sensor integrating force sense and visual touch sense according to claim 1, characterized in that: A recessed groove is arranged on the inner side of the shell above the reflector for pasting and fixing the light emitting diode.

7. The multifunctional sensor integrating force sense and visual touch sense according to claim 1, characterized in that: The number of the light emitting diodes is set to 3, which are red, green and blue light diodes, and are evenly distributed on the circumference of the inner side of the shell and above the reflector at an interval of 120 degrees.

8. The multifunctional sensor integrating force sense and visual touch sense according to claim 2, characterized in that: The three-dimensional force measurement algorithm module calculates the three-dimensional force load of the output contact body according to the central wavelength drift of the optical fiber. The specific calculation formula is: ； The center point of the lower platform is taken as the origin, and the position point of any optical fiber on the lower platform along the direction of the center point of the lower platform is taken as the positive direction of the x-axis. A three-dimensional rectangular coordinate system is established using the right-hand rule to obtain the directions of the y-axis and the z-axis. is the three-dimensional force on the contact body, is the temperature change, is the central wavelength of the optical fiber, is the center wavelength drift of the three optical fibers, is the coefficient of thermal expansion, is the thermo-optical effect coefficient, is the photoelastic coefficient; are all intermediate variables. ， ， ； in, represents the equivalent moment of inertia of the elastic structure as a whole about the y-axis, represents the equivalent moment of inertia of the elastic structure as a whole about the x-axis, represents the Young's modulus of the elastic structural material, represents the height of the platform under the elastic structure, represents the height of the spring structure, Indicates the height of the upper platform, is the stiffness of the hollow spring with rectangular cross section, is the stiffness of the FBG; or, The three-dimensional force measurement algorithm module has a pre-trained neural network-based three-dimensional force prediction model built in, which is used to predict the three-dimensional force load of the output contact body according to the central wavelength drift of the optical fiber.

9. The multifunctional sensor integrating force sense and visual touch sense according to claim 1, characterized in that: The depth reconstruction algorithm module has a pre-trained neural network-based depth reconstruction model built in, which is used to reconstruct and output a depth map of objects on the contact body according to the color intensity of the contact body image taken by the camera.

10. A method for measuring force sense and visual touch, characterized in that: The multifunctional sensor integrating force sense and visual touch sense as described in any one of claims 1 to 9 is used to simultaneously measure and output three-dimensional force load and tactile image.

Images