Vision-based tactile sensor based on lensless imaging, and measurement method using same

Abstract

A vision-based tactile sensor based on lensless imaging, comprising: a lensless imaging system consisting of a detachable amplitude mask (4), an image sensor (5), and a supporting structure (6); a reflective film (1); transparent elastomer (2); and multiple light sources (3) of different colors. The transparent elastomer (2) is located on the light incident side of the lensless imaging system, and covers and is in close contact with the detachable amplitude mask (4); the reflective film (1) covers the surface of the side of the transparent elastomer (2) facing away from the detachable amplitude mask (4); the light sources (3) are uniformly arranged on the supporting structure (6) around the detachable amplitude mask (4), so as to ensure that light emitted by the light sources (3) is reflected by the reflective film (1) and then passes through the transparent elastomer (2) to reach the detachable amplitude mask (4). Also provided is a measurement method using the vision-based tactile sensor based on lensless imaging.

Description

A lensless imaging-based visual-tactile sensor and its measurement method Technical Field

[0001] This application relates to the field of sensor technology, and in particular to a tactile sensor based on lensless imaging and its measurement method. Background Technology

[0002] Driven by societal development needs, the market demand for service robots continues to grow, and tactile sensing is one of the crucial senses for assisting service robots in performing tasks in unstructured environments. Visuotactile sensors can acquire rich contact information, and their spatial resolution surpasses that of human fingertips, and they are increasingly being widely adopted by academia and industry.

[0003] Lensless imaging, an emerging imaging technology in recent years, uses an optical encoder to replace traditional lenses, encoding scene information into the observed image of a visual-tactile sensor. Due to the absence of lenses in lensless imaging, each point in the scene cannot form a focused point on the image sensor, resulting in a wide range of responses. Therefore, the image captured by the image sensor is completely dissimilar to the scene image, requiring the use of appropriate algorithms to reconstruct the scene.

[0004] Currently, most visual-tactile sensors use cameras for imaging, resulting in their large thickness and volume, which limits their use in space-constrained environments. At the same time, existing lensless imaging reconstruction algorithms have a heavy computational burden and cannot meet the real-time requirements of visual-tactile sensors. Summary of the Invention

[0005] This application provides a lensless imaging-based visual-tactile sensor and its measurement method, which can be applied to service robots, such as service robot perception, service robot operation and interaction. The visual-tactile sensor is small in size, flexible in operation, and easy to implement. It also overcomes the problem that the slow speed of lensless close-range imaging prevents its application to visual-tactile sensors. It has many applicable scenarios and high application value.

[0006] In a first aspect, embodiments of this application provide a lensless imaging-based visual-tactile sensor, comprising: a lensless imaging system consisting of a separable amplitude mask, an image sensor, and a support structure; a reflective film, a transparent elastomer, and multiple light sources of different colors; in the lensless imaging system, the support structure is used to fix the separable amplitude mask and the image sensor respectively; the separable amplitude mask is placed in front of the image sensor, with their center lines aligned and their horizontal and vertical directions aligned; light from the scene under test passes through the separable amplitude mask and illuminates the image sensor, and the image sensor outputs a corresponding observed image to achieve image reconstruction of the scene under test; the transparent elastomer is located on the side of the incident light in the lensless imaging system, covering and closely adhering to the separable amplitude mask; the reflective film covers the surface of the transparent elastomer away from the separable amplitude mask; each light source is uniformly placed on the support structure around the separable amplitude mask to ensure that the light emitted from the light source is reflected by the reflective film and then illuminates the separable amplitude mask through the transparent elastomer.

[0007] In this technical solution, a lensless imaging system is used instead of a camera and lens imaging system, which greatly reduces the size and thickness of the sensor. This overcomes the problem that the existing visual and tactile sensors are too thick and therefore have limited space. It can perform tasks in confined spaces and can be applied to service robots. This allows these small and compact tactile sensors to be flexibly integrated into components such as robot dexterity hands, enabling service robots to interact better with humans and the environment.

[0008] In one implementation, the number of colors of the light source is greater than or equal to three.

[0009] In one implementation, the transparent elastomer is an elastic material with a light transmittance greater than 80% at visible light frequencies, and is in the shape of a flat plate or a trapezoidal shape.

[0010] In one implementation, the reflective film is a coating or film with a refractive index greater than 50% for visible light frequencies, which is applied to the surface of a transparent elastomer by spraying or adhesive.

[0011] Secondly, embodiments of this application provide a measurement method based on the above-mentioned visual-tactile sensor. The method includes: calibrating the system matrix of the lensless imaging system within the visual-tactile sensor before covering the transparent elastomer surface with a reflective film during the construction of the visual-tactile sensor; performing joint filter optimization on the calibrated lensless imaging system; covering the transparent elastomer surface with a reflective film; calibrating the visual-tactile sensor based on the optimization results of the joint filter; and performing measurements using the calibrated visual-tactile sensor.

[0012] In this technical solution, the calibration and measurement method of the visual-touch sensor based on lensless imaging enables the visual-touch sensor to be used to acquire the depth of the measured object and realize the visual-touch function. The system matrix calibration method of the lensless imaging system in the visual-touch sensor and the high-speed non-iterative image fast reconstruction method use analytical solutions to describe the reconstruction imaging process. The required calculation matrices can be obtained in advance, so they can be matched with the sensing speed of the image sensor to perform high-speed reconstruction solution. This overcomes the problem that existing methods require multiple iterations and cannot reconstruct in real time, greatly improving the imaging efficiency.

[0013] In one implementation, calibrating the system matrix of the lensless imaging system within the visual-touch sensor includes:

[0014] 1) Image acquisition calibration; the specific steps are as follows:

[0015] 1-1) Place the visual-touch sensor without the reflective film covering it parallel to a pixel size of N. x ×M x In front of the display, the horizontal and vertical directions of the display are consistent with the horizontal and vertical directions of the lensless imaging system, and the display is in close contact with the transparent elastomer surface of the visual-tactile sensor; the display screen is larger than the transparent elastomer, and the pixel size of the display is smaller than the pixel size of the reconstructed image;

[0016] 1-2) Creating an image collection Where the i-th image A in this set i This is an image where only the pixels in the i-th row are white, and the rest are black.

[0017] 1-3) Creating an image collection Where the i-th image in this set is B i This is an image where only the pixels in the i-th column are white, and the rest are black.

[0018] 1-4) The display is arranged from 1 to N x The image collection is displayed in sequence. Each image in the image set is acquired by a lensless imaging system within the visual-touch sensor. calculate The set consists of the average brightness of the entire image in each image. Where c i For image C i The average brightness of the entire image, for Taking the derivative, we obtain the image index i = e corresponding to the center point of the brightness mean. Then, we select the image set. The images in this set are then renumbered sequentially from smallest to largest to form the first calibration image set. Where Y hiLet i be the i-th image in the image set;

[0019] 1-5) The monitor is arranged from 1 to M x The image collection is displayed in sequence. Each image in the image set is acquired by a lensless imaging system within the visual-touch sensor. calculate The set consists of the average brightness of the entire image in each image. Where d i For image D i The average brightness of the entire image, for Taking the derivative, we obtain the image index i = g corresponding to the center point of the brightness mean. Then, we select the image set. The images in this set are then renumbered in ascending order to form the second calibration image set. Where Y vi The i-th image in the second calibration image set;

[0020] 2) Calculate the calibration matrix; the specific steps are as follows:

[0021] 2-1) Let the system matrix of the lensless imaging system be P c Q c P o Q o Where P represents the left multiplication matrix, Q represents the right multiplication matrix, the subscript c indicates that the system matrix contains coded components, and the subscript o indicates that the system matrix is ​​the response of the image sensor to a point light source without a mask; let p oi ,p ci ,q oi ,q ci P respectively o P c Q o Q c The i-th column, p o ,p c ,q o ,q c P respectively o P c Q o Q c The vector obtained by summing each row;

[0022] 2-2) Suppose that, given a first calibration image set and q o ,q c Calculate P under the condition o P c The process is as follows:

[0023] Let i range from 1 to N, and calculate:

[0024] Get P o =[p o1 ,p o2 ,…,p oN ], P c =[p c1 ,p c2 ,…,p cN ];

[0025] Suppose that under the given conditions and p o ,p c Calculate Q under the condition o Q c The process is as follows:

[0026] In the formula, For Y vi Transpose of;

[0027] Where i ranges from 1 to M, and the following calculations are performed:

[0028] Get Q o =[q o1 ,q o2 ,…,q oM ], Q c =[q c1 ,q c2 ,…,q cM ];

[0029] 2-3) Generate q o The initial estimate and P are obtained. o ;

[0030] Where, let q o =1, that is, a column vector whose elements are all 1s. Take i from 1 to N and calculate the following formula:

[0031] Get P o =[p o1 ,p o2 ,…,p oN ];

[0032] 2-4) Using calibration images Estimate q c ;

[0033] in for:

[0034] right Performing singular value decomposition, we obtain:

[0035] Where U, Σ, and V are The resulting matrix after singular value decomposition;

[0036] Take q c Equal to the first column of V, and take i from 1 to N and repeat the following calculation: Y hi =Y hi -p oi 1 T

[0037] Get P c =[p c1 ,p c2 ,…,p cN ];

[0038] 2-5) Based on P o P c Initial value, calculate Q o Q c ;

[0039] Where, let p o =P o 1, p c =P c 1;

[0040] Depend on Get Q o Q c ;

[0041] 2-6) Iteratively optimize the system matrix;

[0042] P o P c Q o Q c Rename them to P oa P ca Q oa Q ca The system matrix before optimization;

[0043] First, scale P. oa Q oa Make the F-norms of both equal, then scale P. ca Q ca Make their F-norms equal;

[0044] Let q o =Q oa 1, q c =Q ca 1. By The optimized system matrix P is obtained ob ,P cb ;

[0045] Let p o =P ob 1, p c =P cb 1. By The optimized system matrix Q is obtained. ob Q cb ;

[0046] Calculate the system matrix P before optimization oa P ca Q oa Q ca With the optimized system matrix P ob P cb Q ob Q cb If the difference between the intervals converges or reaches the set maximum number of iterations, the optimization is complete, and the optimized P is obtained. ob P cb Q ob Q cb Let this be denoted as the system matrix of the lensless imaging system.

[0047] In one implementation, joint filter optimization is performed on the calibrated lensless imaging system, including:

[0048] 1) For the scene to be tested X and the observed image Y from the lensless imaging system within the visual-tactile sensor, the following relationship exists: Y = Y c +Y o (1)

[0049] Among them, Y c Y is a rich coding component of the observation results. o The undercoded components of the observation results;

[0050] 2) Design a frequency domain filtering unit for the observed image Y. Spatial filtering unit The frequency-spatial joint filter obtained by concatenating the two get:

[0051] in, It is a frequency domain filtering unit The result of Y-filtering the observed image, It is a spatial filtering unit The result of Y-filtering the observed image, It is a joint filter The result of Y-filtering the observed image; Φ f It is the parameter matrix of the frequency domain filtering unit, with the same dimensions as Y; and These are the discrete cosine transform and its inverse transform; ⊙ represents the Hadamard product of the matrix; Φ s It is the parameter matrix of the spatial filtering unit, with the same dimension as Y;

[0052] The objective of this frequency-spatial joint filter is:

[0053] Result: Y c =YY o (8)

[0054] 3) Prepare the image set of the scene to be tested. This image set contains K scene images of size N×M, where X i Let i be the i-th image of the scene to be tested in the image set of the scene to be tested;

[0055] Among them, for the scene image X to be tested i The following relationship exists: Y i =Y ci +Y oi (11)

[0056] The joint filter parameter matrix is ​​optimized to obtain The optimization results.

[0057] In one implementation, the visual-touch sensor is calibrated based on the optimization results of the joint filter, including:

[0058] 1) Press balls of different sizes onto the surface of the reflective film at different depths, causing deformation of the reflective film and the transparent elastomer underneath it;

[0059] 2) The light projected by the light source passes through the transparent elastomer and shines on the deformed reflective film. The reflected light then passes through the transparent elastomer and shines on the separable amplitude mask.

[0060] 3) A separable amplitude mask modulates the amplitude of the reflected light, and the modulated light illuminates the image sensor, generating a corresponding image signal in the lensless imaging system;

[0061] 4) Based on the image signal from step 3), perform image reconstruction to obtain the reconstructed image signal;

[0062] 5) The light intensity-gradient-depth mapping relationship is obtained from the reconstructed image signal, the size of the ball and the pressing depth, so as to complete the calibration of the visual tactile sensor.

[0063] In one implementation, measurement is performed using a calibrated visual-tactile sensor, including:

[0064] 1) Press the reflective film of the visual-tactile sensor onto the surface of the object being measured, and the reflective film and the transparent elastomer underneath the reflective film will deform;

[0065] 2) The light projected from the light source passes through the transparent elastomer and irradiates the deformable reflective film. The reflected light passes through the transparent elastomer and irradiates the separable amplitude mask.

[0066] 3) A separable amplitude mask modulates the amplitude of the reflected light, and the modulated light illuminates the image sensor, generating a corresponding image signal in the lensless imaging system;

[0067] 4) Based on the image signal from step 3), perform image reconstruction to obtain the intensity of the reflected light before modulation;

[0068] 5) Calculate the deformation depth of the reflective film based on the calibrated light intensity-gradient-depth relationship to obtain the corresponding measurement result of the pressure depth of the object being tested.

[0069] In one implementation, image reconstruction includes:

[0070] 1) For any scene image X to be measured, the observed image obtained after observation by the lensless imaging system is Y, and the corresponding reconstructed image is

[0071] The rich coding components of the observed image are obtained based on the joint filter, namely: Y c =YY o (14)

[0072] Solve the following optimization problem:

[0073] Where τ is the regularization coefficient, Let F be the F-norm of the matrix;

[0074] The solution to this optimization problem is:

[0075] Equations (16) and (17) are for P c Q c Each performs singular value decomposition;

[0076] σ P With σ Q respectively using and A column vector consisting of the diagonal elements;

[0077] . / represents element-wise division of a matrix, and 1 represents a column vector with all elements being 1; The solution is the image reconstruction result. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0079] Figure 1 is a schematic diagram of the structure of a lensless imaging-based visual-tactile sensor provided in an embodiment of this application;

[0080] Figure 2 is a cross-sectional view of a lensless imaging-based visual-tactile sensor according to an embodiment of this application;

[0081] Figure 3 is a top view of a lensless imaging-based visual-tactile sensor according to an embodiment of this application;

[0082] Figure 4 is a pattern on the separable amplitude mask according to an embodiment of this application. Detailed Implementation

[0083] The following section, with reference to the accompanying drawings, provides a detailed description of the lensless imaging-based visual-tactile sensor and its measurement method provided in this application.

[0084] Please refer to Figure 1, which is a schematic diagram of a lensless imaging-based visual-tactile sensor provided in an embodiment of this application. As shown in Figure 1, the visual-tactile sensor may include, but is not limited to, the following structures:

[0085] 1. Reflective film; 2. Transparent elastomer; 3. Multiple light sources of different colors; 4. Separable amplitude mask; 5. Image sensor; 6. Support structure.

[0086] The system comprises a separable amplitude mask 4, an image sensor 5, and a support structure 6, forming a lensless imaging system for imaging. Both the separable amplitude mask 4 and the image sensor 5 are connected to the support structure 6, which is used to fix the separable amplitude mask 4 and the image sensor 5 respectively. The separable amplitude mask 4 is placed in front of the image sensor 5, with their center lines aligned and their horizontal and vertical directions aligned. The imaging process of this lensless imaging system is as follows: light from the scene X to be measured passes through the separable amplitude mask 4 and illuminates the image sensor 5, which then outputs the corresponding observed image Y. The observed image is then reconstructed to obtain the reconstructed image. The reconstructed image is the imaging result of the lensless imaging system.

[0087] By implementing the embodiments of this application, a lensless imaging system is used instead of a camera and lens imaging system, which greatly reduces the size and thickness of the sensor. This overcomes the problem that the existing visual-tactile sensors are too thick and therefore have limited space. The sensor can perform tasks in confined spaces and can be applied to service robots. This allows such small and compact tactile sensors to be flexibly integrated into components such as robot dexterity hands, enabling service robots to interact better with humans and the environment.

[0088] In one implementation, the support structure can be any rigid structure that can serve as a connection and support.

[0089] In one implementation, the transparent elastomer is an elastic material with a visible light transmittance greater than 80% at visible light frequencies, in the form of a flat or trapezoidal plate.

[0090] In one implementation, the reflective film is a coating or film with a refractive index greater than 50% for visible light frequencies that can be applied to the surface of a transparent elastomer by spraying or adhesive.

[0091] In one implementation, multiple light sources of different colors are used to excite electronic devices that can maintain stable color and light intensity within the visible light frequency range, including but not limited to LEDs or other devices that can achieve the same function.

[0092] In one implementation, a lensless imaging-based tactile sensor is shown in cross-sectional view as shown in Figure 2 and top view as shown in Figure 3. The separable amplitude mask 4 is made by spraying a black pattern onto a square transparent resin sheet, as shown in Figure 4, with a side length of 15.4 mm and a thickness of 1 mm. The image sensor 5 is a Teledyne Lince 5M CMOS image sensor used in the GO 5000C USB industrial camera. The support structure 6 is made of aluminum alloy, 70 mm long, 28 mm wide, and 4.8 mm thick at the thin end. It is connected to the image sensor 5 by bolts, with the separable amplitude mask 4 placed in the support structure 6 on top of the image sensor 5. Further, in this embodiment, the support structure 6 also includes a 14 mm thick rear end, as shown on the left side of Figures 2 and 3, for accommodating the circuitry of the image sensor 5. The transparent elastomer 2 is made of Smooth On Solaris transparent silicone, cast directly onto the mask and support structure to form a trapezoidal shape, approximately 4.8 mm high, 35 mm long, and 28 mm wide. The reflective film 1 is prepared by mixing Smooth-On EcoFlex addition-cured silicone with a Shore hardness of 00-10, Smooth-On Silc Pig white silicone pigment, and Smooth-On NOVOCS organosilicon solvent, and is uniformly sprayed in three layers using a spray gun. The light source 3 uses four colors of LED devices (red, green, blue, and white), which are evenly placed on the support structure 6 around the separable amplitude mask 4, with one color of light source placed on each of the four sides.

[0093] This application also provides a measurement method based on the above-described visual-tactile sensor, which may include, but is not limited to, the following steps:

[0094] 1) Calibrate the system matrix of the lensless imaging system within the visual-tactile sensor.

[0095] In this embodiment, the system matrix P of the lensless imaging system is first defined. c Q c P o Q o Calibration is performed. Here, P represents the left-multiplication matrix, Q represents the right-multiplication matrix, the subscript c indicates that the system matrix contains coded components, and the subscript o indicates that the system matrix is ​​the image sensor's response to a point light source without a mask. The pixel size of the reconstructed image from the lensless imaging system is N×M. These four system matrices depend only on the lensless imaging system. It is important to note that this calibration process should be performed before the visual-tactile sensor is completely covered by a reflective film on the transparent elastomer surface. The specific steps are as follows:

[0096] 1-1) Image acquisition for calibration; the specific steps are as follows:

[0097] 1-1-1) Place the visual-touch sensor without the reflective film covering it parallel to a pixel size of N. x ×M x The display is positioned in front of the sensor, with its horizontal and vertical axes aligned with those of the lensless imaging system. The display is flush against the surface of the transparent elastomer of the haptic sensor. The display screen must be larger than the transparent elastomer, and its pixel size is generally smaller than the pixel size of the reconstructed image.

[0098] 1-1-2) Creating an image collection Where the i-th image A in this set i This is an image where only the pixels in the i-th row are white, and the rest are black.

[0099] 1-1-3) Creating an image collection Where the i-th image in this set is B i This is an image where only the pixels in the i-th column are white and the rest are black.

[0100] 1-1-4) The display is arranged from 1 to N x The image collection is displayed in sequence. Each image in the image set is acquired by a lensless imaging system within the visual-touch sensor. calculate The set consists of the average brightness of the entire image in each image. Where c i For image C i The average brightness of the entire image, for Taking the derivative, we obtain the image index i = e corresponding to the center point of the brightness mean. Then, we select the image set. The images in this set are then renumbered sequentially from smallest to largest to form the first calibration image set. Where Y hi Let i be the i-th image in the image set.

[0101] 1-1-5) The monitor is arranged from 1 to M x The image collection is displayed in sequence. Each image in the image set is acquired by a lensless imaging system within the visual-touch sensor. calculate The set consists of the average brightness of the entire image in each image. Where d i For image D i The average brightness of the entire image, for Taking the derivative, we obtain the image index i = g corresponding to the center point of the brightness mean. Then, we select the image set. The images in this set are then renumbered in ascending order to form the second calibration image set. Where Y vi This refers to the i-th image in the second calibration image set.

[0102] In one specific embodiment of the present invention, the display is a 5.5-inch LCD display with a resolution of 3840×2160. The reconstructed image resolution is 512×400, thus the first calibration image set is finally acquired. Second calibration image set

[0103] 1-2) Calculate the calibration matrix; the specific steps are as follows:

[0104] 1-2-1)p oi ,p ci ,q oi ,q ci P respectively o P c Q o Q c The i-th column, p o ,p c ,q o ,q c P respectively o P c Q o Q c The vector obtained by summing each row.

[0105] 1-2-2) Suppose that, given a first calibration image set and q o ,q c Calculate P under the condition o P c The process is as follows:

[0106] Let i range from 1 to N, and calculate:

[0107] Get P o =[p o1 ,p o2 ,…,p oN ], P c =[p c1 ,p c2 ,…,p cN ];

[0108] Suppose that under the given conditions and p o ,p c Calculate Q under the condition o Q c The process is as follows:

[0109] In the formula, For Y vi Transpose of;

[0110] Where i ranges from 1 to M, and the following calculations are performed:

[0111] Get Q o =[q o1 ,q o2 ,…,q oM ], Q c =[q c1 ,q c2 ,…,q cM ];

[0112] 1-2-3) Generate q o The initial estimate and P are obtained. o ;

[0113] Where, let q o =1, that is, a column vector whose elements are all 1s. Take i from 1 to N and calculate the following formula:

[0114] Get P o =[p o1 ,p o2 ,…,p oN ];

[0115] 1-2-4) Using calibration images Estimate q c ;

[0116] in for:

[0117] right Performing singular value decomposition, we obtain:

[0118] Where U, Σ, and V are The resulting matrix after singular value decomposition;

[0119] Take q c Equal to the first column of V, and take i from 1 to N and repeat the following calculation: Y hi =Y hi -p oi 1 T

[0120] Get P c =[p c1 ,p c2 ,…,p cN].

[0121] 1-2-5) Based on P o P c Initial value, calculate Q o Q c .

[0122] In this embodiment, let p o =P o 1, p c =P C 1;

[0123] Depend on Get Q o Q c .

[0124] 1-2-6) Iteratively optimize the system matrix.

[0125] P o P c Q o Q c Rename them to P oa P ca Q oa Q ca As the system matrix before optimization, first scale P. oa Q oa Make the Frobenius norm (F-norm) of both equal, then scale P. ca Q ca Make the Frobenius norm (F-norm) of both equal. Let q o =Q oa 1, q c =Q ca 1. By The optimized system matrix P is obtained ob ,P cb Let p o =P ob 1, p c =P CB 1. By

[0126] The optimized system matrix Q is obtained. OB Q CB Calculate the system matrix P before optimization. Oa P ca Q oa Q cA With the optimized system matrix P OB P CB Q OB Q CBIf the difference between the intervals converges or reaches the maximum number of iterations, the optimization is considered complete, and the optimized P is... OB P CB Q ob Q cb As a system matrix for lensless imaging systems.

[0127] In one implementation, the 912 collected calibration images are processed, a convergence value of 0.0001 is selected, and the maximum number of iterations is 10 to obtain the optimized calibration system matrix.

[0128] 2) Perform joint filter optimization on the calibrated lensless imaging system; the specific steps are as follows:

[0129] 2-1) For the scene to be tested X and the observed image Y of the lensless imaging system within the visual-tactile sensor, the following relationship exists: Y = Y c +Y o (1)

[0130] Among them, Y c Y is a rich coding component of the observation results. o This refers to the undercoded components of the observation results.

[0131] 2-2) Design a frequency domain filtering unit for the observed image Y Spatial filtering unit The frequency-spatial joint filter obtained by concatenating the two get:

[0132] in, It is a frequency domain filtering unit The result of Y-filtering the observed image, It is a spatial filtering unit The result of Y-filtering the observed image, It is a joint filter The result of Y-filtering the observed image; Φ f It is the parameter matrix of the frequency domain filtering unit, with the same dimensions as Y; and These are the discrete cosine transform and its inverse transform; ⊙ represents the Hadamard product of the matrix; Φ s It is the parameter matrix of the spatial filtering unit, with the same dimension as Y;

[0133] The objective of this frequency-spatial joint filter is:

[0134] Result: Y c =YY o (8)

[0135] 2-3) Prepare the image set of the scene to be tested This image set contains K scene images of size N×M, where X i Let i be the i-th image of the scene to be tested in the image set of the scene to be tested;

[0136] Among them, for the scene image X to be tested i ,exist: Y i =Y ci +Y oi (11)

[0137] Since only the parameter matrix of the joint filter is unknown, optimization methods can be used to optimize the joint filter parameter matrix to obtain... The optimization results.

[0138] 3) Calibrate the visual-tactile sensor; the specific steps are as follows:

[0139] 3-1) Before calibrating the visual-tactile sensor, cover the surface of the transparent elastomer with a reflective film.

[0140] 3-2) Calibrate the visual-tactile sensor; the specific steps are as follows:

[0141] 3-2-1) Press balls of different sizes onto the surface of the reflective film at different depths, causing the reflective film and the transparent elastomer underneath it to deform.

[0142] 3-2-2) The light projected by the light source passes through the transparent elastomer and shines on the deformed reflective film. The reflected light passes through the transparent elastomer and shines on the separable amplitude mask.

[0143] 3-2-3) A separable amplitude mask modulates the amplitude of the reflected light. The modulated light illuminates the image sensor, and the lensless imaging system generates the corresponding image signal.

[0144] 3-2-4) Based on the image signal from step 3-2-3), perform image reconstruction to obtain the reconstructed image signal.

[0145] In this embodiment, the image reconstruction method is as follows:

[0146] For any scene image X to be measured, the observed image obtained after observation by a lensless imaging system is Y, and the corresponding reconstructed image is...

[0147] The joint filter obtained based on the joint filter acquires the rich coding components of the observed image, namely: Y c =YYo (14)

[0148] Solve the following optimization problem:

[0149] Where τ is the regularization coefficient, Let F be the F-norm of the matrix;

[0150] The solution to this optimization problem is:

[0151] Equations (16) and (17) are for P c Q c Each performs singular value decomposition;

[0152] σ P With σ Q respectively using and A column vector consisting of the diagonal elements;

[0153] . / represents element-wise division of a matrix, and 1 represents a column vector with all elements being 1; The solution is the image reconstruction result.

[0154] 3-2-5) The light intensity-gradient-depth mapping relationship is obtained from the reconstructed image signal, the size of the ball and the pressing depth.

[0155] The calibration process for the visual-tactile sensor is now complete.

[0156] In one specific embodiment of the present invention, six steel spheres with diameters of 5–10 mm are selected and uniformly distributed. 150 positions are randomly selected on a transparent film, and pressure is applied at a distance equal to half the diameter of the spheres. In this embodiment, a lookup table method is used to establish the light intensity-gradient-depth mapping relationship.

[0157] 4) Use the calibrated visual-tactile sensor for measurement; the specific steps are as follows:

[0158] 4-1) The reflective film of the visual-tactile sensor is pressed against the surface of the object being measured with a certain pressure, causing deformation of the reflective film and the transparent elastomer beneath it. The pressure can be applied to any part of the sensor's support system. The minimum pressure should cause a localized change in the surface normal of the reflective film; the maximum pressure should be less than the pressure value that would cause any damage to the sensor or the surface being measured.

[0159] 4-2) The light projected by the light source passes through the transparent elastomer and shines on the deformed reflective film. The reflected light passes through the transparent elastomer and shines on the separable amplitude mask.

[0160] 4-3) A separable amplitude mask modulates the amplitude of the reflected light. The modulated light illuminates the image sensor, and the lensless imaging system generates the corresponding image signal.

[0161] 4-4) Use image reconstruction methods to reconstruct the image signal from step 4-3) to obtain the intensity of the reflected light before modulation.

[0162] 4-5) Calculate the deformation depth generated by the reflective film based on the light intensity-gradient-depth relationship calibrated in step 3), thereby obtaining the corresponding measurement result of the pressing depth of the object under test. The measurement process is now complete.

[0163] In one implementation, when measuring an object, the pressure is applied at the rear end of the support structure at a pressure of 3N. A reflective film deforms upon contact with the surface of the object being measured. Light generated by a light source passes through a transparent elastomer, illuminates the deformed reflective film, and is reflected through a separable amplitude mask before being captured by an image sensor. The captured image signal is then reconstructed to obtain a reconstructed image. In this embodiment, the Poisson equation is used to reconstruct the depth result based on the light intensity-gradient-depth mapping relationship recorded in a lookup table.

[0164] By implementing the embodiments of this application, a calibration and measurement method for a lensless imaging-based visual-touch sensor is adopted, enabling the visual-touch sensor to be used to acquire the depth of the measured object and realize visual-touch function. The system matrix calibration method and high-speed non-iterative image fast reconstruction method of the lensless imaging system in the visual-touch sensor use analytical solutions to describe the reconstruction imaging process. The required calculation matrices can be obtained in advance, so they can be matched with the sensing speed of the image sensor to perform high-speed reconstruction solution. This overcomes the problem that existing methods require multiple iterations and cannot reconstruct in real time, greatly improving imaging efficiency.

Claims

1. A visual-tactile sensor based on lensless imaging, characterized in that, The visual-tactile sensor includes: a lensless imaging system consisting of a separable amplitude mask, an image sensor, and a support structure; a reflective film, a transparent elastomer, and multiple light sources of different colors; in the lensless imaging system, the support structure is used to fix the separable amplitude mask and the image sensor respectively; the separable amplitude mask is placed in front of the image sensor, with their center lines aligned and their horizontal and vertical directions aligned; light from the scene under test passes through the separable amplitude mask and illuminates the image sensor, and the image sensor outputs a corresponding observed image to achieve image reconstruction of the scene under test; The transparent elastomer is located on the side of the incident light of the lensless imaging system, covering and closely adhering to the separable amplitude mask; the reflective film covers the surface of the transparent elastomer away from the separable amplitude mask; each of the light sources is uniformly placed on the support structure around the separable amplitude mask to ensure that the light emitted by the light source is reflected by the reflective film and then passes through the transparent elastomer to irradiate the separable amplitude mask.

2. The visual-tactile sensor according to claim 1, characterized in that, The number of colors of the light source is greater than or equal to 3.

3. The visual-tactile sensor according to claim 1, characterized in that, The transparent elastomer is made of an elastic material with a light transmittance greater than 80% at visible light frequencies, and is in the shape of a flat plate or a trapezoidal platform.

4. The visual-tactile sensor according to claim 1, characterized in that, The reflective film is a coating or film with a refractive index greater than 50% for visible light frequencies, which is applied to the surface of the transparent elastomer by spraying or adhesive.

5. A measurement method based on the visual-tactile sensor as described in any one of claims 1-4, characterized in that, The method includes: During the construction of the visual-tactile sensor, before the reflective film is covered on the surface of the transparent elastomer, the system matrix of the lensless imaging system within the visual-tactile sensor is calibrated. Joint filter optimization is performed on the calibrated lensless imaging system; The reflective film is covered on the surface of the transparent elastomer, and the visual-tactile sensor is calibrated based on the optimization results of the joint filter. Measurements were performed using the calibrated visual-tactile sensor.

6. The measurement method according to claim 5, characterized in that, The system matrix for calibrating the lensless imaging system within the visual-tactile sensor includes: 1) Image acquisition calibration; the specific steps are as follows: 1-1) Place the visual-touch sensor without the reflective film covering it parallel to a pixel size of N. x ×M x In front of the display, the horizontal and vertical directions of the display are consistent with the horizontal and vertical directions of the lensless imaging system, and the display is in close contact with the transparent elastomer surface of the visual-tactile sensor; the display screen is larger than the transparent elastomer, and the pixel size of the display is smaller than the pixel size of the reconstructed image; 1-2) Create an image collection Where the i-th image A in this set i This is an image where only the pixels in the i-th row are white, and the rest are black. 1-3) Creating an image collection Where the i-th image in this set is B i This is an image where only the pixels in the i-th column are white, and the rest are black. 1-4) The display is arranged from 1 to N x The image collection is displayed in sequence. Each image in the image set is acquired by a lensless imaging system within the visual-touch sensor. calculate The set consists of the average brightness of the entire image in each image. Where c i For image C i The average brightness of the entire image, for Taking the derivative, we obtain the image index i = e corresponding to the center point of the brightness mean. Then, we select the image set. The images in this set are then renumbered sequentially from smallest to largest to form the first calibration image set. Where Y hi Let i be the i-th image in the image set; 1-5) The monitor is arranged from 1 to M x The image collection is displayed in sequence. Each image in the image set is acquired by a lensless imaging system within the visual-touch sensor. calculate The set consists of the average brightness of the entire image in each image. Where d i For image D i The average brightness of the entire image, for Taking the derivative, we obtain the image index i = g corresponding to the center point of the brightness mean. Then, we select the image set. The images in this set are then renumbered in ascending order to form the second calibration image set. Where Y vi The i-th image in the second calibration image set; 2) Calculate the calibration matrix; the specific steps are as follows: 2-1) Let the system matrix of the lensless imaging system be P c Q c P o Q o Where P represents the left multiplication matrix, Q represents the right multiplication matrix, the subscript c indicates that the system matrix contains coded components, and the subscript o indicates that the system matrix is ​​the response of the image sensor to a point light source without a mask; let p oo ,p ci ,q oi ,q ci P respectively o P c Q o Q c The i-th column, p o ,p c ,q o ,q c P respectively o P c Q o Q c The vector obtained by summing each row; 2-2) Suppose that, given a first calibration image set and q o ,q c Calculate P under the condition o P c The process is as follows: Let i range from 1 to N, and calculate: Get P o =[p o1 ,p o2 ,…,p oN ], P c =[p c1 ,p c2 ,…,p cN ]; Suppose that under the given conditions and p o ,p c Calculate Q under the condition o Q c The process is as follows: In the formula, For Y vi Transpose of; Where i ranges from 1 to M, and the following calculations are performed: Obtain Q o = [q o1 , q o2 , …, q oM , Q c = [q c1 , q c2 , …, q cM ; 2-3) Generate q o The initial estimate and P are obtained o ; Where, let q o =1, that is, a column vector whose elements are all 1s. Take i from 1 to N and calculate the following formula: Get P o =[p o1 ,p o2 ,…,p oN ]; 2-4) Using calibration images Estimate q c ; in for: right Performing singular value decomposition, we obtain: Where U, Σ, and V are The resulting matrix after singular value decomposition; Take q c The first column equals V, and the following formula is calculated repeatedly for i from 1 to N: AND hi =Y hi -p oi 1 T Get P c =[p c1 ,p c2 ,…,p cN ]; 2-5) Based on P o P c Initial value, calculate Q o Q c ; Where, let p o =P o 1, p c =P c 1; Depend on Get Q o Q c ; 2-6) Iteratively optimize the system matrix; P o P c Q o Q c Rename them to P oa P ca Q oa Q ca The system matrix before optimization; First, scale P. oa Q oa Make the F-norms of both equal, then scale P. ca Q ca Make their F-norms equal; Let q o =Q oa 1, q c =Q ca 1. By The optimized system matrix P is obtained ob ,P cb ; Let p o =P ob 1, p c =P cb 1. By The optimized system matrix Q is obtained. ob Q cb ; Calculate the system matrix P before optimization oa P ca Q oa Q ca With the optimized system matrix P ob P cb Q ob Q cb If the difference between the intervals converges or reaches the set maximum number of iterations, the optimization is complete, and the optimized P is obtained. ob P cb Q ob Q cb Let this be denoted as the system matrix of the lensless imaging system.

7. The measurement method according to claim 6, characterized in that, The joint filter optimization of the calibrated lensless imaging system includes: 1) For the scene to be tested X and the observed image Y from the lensless imaging system within the visual-tactile sensor, the following relationship exists: Y / Y c +And o (1) Among them, Y c Y is a rich coding component of the observation results. o The undercoded components of the observation results; 2) Design a frequency domain filtering unit for the observed image Y. Spatial filtering unit The frequency-spatial joint filter obtained by concatenating the two get: in, It is a frequency domain filtering unit The result of Y-filtering the observed image, It is a spatial filtering unit The result of Y-filtering the observed image, It is a joint filter The result of Y-filtering the observed image; Φ f It is the parameter matrix of the frequency domain filtering unit, with the same dimension as Y; and These are the discrete cosine transform and its inverse transform; ⊙ represents the Hadamard product of the matrix; Φ s It is the parameter matrix of the spatial filtering unit, with the same dimension as Y; The objective of this frequency-spatial joint filter is: get: AND c =YY o (8) 3) Prepare the image set of the scene to be tested. This image set contains K scene images of size N×M, where X i Let i be the i-th image of the scene to be tested in the image set of the scene to be tested; Among them, for the scene image X to be tested i The following relationship exists: AND i =Y ci +And oi (11) The joint filter parameter matrix is ​​optimized to obtain The optimization results.

8. The measurement method according to claim 7, characterized in that, The calibration of the visual-tactile sensor based on the optimization results of the joint filter includes: 1) Press balls of different sizes onto the surface of the reflective film at different depths, causing deformation of the reflective film and the transparent elastomer underneath it; 2) The light projected by the light source passes through the transparent elastomer and shines on the deformed reflective film. The reflected light then passes through the transparent elastomer and shines on the separable amplitude mask. 3) A separable amplitude mask modulates the amplitude of the reflected light, and the modulated light illuminates the image sensor, generating a corresponding image signal in the lensless imaging system; 4) Based on the image signal from step 3), perform image reconstruction to obtain the reconstructed image signal; 5) The light intensity-gradient-depth mapping relationship is obtained from the reconstructed image signal, the size of the ball and the pressing depth, so as to complete the calibration of the visual tactile sensor.

9. The measurement method according to claim 8, characterized in that, The measurement using the calibrated visual-tactile sensor includes: 1) Press the reflective film of the visual-tactile sensor onto the surface of the object being measured, and the reflective film and the transparent elastomer underneath the reflective film will deform; 2) The light projected from the light source passes through the transparent elastomer and irradiates the deformable reflective film. The reflected light passes through the transparent elastomer and irradiates the separable amplitude mask. 3) A separable amplitude mask modulates the amplitude of the reflected light, and the modulated light illuminates the image sensor, generating a corresponding image signal in the lensless imaging system; 4) Based on the image signal from step 3), perform image reconstruction to obtain the intensity of the reflected light before modulation; 5) Calculate the deformation depth of the reflective film based on the calibrated light intensity-gradient-depth relationship to obtain the corresponding measurement result of the pressure depth of the object being tested.

10. The measurement method according to claim 9, characterized in that, The image reconstruction includes: 1) For any scene image X to be measured, the observed image obtained after observation by the lensless imaging system is Y, and the corresponding reconstructed image is The rich coding components of the observed image are obtained based on the joint filter, namely: AND c =YY o (14) Solve the following optimization problem: Where τ is the regularization coefficient, Let F be the F-norm of the matrix; The solution to this optimization problem is: Equations (16) and (17) are for P c Q c Each performs singular value decomposition; σ P With σ Q respectively using and A column vector consisting of the diagonal elements; . / represents element-wise division of a matrix, and 1 represents a column vector with all elements being 1; The solution is the image reconstruction result.

Images