A visual-tactile sensor and a multi-object hardness real-time measurement method thereof

Abstract

A kind of visual tactile sensor and its multi-object hardness real-time measurement method belong to robot tactile and hardness measurement field.It includes camera support frame, light source, monocular camera and tactile perception module, monocular camera is installed in camera support frame side by support plate, light source is used to illuminate camera support frame inside;Tactile perception module is composed of deformable gel layer, mark layer and reflective protective layer, one side of deformable gel layer is installed in the other side of camera support frame by transparent connecting plate, the distance between deformable gel layer and monocular camera is not less than 2 times camera focal length;The other side of deformable gel layer is marked with random pixel as mark layer, reflective protective layer is attached to the remaining surface of deformable gel layer except the surface opposite to monocular camera.The visual tactile sensor structure presented in the application is simple, wide range, can detect the hardness of multiple objects simultaneously, easy to operate.It is more suitable for perception tasks in complex environments such as robot grasping, dexterous hand operation.

Description

Technical Field

[0001] This invention belongs to the field of robot tactile and hardness measurement, and specifically relates to a visual tactile sensor and a method for real-time measurement of the hardness of multiple objects. Background Technology

[0002] Robotic tactile sensing is an important branch of the field of robotics and is one of the essential functions of embodied intelligence.

[0003] Traditional tactile sensors use an array of multiple force sensors to calculate force distribution through interpolation. However, most of these sensors are quite rigid and cannot compare to skin. With the development of flexible sensors, various electronic skin-like sensors such as FPC, flexible capacitive, and flexible piezoelectric sensors have emerged. However, they still have not overcome the shortcomings of array sensors, such as low spatial resolution, limited functionality, cable interference, and wiring difficulties.

[0004] Vision-tactile sensors mimic human tactile receptors in a completely different way, measuring multiple tactile information while achieving dense reconstruction. Current vision-tactile sensors can be categorized by camera type into monocular vision-tactile sensors, represented by GelSight and GelSlim, and binocular vision-tactile sensors, represented by GelStereo. However, most existing technologies rely on deep learning algorithms to extract features for computation, resulting in slow sensor response speeds and dynamic characteristics far lower than array-type sensors. Furthermore, existing vision-tactile sensors do not fully utilize their multimodal tactile sensing capabilities, limiting themselves to measuring traditional tactile quantities such as object surface shape, contact force, and sliding sensation, and have not yet achieved the measurement of object hardness. However, object hardness information plays a crucial role in robotic arm operation feedback and is key to robot perception and grasping. Additionally, most existing hardness measurement methods employ probe methods, which damage the surface of the object being measured, and existing hardness testers are generally large, making them difficult to integrate into the dexterous hand of a robotic arm and unsuitable for complex environments such as robotic tactile perception.

[0005] Therefore, how to achieve accurate universal hardness perception of multiple objects in complex and ever-changing environments is a challenging problem in the field of robotic tactile sensing. Summary of the Invention

[0006] To address the aforementioned issues, the present invention aims to provide a visual-tactile sensor and a method for real-time measurement of the hardness of multiple objects. This method solves the problems of complex traditional hardness measurement methods and the inability of visual-tactile sensors to measure the hardness of objects in real time. While ensuring the basic morphology and three-dimensional force measurement capabilities of the visual-tactile sensor, it also meets the requirements for real-time measurement and feedback of the hardness information of multiple objects in complex environments such as robot grasping. This improves the tactile perception capability of robots and provides multimodal, high-real-time tactile perception for embodied intelligent robots.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A visual-tactile sensor for real-time measurement of the hardness of multiple objects includes a camera support frame, a light source, a monocular camera, and a tactile sensing module. The monocular camera is mounted on one side of the camera support frame via a support plate, and the light source is used to illuminate the interior of the camera support frame.

[0009] The tactile sensing module comprises a deformable gel layer, a marker layer, and a reflective protective layer. One side of the deformable gel layer is mounted on the other side of the camera support frame via a transparent connecting plate, with the distance between the deformable gel layer and the monocular camera being no less than twice the camera's focal length. The other side of the deformable gel layer is marked with random pixels as a marker layer. The reflective protective layer is attached to all sides of the deformable gel layer except for the side facing the monocular camera. The random pixels can be directly printed onto the side of the deformable gel layer using water transfer printing or screen printing. Compared to marker points, random pixels provide more accurate dense optical flow calculations, improving the accuracy of depth, force, and hardness estimations.

[0010] Furthermore, the camera support frame is made of an opaque material, and the light source is installed inside the camera support frame.

[0011] Furthermore, the light source is a monochromatic light source.

[0012] Furthermore, the random pixels are distributed across the entire side of the deformable gel layer, and the color of each pixel is random.

[0013] Furthermore, the deformable gel layer has a transparent structure and a Shore A hardness between 20 and 40.

[0014] Furthermore, the reflective protective layer is a metal layer, preferably aluminum, silver, or platinum, and is prepared by sputtering using an ion sputtering machine.

[0015] The multi-object hardness real-time measurement method based on the above-mentioned visual-tactile sensor includes:

[0016] (1) The deformed gel map under the pressure of multiple objects is acquired by a monocular camera as a tactile perception image, and the undeformed initial gel map is used as a reference image. The optical flow field is calculated based on the tactile perception image and the reference image. Preferably, the tactile perception image and the reference image are filtered, sharpened and grayscaled before calculating the optical flow field.

[0017] (2) Using the optical flow field described in the Helmholtz decomposition step, the Helmholtz decomposition vector field is obtained, including the irrotational field, the source-free field and the harmonic field; the Helmholtz decomposition vector field obtained by decomposition is input into the pre-trained three-dimensional force distribution estimation model to solve for the contact three-dimensional force information, and the magnitude of the three-dimensional force is adjusted to meet the hardness measurement requirements.

[0018] (3) Under the condition of three-dimensional force magnitude that meets the hardness measurement requirements, update the tactile perception image and calculate the optical flow field. Calculate the two-dimensional Gaussian density distribution map according to the optical flow field amplitude based on the updated optical flow field. Identify the contact area based on the two-dimensional Gaussian density distribution map and remove invalid contact areas according to a preset area threshold, retaining only the effective contact areas. Calculate the contact area of ​​each region based on the boundary of the effective contact areas. Using the two-dimensional Gaussian density distribution map of the effective contact areas as input, solve the contact depth of the effective contact areas using a pre-trained depth estimation model. The number of effective contact areas is the number of actual contact objects.

[0019] (4) Calculate the hardness influence factor based on the contact area and contact depth of each effective contact area, and use the pre-trained hardness estimation model to solve the hardness value of each effective contact area to realize real-time measurement of the hardness of multiple objects.

[0020] Furthermore, the three-dimensional force distribution estimation model and the hardness estimation model adopt a polynomial fitting model, and the depth estimation model adopts a polynomial fitting model or a multilayer perceptron.

[0021] The pre-training process for the aforementioned three-dimensional force distribution estimation model, hardness estimation model, and depth estimation model is as follows:

[0022] A set of tactile information of a series of standard hardness value indenter samples is obtained. The set of tactile information includes tactile perception images of indenter samples with different gradient three-dimensional forces, different gradient contact depths, and different hardness values. The effective contact area corresponding to each tactile perception image is labeled with its contact area, contact depth, and three-dimensional force distribution. The two-dimensional Gaussian density distribution map, Helmholtz decomposition vector field, and hardness influence factor of the effective contact area corresponding to each tactile perception image are calculated to construct a training sample set.

[0023] Using the irrotational field, source-free field, and harmonic field in the Helmholtz decomposition vector field of the training sample set as inputs and the contact three-dimensional force information as outputs, the three polynomial fitting formulas of the three-dimensional force distribution estimation model are trained based on the true value of the three-dimensional force distribution to obtain the pre-trained three-dimensional force distribution estimation model.

[0024] Using the two-dimensional Gaussian density distribution map of the effective contact area in the training sample set as input and the depth of each pixel in the effective contact area as output, the polynomial fitting formula or multilayer perceptron of the depth estimation model is trained according to the true contact depth of the effective contact area to obtain the pre-trained depth estimation model.

[0025] Using the hardness influence factor of each effective contact area in the training sample set as input and the hardness value of each effective contact area as output, the polynomial fitting formula of the hardness estimation model is trained according to the true hardness value to obtain the pre-trained hardness estimation model.

[0026] Furthermore, the formula for calculating the hardness influence factor is as follows: or Where H represents the hardness influence factor of the effective contact area, A represents the contact area of ​​the effective contact area, and d represents the contact depth of the effective contact area.

[0027] Furthermore, the polynomial fitting model includes, but is not limited to, first-order, second-order, third-order, and fourth-order polynomials.

[0028] The beneficial effects of this invention are:

[0029] (1) The visual-touch sensor proposed in this invention uses only a monochromatic light source and a monocular camera. It has a simple structure, is easy to manufacture, has low cost, can detect the hardness of multiple objects at once, and is easy to operate.

[0030] (2) This invention uses the Hertzian contact principle and the theory of nonlinear continuous media to derive a hardness measurement formula under linear large deformation, which has good real-time performance and dynamic performance.

[0031] (3) The hardness measurement method proposed in this invention can measure Shore A (type A hardness) 10-90, with a wide range. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the structure of a visual-tactile sensor provided by the present invention.

[0033] Figure 2 This is a flowchart illustrating the steps of real-time multi-object hardness measurement based on a visual-tactile sensor proposed in this invention.

[0034] Figure 3 It is a tactile perception map and its depth map obtained by using the visual-tactile sensor made in this invention.

[0035] Figure 4 This is an exploded view of the structure of the visual-tactile sensor described in this invention.

[0036] Figure 5This is a schematic diagram of the contact conditions to which the hardness measurement formula proposed in this invention is applicable.

[0037] Figure 6 This invention relates to a multi-object hardness measurement range and its accuracy diagram based on a visual-tactile sensor. Detailed Implementation

[0038] The present invention will be further described in detail below with reference to preferred embodiments.

[0039] It should be added that the specific embodiments described herein are only used to explain the visual-tactile sensor and its real-time multi-object hardness measurement method involved in the present invention, and are not intended to limit the present invention and its implementation.

[0040] Referring to the present invention Figure 1 , 4 The basic structure of the visual-touch sensor shown includes a camera support frame 1, a light source 2, a monocular camera 7, and a touch sensing module 8. The monocular camera 7 is mounted on one side of the camera support frame 1 via a support plate 6, and the light source 2 is used to illuminate the inside of the camera support frame 1.

[0041] The tactile sensing module 8 consists of a deformable gel layer 3, a marker layer 5, and a reflective protective layer 4. One side of the deformable gel layer 3 is mounted on the other side of the camera support frame 1 via a transparent connecting plate. The distance between the deformable gel layer 3 and the monocular camera 7 is not less than twice the camera focal length. The other side of the deformable gel layer 3 is marked with random pixels as the marker layer 5. The reflective protective layer 4 is attached to the remaining sides of the deformable gel layer 3 except for the side facing the monocular camera 7.

[0042] like Figure 4 As shown, this embodiment uses polydimethylsiloxane (PDMS) as the deformable gel layer 3, with its Shore A hardness controlled within the range of 20-40 and a size of 34×34mm. Random pixels in the marking layer are printed onto the bottom surface of the PDMS using a water transfer printing method to sense tactile changes. These random pixels are distributed across the entire side of the deformable gel layer, and each pixel has a random color. The hardness of the deformable gel layer can be adjusted by changing the ratio of PDMS to curing agent, curing time, and curing temperature according to actual needs. The deformable gel layer can be made of any transparent silicone rubber with a Shore A hardness not exceeding 40.

[0043] In this embodiment, a platinum layer is used as the reflective protective layer 4, which is attached to the bottom surface of the PDMS by ion sputtering. Its thickness is no more than 1 mm. This layer is used to improve the gel's ability to perceive tactile images and enhance the internal light intensity while blocking external light. Of course, the platinum layer 8 can also be any sputterable metal, including but not limited to aluminum, silver, gold, and copper.

[0044] In this embodiment, a transparent acrylic support plate is used as a connecting plate, and an opaque polylactic acid support frame or a machined metal frame is used as the camera support frame 1. The transparent acrylic support plate is first bonded to PDMS, and then fixed to the opaque support frame with a nut assembly. The support frame has dimensions of 35×35×40mm.

[0045] In this embodiment, a strip-shaped white LED light source is used as light source 2, which is connected to an external power supply via a USB (Universal Serial Bus) interface to power the light source. The LED light source is adhered inside the support frame to increase the light intensity inside the sensor.

[0046] In this embodiment, the monocular camera 7 is fixedly connected to the camera support frame 1 via a support plate 6, and the distance between the camera lens and the acrylic support plate is 36mm. The camera is connected to a host PC via a USB (Universal Serial Bus) interface. The support plate 6 can be manufactured using CNC machining.

[0047] like Figure 2 As shown, the real-time measurement method for multiple objects based on a visual-tactile sensor proposed in this invention includes the following specific steps:

[0048] (1) An undeformed initial gel image was acquired using an RGB monocular camera as a reference image, while the deformed gel image under the pressure of multiple objects was used as a tactile perception image, such as... Figure 3 As shown in (a) and (b), the optical flow field is calculated after preprocessing the tactile perception image and the reference image.

[0049] In one specific embodiment of the present invention, the tactile perception image and the reference image are filtered, sharpened, and grayscaled to enhance the features of the deformed region. The filtering algorithms include, but are not limited to, Gaussian filtering, mean filtering, or guided filtering. Filtering, sharpening, and grayscale processing can also be directly implemented using OpenCV (Open Source Computer Vision Library) algorithms. Optical flow calculation can employ the Lucas-Kanade optical flow algorithm or the DIS (Dense Inverse Search) optical flow algorithm. Preprocessing and optical flow calculation are common knowledge in the art and will not be elaborated upon here.

[0050] (2) Using the Helmholtz decomposition method, the above optical flow field is decomposed into an irrotational field, a source-free field and a harmonic field. The three vector fields obtained by the above decomposition are input into the pre-trained three-dimensional force distribution estimation model to solve for the contact three-dimensional force information. The pressure of multiple objects is adjusted to change the magnitude of its three-dimensional force until the hardness measurement requirements are met.

[0051] The Helmholtz decomposition method is as follows: The optical flow field calculated in step (1) is... The irrotational field, divergence-free field, and harmonic field obtained from the decomposition of the optical flow field are defined as follows:

[0052] In a specific embodiment of the present invention, the pre-trained three-dimensional force distribution estimation model adopts a linear fitting formula, y=ax+b. (3) Under the condition of three-dimensional force magnitude that meets the hardness measurement requirements, the tactile perception image is updated and the optical flow field is calculated. Based on the updated optical flow field, a two-dimensional Gaussian density distribution map according to the optical flow field amplitude distribution is calculated. The contact area is identified based on the two-dimensional Gaussian density distribution map, and the invalid contact area is removed according to the preset area threshold, leaving only the effective contact area. The contact area of ​​each region is calculated based on the boundary of the effective contact area. Using the two-dimensional Gaussian density distribution map of the effective contact area as input, the contact depth of the effective contact area is solved using the pre-trained depth estimation model. The depth map is shown in the figure. Figure 3 As shown in (c) and (d); the number of effective contact areas refers to the number of actual contacting objects. In this embodiment, invalid contact areas are removed according to a minimum area threshold of 1000 (per pixel).

[0053] In one specific embodiment of the present invention, the two-dimensional Gaussian density distribution map based on the optical flow field amplitude distribution is calculated using the two-dimensional Gaussian density estimator Freud (an open-source particle density estimation algorithm).

[0054] The pre-trained depth estimation model uses a linear fitting formula, which can be expressed as y = ax + b and depth d = aD + b, where D is a two-dimensional Gaussian density that estimates the density of its Freud output.

[0055] (4) Calculate the hardness influence factor based on the contact area and contact depth of each effective contact area, and use the pre-trained hardness estimation model to solve the hardness value of each effective contact area to realize real-time measurement of the hardness of multiple objects.

[0056] In one specific embodiment of the present invention, the pre-trained stiffness estimation model adopts the quadratic fitting formula y = ax 2 +bx+c.

[0057] The aforementioned three-dimensional force distribution estimation model, depth estimation model, and hardness estimation model together constitute the tactile perception model of this invention. In a specific embodiment of this invention, the hardness influence factor H is a function of the contact area (A) and the contact depth (d), and the calculation formula is as follows: or Where A represents the contact area of ​​the effective contact region, expressed in meters. 2 The unit is d; d represents the contact depth of the effective contact area, in meters.

[0058] The aforementioned hardness influence factor H under large elastic deformation is based on, for example... Figure 5 The above is derived from the Hertzian contact principle and the theory of nonlinear continuum mechanics. The derivation process is as follows:

[0059] Depend on Figure 5 We can obtain:

[0060]

[0061] Where ω1 and ω2 are the deformation displacements of the measured object and the sensor gel when they come into contact, z1 and z2 are the perpendicular distances of the measured object and the gel plane from the contact point, the contact surface is approximated by a fourth-order curved surface, r is the equivalent radius of the contact surface, and the equivalent area of ​​the contact surface A = πr 2 E1 and E2 are the Young's moduli of the tested object and the sensor gel, respectively.

[0062] Based on the actual working conditions, the gel of this sensor is planar, so R1→+∞, z1=0, therefore we can obtain:

[0063]

[0064] Under small deformation conditions, we have:

[0065] Where S represents Shore A hardness. As can be seen from this, the hardness of the object being measured is only related to the contact area A and the contact depth d, and Include Based on experimental data, the hardness influence factor was obtained from factors A and B. or

[0066] The pre-trained models in steps (2), (3), and (4) above are obtained through the following steps:

[0067] A set of tactile information for a series of standard hardness value indenter samples is obtained. The set of tactile information includes tactile perception images of indenter samples with different gradient three-dimensional forces, different gradient contact depths, and different hardness values. The effective contact area, contact area, contact depth, and three-dimensional force distribution corresponding to each tactile perception image are labeled. The two-dimensional Gaussian density distribution map, Helmholtz decomposition vector field, and hardness influence factor of the effective contact area corresponding to each tactile perception image are calculated to construct a training sample set.

[0068] Using the irrotational field, source-free field, and harmonic field from the Helmholtz decomposition vector field in the training sample set as inputs, and the contact three-dimensional force information as output, three polynomial fitting formulas for the three-dimensional force distribution estimation model are trained based on the true value of the three-dimensional force distribution, resulting in a pre-trained three-dimensional force distribution estimation model. In this embodiment, the pre-trained three-dimensional force distribution estimation model is as follows:

[0069] positive pressure Tangential force In-plane torque

[0070] Using the two-dimensional Gaussian density distribution map of the effective contact area in the training sample set as input and the depth of each pixel in the effective contact area as output, a multinomial fitting formula or a multilayer perceptron is trained based on the ground truth contact depth of the effective contact area to obtain a pre-trained depth estimation model. In this embodiment, taking the multinomial fitting formula as an example, the pre-trained depth estimation model is as follows:

[0071] d = 0.59D + 0.05, where d is the depth and D is the density output by the two-dimensional Gaussian density estimator.

[0072] Using the hardness influence factor of each effective contact area in the training sample set as input and the hardness value of each effective contact area as output, the polynomial fitting formula of the hardness estimation model is trained based on the true hardness value to obtain the pre-trained hardness estimation model; in this embodiment, the pre-trained hardness estimation model is as follows:

[0073] S = 0.003H 2 +1.08H+2.57

[0074] Where S represents Shore A hardness and H represents the hardness influence factor proposed in this invention. In a specific embodiment of this invention, when fitting the above formula, the given error function can be MSE (Mean Squared Error) or MAE (Mean Absolute Error).

[0075] Using the multi-object hardness measurement method based on the aforementioned visual-tactile sensor, the hardness indenter curves for Shore A (10-90) can be obtained as follows: Figure 6 As shown, the error of a single test is ±8, and the RSME (Root Mean Squared Error) is 3.07.

[0076] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for real-time measurement of the hardness of multiple objects using a visual-tactile sensor, characterized in that, The visual-tactile sensor includes a camera support frame (1), a light source (2), a monocular camera (7), and a tactile sensing module (8). The monocular camera (7) is mounted on one side of the camera support frame (1) via a support plate (6), and the light source (2) is used to illuminate the inside of the camera support frame (1). The tactile sensing module (8) consists of a deformable gel layer (3), a marker layer (5), and a reflective protective layer (4). One side of the deformable gel layer (3) is mounted on the other side of the camera support frame (1) through a transparent connecting plate. The distance between the deformable gel layer (3) and the monocular camera (7) is not less than twice the focal length of the camera. The other side of the deformable gel layer (3) is marked with random pixels as the marker layer (5). The reflective protective layer (4) is attached to the remaining sides of the deformable gel layer (3) except for the side facing the monocular camera (7). Measurement methods include: (1) The deformed gel map under the action of multiple object pressure is acquired by a monocular camera (7) as a tactile perception image. The undeformed initial gel map is used as a reference image. The optical flow field is calculated based on the tactile perception image and the reference image. (2) Using the optical flow field described in step (1) of the Helmholtz decomposition, the Helmholtz decomposition vector field is obtained, including the irrotational field, the source-free field and the harmonic field; the Helmholtz decomposition vector field obtained by decomposition is input into the pre-trained three-dimensional force distribution estimation model, and the contact three-dimensional force information is obtained by solving the problem. The magnitude of the three-dimensional force is adjusted to meet the hardness measurement requirements. (3) Under the condition of three-dimensional force magnitude that meets the hardness measurement requirements, update the tactile perception image and calculate the optical flow field. Calculate the two-dimensional Gaussian density distribution map according to the amplitude distribution of the optical flow field based on the updated optical flow field. Identify the contact area based on the two-dimensional Gaussian density distribution map and remove invalid contact areas according to a preset area threshold, retaining only the effective contact areas. Calculate the contact area of ​​each region based on the boundary of the effective contact area. Using the two-dimensional Gaussian density distribution map of the effective contact area as input, solve the contact depth of the effective contact area using a pre-trained depth estimation model. The number of effective contact areas is the number of actual contact objects. (4) Calculate the hardness influence factor based on the contact area and contact depth of each effective contact area, and use the pre-trained hardness estimation model to solve the hardness value of each effective contact area to realize real-time measurement of the hardness of multiple objects.

2. The method for real-time measurement of multi-object hardness using a visual-tactile sensor according to claim 1, characterized in that, The three-dimensional force distribution estimation model and the hardness estimation model adopt a polynomial fitting model, and the depth estimation model adopts a polynomial fitting model or a multilayer perceptron.

3. The method for real-time measurement of multi-object hardness using a visual-tactile sensor according to claim 2, characterized in that, The pre-training process for the aforementioned three-dimensional force distribution estimation model, hardness estimation model, and depth estimation model is as follows: A set of tactile information of a series of standard hardness value indenter samples is obtained. The set of tactile information includes tactile perception images of indenter samples with different gradient three-dimensional forces, different gradient contact depths, and different hardness values. The effective contact area corresponding to each tactile perception image is labeled with its contact area, contact depth, and three-dimensional force distribution. The two-dimensional Gaussian density distribution map, Helmholtz decomposition vector field, and hardness influence factor of the effective contact area corresponding to each tactile perception image are calculated to construct a training sample set. Using the irrotational field, source-free field, and harmonic field in the Helmholtz decomposition vector field of the training sample set as inputs and the contact three-dimensional force information as outputs, the three polynomial fitting formulas of the three-dimensional force distribution estimation model are trained based on the true value of the three-dimensional force distribution to obtain the pre-trained three-dimensional force distribution estimation model. Using the two-dimensional Gaussian density distribution map of the effective contact area in the training sample set as input and the depth of each pixel in the effective contact area as output, the polynomial fitting formula or multilayer perceptron of the depth estimation model is trained according to the true contact depth of the effective contact area to obtain the pre-trained depth estimation model. Using the hardness influence factor of each effective contact area in the training sample set as input and the hardness value of each effective contact area as output, the polynomial fitting formula of the hardness estimation model is trained according to the true hardness value to obtain the pre-trained hardness estimation model.

4. The method for real-time measurement of multi-object hardness using a visual-tactile sensor according to claim 1 or 3, characterized in that, The formula for calculating the hardness influence factor is as follows: or Where H represents the hardness influence factor of the effective contact area, A represents the contact area of ​​the effective contact area, and d represents the contact depth of the effective contact area.

5. The method for real-time measurement of multi-object hardness using a visual-tactile sensor according to claim 1, characterized in that, The camera support frame (1) is made of opaque material, and the light source (2) is installed inside the camera support frame (1).

6. The method for real-time measurement of multi-object hardness using a visual-tactile sensor according to claim 1, characterized in that, The light source (2) mentioned above is a monochromatic light source.

7. The method for real-time measurement of multi-object hardness using a visual-tactile sensor according to claim 1, characterized in that, The random pixels are distributed on the entire side of the deformable gel layer (3), and the color of each pixel is random.

8. The method for real-time measurement of multi-object hardness using a visual-tactile sensor according to claim 1, characterized in that, The deformable gel layer (3) is a transparent structure with a Shore A hardness between 20 and 40.

9. The method for real-time measurement of multi-object hardness using a visual-tactile sensor according to claim 1, characterized in that, The reflective protective layer (4) is made of metal.

Images