Visual-tactile sensor detection method and device, visual-tactile sensor and electronic equipment

Abstract

The application provides a visual tactile sensor detection method and device, a visual tactile sensor and an electronic device. The method comprises: collecting feature point imaging information of feature points by a micro camera, wherein the feature points are distributed on the lower surface of the elastic body of the visual tactile sensor; collecting reference point imaging information of reference points by the micro camera, wherein the reference points are distributed on the surface of the light-transmitting plate; performing position deviation calculation according to the reference point imaging information and preset reference point imaging information to obtain a deviation calculation result; correcting the feature point imaging information according to the deviation calculation result to obtain corrected feature point imaging information; and substituting the corrected feature point imaging information into a pre-set mechanical model to obtain a visual tactile sensor detection result. Since the application compensates for the difference in feature point imaging information caused by the temperature rise of the micro camera by using the reference points, the detection accuracy of the visual tactile sensor is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a visual-tactile sensor detection method, apparatus, visual-tactile sensor, electronic device and storage medium. Background Technology

[0002] Based on existing research, humans primarily rely on the magnitude and distribution of contact force perceived through tactile senses to grasp objects. However, as the application fields of robots continue to expand, users' demand for robots to replace human labor in performing operations is gradually increasing. This requires robots to be able to operate objects reliably and flexibly like humans, thus creating a greater demand for tactile sensors that can endow robots with tactile perception capabilities.

[0003] To address these needs, existing technologies often use visual-tactile sensors to sense contact force, employing miniature cameras to collect deformation information of the elastic body to determine the magnitude and intensity of the contact force, thus achieving tactile perception. However, the data analysis of such visual-tactile sensors relies on image data acquired by the miniature camera. Furthermore, the miniature camera generates heat during operation, which can cause deformation of the camera lens and mounting structure, altering the camera's intrinsic and extrinsic parameters. Consequently, the image data affected by these parameters also changes, resulting in low detection accuracy for the visual-tactile sensor. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for visual-tactile sensor detection. By determining the deviation calculation results after changes in camera intrinsic and extrinsic parameters through a reference point set on a light-transmitting plate, and correcting the feature point imaging information corresponding to feature points on the elastic body based on the deviation results, this solves the problem of low detection accuracy in existing visual-tactile sensors.

[0005] In a first aspect, this application provides a visual-tactile sensor detection method, the method comprising:

[0006] Feature point imaging information of feature points is acquired by a miniature camera, wherein the feature points are distributed on the lower surface of the elastic body of the visual-tactile sensor;

[0007] The reference point imaging information of the reference point is acquired by the miniature camera, wherein the reference point is distributed on the surface of the light-transmitting plate of the visual-tactile sensor, and the light-transmitting plate is fixed to the upper surface of the elastomer;

[0008] Position deviation is calculated based on the imaging information of the reference point and the imaging information of the preset reference point, and the deviation calculation result is obtained.

[0009] The feature point imaging information is corrected based on the deviation calculation result to obtain the corrected feature point imaging information;

[0010] The corrected feature point imaging information is substituted into a pre-set mechanical model to obtain the detection results of the visual-tactile sensor.

[0011] Optionally, the visual-tactile sensor detection method provided in this application further includes:

[0012] Position deviation calculation is performed based on the imaging information of the first reference point and the imaging information of the first preset reference point to obtain a first intermediate result, wherein the reference point corresponding to the imaging information of the first reference point is located on the lower surface of the light-transmitting plate.

[0013] The position deviation is calculated based on the imaging information of the second reference point and the imaging information of the second preset reference point to obtain a second intermediate result, wherein the reference point corresponding to the imaging information of the second reference point is located on the upper surface of the light-transmitting plate.

[0014] The deviation calculation result is obtained based on the first intermediate result and the second intermediate result.

[0015] Optionally, the visual-tactile sensor detection method provided in this application further includes:

[0016] The position deviation is calculated based on the imaging information of the third reference point and the imaging information of the third preset reference point to obtain a third intermediate result. The horizontal distance from the reference point corresponding to the imaging information of the third reference point to the first side of the light-transmitting plate is less than the horizontal distance from any feature point to the first side of the light-transmitting plate.

[0017] The position deviation is calculated based on the imaging information of the fourth reference point and the imaging information of the fourth preset reference point to obtain a fourth intermediate result. The horizontal distance from the reference point corresponding to the imaging information of the fourth reference point to the second side of the light-transmitting plate is less than the horizontal distance from any feature point to the second side of the light-transmitting plate. The second side is different from the first side.

[0018] The deviation calculation result is obtained based on the third intermediate result and the fourth intermediate result.

[0019] Optionally, the visual-tactile sensor detection method provided in this application further includes:

[0020] The miniature camera captures the first mirror image information of the feature points formed on the first mirror surface;

[0021] The feature points are captured by the miniature camera to form a second mirror image on the second mirror surface, wherein both the first mirror surface and the second mirror surface are within the lens range of the miniature camera and the angles between the first mirror surface and the second mirror surface and the horizontal plane are different.

[0022] The feature point imaging information is synthesized based on the first mirror information and the second mirror information.

[0023] Optionally, the visual-tactile sensor detection method provided in this application further includes:

[0024] The deviation calculation result is regularized according to the pre-set regularization parameters to obtain the regularization result;

[0025] The offset calculation result of the feature point is obtained based on the regularization processing result;

[0026] The feature point imaging information is corrected based on the deviation calculation results of the feature points to obtain the corrected feature point imaging information.

[0027] Secondly, this application also provides a visual-tactile sensor detection device, comprising:

[0028] The feature point imaging acquisition module is used to acquire feature point imaging information through a miniature camera, wherein the feature points are distributed on the lower surface of the elastic body of the visual-tactile sensor;

[0029] The reference point imaging acquisition module is used to acquire reference point imaging information through the miniature camera, wherein the reference feature points are distributed on the surface of the light-transmitting plate of the visual-tactile sensor, and the light-transmitting plate is fixed to the upper surface of the elastomer.

[0030] The reference point deviation calculation module is used to calculate the position deviation based on the reference point imaging information and the preset reference point imaging information, and obtain the deviation calculation result.

[0031] The feature point information correction module is used to correct the feature point imaging information according to the deviation calculation result, so as to obtain the corrected feature point imaging information.

[0032] The detection result generation module is used to substitute the corrected feature point imaging information into a pre-set mechanical model to obtain the detection result of the visual-tactile sensor.

[0033] Thirdly, this application also provides a visual-tactile sensor, which performs tactile detection using any of the visual-tactile sensor detection methods of the first aspect, including:

[0034] Miniature camera, first mirror, second mirror, elastomer, and light-transmitting sheet;

[0035] The elastomer is fixed to the lower surface of the light-transmitting plate, and the lower surface of the elastomer is provided with feature points, while the surface of the light-transmitting plate is provided with reference points.

[0036] The first mirror and the second mirror are positioned directly above the light-transmitting plate and within the lens range of the miniature camera. The first mirror and the second mirror have different angles relative to the horizontal plane, which are used to form the first mirror image information corresponding to the first mirror and the second mirror image information corresponding to the second mirror, based on the feature point and the reference point.

[0037] The miniature camera is used to collect the first mirror image information formed by the feature point and the reference point through the first mirror surface, and to collect the second mirror image information formed by the feature point and the reference point through the second mirror surface.

[0038] Optionally, in the visual-tactile sensor provided in this application, a plurality of the feature points form a feature point array, and a plurality of reference points surround the feature point array in the horizontal direction and are respectively disposed on the upper and lower surfaces of the light-transmitting plate.

[0039] The miniature camera is used to simultaneously acquire first mirror information corresponding to the feature point array and reference point formed by the first mirror, and second mirror information corresponding to the feature point array and reference point formed by the second mirror.

[0040] Fourthly, this application also provides an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions being executed by the processor to implement the steps of the visual-tactile sensor detection method as described in the first aspect.

[0041] Fifthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the visual-tactile sensor detection method as described in the first aspect.

[0042] In the technical solution provided in this application, the position deviation of the miniature camera after temperature changes during operation is calculated using the imaging information of reference points set on the light-transmitting plate and preset reference points. Based on the position deviation calculation results, the feature point imaging information of the reference points set on the elastic body is corrected, thereby obtaining the detection results of the visual-tactile sensor. Since the reference points are set on the light-transmitting plate, the plate does not deform due to the magnitude or distribution of the contact force during contact force detection. The changes in the reference points before and after detection are only affected by the changes in the internal and external parameters of the miniature camera when it heats up. The result obtained from the position deviation calculation is the difference in image data after the changes in the internal and external parameters of the miniature camera. However, the elastic body deforms due to the magnitude and distribution of the contact force, and the feature points set on the elastic body also change position. When the internal and external parameters of the miniature camera undergo slight deformation due to temperature, the changes in the feature point imaging information are simultaneously affected by both the camera's internal and external parameters and the contact force, and therefore cannot characterize the actual contact force.

[0043] The position deviation calculation result obtained in this application corrects the feature point imaging information corresponding to the reference point set on the elastic body. This compensates for the difference in feature point imaging information caused by the temperature rise of the miniature camera. At this time, the change in feature point imaging information is only affected by the single factor of contact force. The corrected feature point imaging information can be used to characterize the actual contact force, avoid the contact force measurement error caused by the temperature rise of the miniature camera, and improve the detection accuracy of the visual tactile sensor.

[0044] The above description is merely an overview of the technical solution provided in this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are described below. Attached Figure Description

[0045] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0046] Figure 1 This is an example of the appearance of a visual-tactile sensor provided in this application;

[0047] Figure 2 This is an example of the internal structure of a visual-tactile sensor provided in this application;

[0048] Figure 3 This is an example of a visual-tactile sensor imaging provided in this application;

[0049] Figure 4 This is one of the schematic diagrams of the visual-tactile sensor detection method provided in the embodiments of this application;

[0050] Figure 5 This is a second schematic diagram of the visual-tactile sensor detection method provided in the embodiments of this application;

[0051] Figure 6 This is the third schematic diagram of the visual-tactile sensor detection method provided in the embodiments of this application;

[0052] Figure 7 This is the fourth schematic diagram of the visual-tactile sensor detection method provided in the embodiments of this application;

[0053] Figure 8 This is the fifth schematic diagram of the visual-tactile sensor detection method provided in the embodiments of this application;

[0054] Figure 9 This is an example of the layout of reference feature points provided in this application;

[0055] Figure 10 This is an example of an image captured by a miniature camera provided in this application;

[0056] Figure 11 This is a schematic diagram of the visual-tactile sensor detection device provided in the embodiments of this application;

[0057] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0058] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0059] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0060] Existing research indicates that the reliability and safety of grasping objects often depend on the contact force and distribution information perceived by the human hand's tactile senses. As the application of robots continues to expand, user demand for robots to replace human operators in performing tasks, such as grasping and picking up objects, is gradually increasing. However, robots can only manipulate objects reliably and dexterously like humans once they possess the tactile perception capability to sense the magnitude and distribution of contact force at the fingertips.

[0061] Existing technologies often involve mounting various sensors on robots to collect and analyze data, achieving tactile perception capabilities similar to those of a human hand. These include numerous tactile sensors, such as arrays based on piezoresistive and capacitive principles, and vision-based tactile sensors based on optical images. Among these, vision-based tactile sensors possess unique advantages in perceiving dense distributions of contact forces and simultaneously sensing multiple forms of tactile information, particularly in the magnitude and distribution of contact forces.

[0062] The visual-tactile sensor based on the virtual binocular method includes an elastic body with a marked pattern, such as feature points, a miniature camera, and two reflectors. Since the miniature camera can simultaneously capture the reflections in both reflectors, it can also capture virtual images of the elastic body in both reflectors, approximating images of the marked elastic body from two different perspectives. At this point, the 3D positions of the feature points in the marked pattern can be determined based on the principle of binocular vision, and the 3D displacement of each feature point can be measured during the deformation of the elastic body in contact with an external object. Finally, the 3D displacements of the feature points can be substituted into the mechanical model of the elastic body to solve for the magnitude and distribution of the contact force acting on the elastic body.

[0063] Specifically, the elastomer includes both a marking layer and a contact body. The contact body is made of latex or silicone. Silicone not only possesses excellent resilience, ensuring that the contact body returns to its original shape after the contact force test, preventing deformation of the marking layer, but also has transparent optical properties, ensuring that the patterns and shapes of the marking layer after deformation during the contact force test can be mirrored in a reflector for acquisition by a miniature camera. The marking layer includes marking patterns, such as a dot matrix composed of feature points. The positions of these feature points change as the contact body deforms during the contact force test. These changes, such as displacement direction and distance, characterize the magnitude and distribution of the contact force. By comparing the mirror image acquired by the miniature camera with the actual dot matrix image of the feature points, the magnitude and distribution of the contact force during the contact force test can be obtained through mechanical model analysis.

[0064] like Figure 1 As shown, taking a visual-tactile sensor based on a virtual binocular method as an example, Figure 2 The internal structure of the visual-tactile sensor includes an elastomer (3) on the lower surface of one side. A marking pattern (3.1) is also present on the lower surface of the elastomer (3), and this marking pattern (3.1) changes with the deformation of the elastomer (3). The upper surface of the elastomer (3) is connected to a light-transmitting plate, such as a fixed glass plate (4). The upper part of the fixed glass plate (4) is hollow, and two first reflectors (2.1) and second reflectors (2.2) with different angles to the horizontal plane are provided. The visual-tactile sensor also includes a miniature camera (1) capable of simultaneously capturing images in the first reflector (2.1) and the second reflector (2.2).

[0065] Taking a feature point in the marked pattern (3.1) as an example, after the feature point has been displaced due to the deformation of the elastic body (3), the light emitted by the light source passes through the elastic body (3) and the fixed glass plate (4), onto the first reflector (2.1) and the second reflector (2.2) respectively, and is reflected into the lens of the miniature camera (1). Figure 3As shown, the image captured by the miniature camera (1) at this time is equivalent to the result obtained by the miniature camera (1) capturing the feature point at the first virtual image (1.1) and the second virtual image (1.2) respectively and stitching them together.

[0066] Since the intrinsic parameters of the miniature camera (1) can be obtained directly through camera calibration, and the specific positions of the first virtual image (1.1) and the second virtual image (1.2) of the miniature camera (1) are determined based on the distance between the miniature camera (1) and the first reflector (2.1) and the second reflector (2.2), the three-dimensional position of the feature point in the marker pattern (3.1) can be calculated using the same triangulation method as that used with conventional binocular cameras. For example, the position can be determined by parameters such as the intrinsic parameters of the miniature camera (1), distortion parameters, the first virtual image (1.1) and the second virtual image (1.2), and the attitude angle of the camera virtual image during equivalent shooting of the first virtual image (1.1) and the second virtual image (1.2).

[0067] However, the camera parameters change due to heat generated during operation of the miniature camera. After the camera lens undergoes slight deformation due to heat, the camera's intrinsic parameters, including but not limited to distortion parameters, focal length, and optical center position, may change slightly. Meanwhile, the camera's mounting structure, after being deformed by heat, can cause slight changes in the camera's extrinsic parameters, including but not limited to camera position and attitude. These changes in camera parameters lead to systematic deviations in the 3D positions and displacements of feature points measured by the virtual binocular method, ultimately resulting in a systematic error in the calculated distributed force due to changes in camera temperature. This error caused by camera temperature changes is also known as temperature drift in the measurement of feature point positions and distributed forces using a virtual binocular visual-tactile sensor. In other words, existing visual-tactile sensors experience temperature drift as the miniature camera continues to operate, leading to systematic deviations in the feature point positions calculated using triangulation methods, and resulting in low detection accuracy for the visual-tactile sensor.

[0068] The technical solution provided in this application adds a reference point located on a light-transmitting plate, such as a fixed glass plate (4), on top of the feature points. Since the fixed glass plate (4) has a certain hardness, it will not deform significantly with the magnitude of the contact force during the contact force test. The reference point set on the fixed glass plate (4) can be regarded as a fixed point that will not change with the contact force test. At this time, it is only necessary to quickly determine the influence of temperature drift on the feature points based on the position change of the reference point after calibration and during the test. Thus, the feature points are compensated and corrected based on the deviation distance of the reference point. The corrected feature points eliminate the error caused by the temperature change of the miniature camera, and the detection of the magnitude and distribution of the contact force is more accurate.

[0069] The following description, in conjunction with the accompanying drawings, details the visual-tactile sensor detection method, apparatus, visual-tactile sensor, electronic device, and storage medium provided in this application through specific embodiments and application scenarios.

[0070] The first embodiment of this application relates to a visual-tactile sensor detection method, such as... Figure 4 As shown, it includes:

[0071] Step 101: Acquire feature point imaging information of feature points using a miniature camera, wherein the feature points are distributed on the lower surface of the elastic body of the visual-tactile sensor;

[0072] Step 102: Acquire reference point imaging information of the reference points through the miniature camera, wherein the reference feature points are distributed on the surface of the light-transmitting plate of the visual-tactile sensor, and the light-transmitting plate is fixed on the upper surface of the elastomer;

[0073] Step 103: Calculate the position deviation based on the imaging information of the reference point and the imaging information of the preset reference point to obtain the deviation calculation result;

[0074] Step 104: Correct the feature point imaging information according to the deviation calculation result to obtain the corrected feature point imaging information;

[0075] Step 105: Substitute the corrected feature point imaging information into the pre-set mechanical model to obtain the detection results of the visual-tactile sensor.

[0076] Specifically, in the visual-tactile sensor detection method provided in this application, the feature point imaging information of the feature points set on the lower surface of the elastic body (3) and the reference point imaging information set on the light-transmitting plate, such as the fixed glass plate (4), are first collected by a miniature camera (1). Since the reference point imaging information corresponding to the reference point is theoretically fixed, when the reference point imaging information changes, it indicates that the internal parameters of the miniature camera (1) have changed due to the heat generated during its operation. At this time, the image is compared with the reference point imaging information and the preset reference point imaging information. For example, the reference point imaging information when the miniature camera (1) is powered on for a period of time and the reference point imaging information when the miniature camera (1) is first powered on is compared and calculated to obtain the deviation calculation result.

[0077] The deviation calculation result obtained at this time is the error of the image captured by the miniature camera (1) due to temperature drift. Based on the deviation calculation result, the feature point imaging information obtained by the miniature camera (1) is corrected, and the corrected feature point imaging information is substituted into the pre-set mechanical model to obtain the detection result of the visual-touch sensor.

[0078] In the technical solution provided in this application, there can be multiple feature points, which can form a dot matrix, such as a 4×4 dot matrix. The image captured by the miniature camera (1) includes the dot matrix image of these feature points. The reference points can be the same as the feature points, such as solid circular dots, or multi-layered circles, crosses, polygons, checkerboard corner points, or other forms of markers. This application does not impose any restrictions. Both the reference points and the feature points can be acquired by the miniature camera and identified from the image through algorithms, such as target recognition algorithms.

[0079] In the technical solution provided in this application, the position deviation of the miniature camera (1) after temperature change during operation is calculated by using the imaging information of the reference point set on the light-transmitting plate and the preset reference point. Based on the result of the position deviation calculation, the feature point imaging information of the reference point set on the elastic body is corrected, thereby obtaining the detection result of the visual-touch sensor. Since the reference point is set on the light-transmitting plate, the light-transmitting plate will not deform due to the magnitude and distribution of the contact force during contact force detection. The change of the reference point before and after detection is only affected by the change of the internal and external parameters of the miniature camera when the temperature rises. The result obtained by the position deviation calculation is the difference in image data after the change of the internal and external parameters of the miniature camera. However, the elastic body deforms due to the magnitude and distribution of the contact force, and the feature point set on the elastic body will also change position. When the internal and external parameters of the miniature camera undergo slight deformation due to temperature, the change of the feature point imaging information is affected by both the internal and external parameters of the camera and the contact force, and cannot characterize the actual situation of the contact force.

[0080] The position deviation calculation result obtained in this application corrects the feature point imaging information corresponding to the reference point set on the elastic body. This compensates for the difference in feature point imaging information caused by the temperature rise of the miniature camera. At this time, the change in feature point imaging information is only affected by the single factor of contact force. The corrected feature point imaging information can be used to characterize the actual contact force, avoid the contact force measurement error caused by the temperature rise of the miniature camera, and improve the detection accuracy of the visual tactile sensor.

[0081] Based on the above implementation methods, such as Figure 5 As shown, there are multiple reference feature points. The reference point imaging information includes first reference point imaging information and second reference point imaging information. The preset reference point imaging information includes first preset reference point imaging information corresponding to the first reference point imaging information and second preset reference point imaging information corresponding to the second reference point imaging information. In the visual-tactile sensor detection method provided in this application, step 103 includes:

[0082] Step 131: Perform position deviation calculation based on the first reference point imaging information and the first preset reference point imaging information to obtain a first intermediate result, wherein the reference point corresponding to the first reference point imaging information is located on the lower surface of the light-transmitting plate.

[0083] Step 132: Calculate the position deviation based on the second reference point imaging information and the second preset reference point imaging information to obtain a second intermediate result, wherein the reference point corresponding to the second reference point imaging information is located on the upper surface of the light-transmitting plate.

[0084] Step 133: Calculate the deviation calculation result based on the first intermediate result and the second intermediate result.

[0085] Specifically, in the visual-tactile sensor detection method provided in this application, there can be one or more reference points. When there are multiple reference points, they include a first reference point disposed on the lower surface of the fixed glass plate (4) and a second reference point disposed on the upper surface of the fixed glass plate (4). For example, when the number of reference points is greater than or equal to four, it is necessary to ensure that there is both a first reference point disposed on the lower surface of the fixed glass plate (4) and a second reference point disposed on the upper surface of the fixed glass plate (4).

[0086] Subsequently, the first intermediate result and the second intermediate result are calculated based on the first reference point and the second reference point, respectively, and the deviation calculation result of the reference point is calculated based on the first intermediate result and the second intermediate result.

[0087] Based on the above embodiments, since the reference point is set on both the upper and lower surfaces of the transparent plate in the technical solution provided in this application, the refraction phenomenon caused by the different media when light passes through the transparent plate is avoided from affecting the image data collected by the miniature camera, thereby improving the accuracy of the visual-tactile sensor detection.

[0088] Based on the above implementation methods, such as Figure 6 As shown, there are multiple reference feature points. The reference point imaging information includes third reference point imaging information and fourth reference point imaging information. The preset reference point imaging information includes third preset reference point imaging information corresponding to the third reference point imaging information and fourth preset reference point imaging information corresponding to the fourth reference point imaging information. In the visual-tactile sensor detection method provided in this application, step 103 includes:

[0089] Step 134: Calculate the position deviation based on the imaging information of the third reference point and the imaging information of the third preset reference point to obtain a third intermediate result, wherein the horizontal distance from the reference point corresponding to the imaging information of the third reference point to the first side of the light-transmitting plate is less than the horizontal distance from any feature point to the first side of the light-transmitting plate.

[0090] Step 135: Calculate the position deviation based on the imaging information of the fourth reference point and the imaging information of the fourth preset reference point to obtain a fourth intermediate result. The horizontal distance from the reference point corresponding to the imaging information of the fourth reference point to the second side of the light-transmitting plate is less than the horizontal distance from any feature point to the second side of the light-transmitting plate. The second side is different from the first side.

[0091] Step 136: Calculate the deviation calculation result based on the third intermediate result and the fourth intermediate result.

[0092] Specifically, in the visual-tactile sensor detection method provided in this application, there can be one or more reference points. When there are multiple reference points, the horizontal distance between these reference points and each corner of the fixed glass plate (4) is smaller than the horizontal distance between the feature points and each corner of the fixed glass plate (4). For example, the horizontal distance from the third reference point to the first edge is smaller than the horizontal distance from all feature points to the first edge, and the horizontal distance from the fourth reference point to the second edge is smaller than the horizontal distance from all feature points to the second edge. That is, the polygon projected by these reference points onto the horizontal plane can enclose and cover the marking pattern (3.1) on the elastic body (3). For example, the polygon obtained by the projection of the reference points can encompass the polygon obtained by the projection of the square matrix composed of feature points.

[0093] Subsequently, the third intermediate result and the fourth intermediate result are calculated based on the third reference point and the fourth reference point, respectively, and the deviation calculation result of the reference point is calculated based on the third intermediate result and the fourth intermediate result.

[0094] Based on the above embodiments, since the multiple reference points in the technical solution provided in this application are located outside the polygon on the horizontal plane where the feature points are projected, it is possible to accurately distinguish the positional shifts of the reference points and feature points caused by the drift of different parameters in the intrinsic and extrinsic parameters of the miniature camera, thereby improving the accuracy of the visual-tactile sensor detection.

[0095] Based on the above implementation methods, such as Figure 7 As shown, in the visual-tactile sensor detection method provided in this application, step 101 includes:

[0096] Step 111: Acquire the first mirror image information of the feature points formed on the first mirror surface using the miniature camera;

[0097] Step 112: Acquire the second mirror image information of the feature points formed on the second mirror surface using the miniature camera, wherein both the first mirror surface and the second mirror surface are located within the lens range of the miniature camera and the angles between the first mirror surface and the second mirror surface and the horizontal plane are different;

[0098] Step 113: Synthesize the feature point imaging information based on the first mirror information and the second mirror information.

[0099] Specifically, in the visual-tactile sensor detection method provided in this application, the visual-tactile sensor is provided with a first mirror and a second mirror, such as a first reflector (2.1) and a second reflector (2.2).

[0100] Based on the above implementation, since this application uses a visual-tactile sensor based on a virtual binocular method for detection, the angles between the first and second mirrors and the horizontal plane are different, and the images captured by the miniature camera are also different. The three-dimensional position changes of the feature points can be calculated, and the detection results obtained by substituting the three-dimensional position changes into the mechanical model can characterize the distribution of contact force, and the visual-tactile sensor has a better detection effect.

[0101] Based on the above implementation methods, such as Figure 8 As shown, in the visual-tactile sensor detection method provided in this application, step 104 includes:

[0102] Step 141: Perform regularization processing on the deviation calculation result according to the preset regularization parameters to obtain the regularization processing result;

[0103] Step 142: Calculate the offset calculation result of the feature point based on the regularization processing result;

[0104] Step 143: Correct the feature point imaging information based on the deviation calculation results of the feature points to obtain the corrected feature point imaging information.

[0105] Specifically, in the visual-tactile sensor detection method provided in this application, after obtaining the deviation calculation result of the reference point due to temperature change, the position offset of the feature point is estimated based on the pre-derived linear relationship between the position offset of the reference point and the position offset of the feature point.

[0106] Subsequently, the marker position information calculated using the triangulation method is corrected based on the estimated feature point position offset. The corrected feature point imaging information is then input into the mechanical model to obtain the detection results, thereby determining the magnitude and distribution of the contact force during the contact force test.

[0107] Based on the above implementation, this application further improves the ill-conditioning of the matrix describing the linear relationship between the reference point position offset and the feature point position offset by regularization processing, thereby enhancing the stability of the algorithm for estimating the feature point position deviation.

[0108] Based on the above implementation methods, such as Figures 9-10As shown, this application also provides an example of a visual-tactile sensor detection process:

[0109] like Figure 9 As shown, the technical solution provided in this application sets multiple reference feature points (5) on the upper and lower surfaces of the fixed glass plate (4). In the image captured by the miniature camera (1), the position deviation calculation result is obtained by comparing the image (5.1) formed by the reference feature point (5) on the first reflector (2.1), the image (5.2) formed by the reference feature point (5) on the second reflector (2.2), and the image formed by the reference feature point (5) when the miniature camera is not affected by temperature. Subsequently, the images (3.1.1) and (3.1.2) formed by the feature point on the first reflector (2.1) and the second reflector (2.2) respectively, as well as the position deviation result, are substituted into the mechanical model to characterize the degree of deformation of the elastic body (3) during the contact force test, and thus determine the magnitude and distribution of the contact force. Among them, as shown in the figure, Figure 10 As shown, image (3.1.1) is located in the left half of the image from the camera's perspective, and is the position of the feature points in the marker pattern captured by the miniature camera (1) through the first reflector (2.1) in the captured image; image (3.1.2) is located in the right half of the image from the camera's perspective, and is the position of the feature points in the marker pattern captured by the miniature camera (1) through the second reflector (2.2) in the captured image.

[0110] Furthermore, this application also provides a specific method for calculating the deviation of feature point positions:

[0111] The intrinsic parameter of the miniature camera (1) in the technical solution provided in this application is P. I , is a column vector, as shown in Equation 1:

[0112] P I =(f x f y c x c y ) T (1)

[0113] Among them, f x It is the focal length of the miniature camera (1) in the x-direction, f y It is the focal length of the miniature camera (1) in the y direction, c x It is the x-coordinate of the image center point in the pixel coordinate system, c y It is the y-axis coordinate of the image center point in the pixel coordinate system.

[0114] In the technical solution provided in this application, the external parameter matrix of the miniature camera (1) is P. E It takes the form of 4×4, as shown in Equation 2:

[0115]

[0116] Among them, R c L is the rotation matrix of the miniature camera (1) coordinate system in the world coordinate system. c It is the position of the optical center of the miniature camera in the world coordinate system, as shown in Equation 3:

[0117] L c =(x c y c , z c ) T (3)

[0118] Where, x c The x-coordinate and y-coordinate of the optical center of the miniature camera in the world coordinate system are... c It is the y-coordinate of the optical center of the miniature camera in the world coordinate system, z-coordinate. c It is the z-axis coordinate of the optical center of the miniature camera in the world coordinate system.

[0119] M1 is the mirror image matrix formed by the first reflecting mirror (2.1), and M2 is the mirror image matrix formed by the second reflecting mirror (2.2). At this point, M1P E It is the extrinsic parameter matrix of the first virtual image (1.1), denoted by P. E1 Indicates M2P E It is the extrinsic parameter matrix of the second virtual image (1.2), denoted by P. E2 express.

[0120] The marking pattern printed on the surface of an elastomer can be considered as a set of n feature points. For the i-th feature point, the camera can simultaneously capture its virtual image in both mirrors. Let p... L，i and p R，i Let S and y represent the positions of the i-th feature point in the left and right halves of the camera's field of view, respectively. The three-dimensional coordinates S of this feature point in the world coordinate system are: i It can be determined by the known P I P E1 P E2 p L，i and p R，i The result is obtained from the triangulation method, as shown in Equation 4:

[0121] S i =F(P I P E1 P E2 p L，i p R,i (4)

[0122] Since the logo pattern is a set of n feature points, all feature points can be represented by their three-dimensional coordinates S in the world coordinate system. i Use a column vector S AAs shown in Equation 4:

[0123] S A =(S1) T , ...,S n T ) T (5)

[0124] Among them, from S1 to S n These n three-dimensional coordinates can each be represented by a 3×1 column vector. Concatenating these n 3×1 column vectors results in a longer column vector S. A .

[0125] Furthermore, the positions of m reference feature points (5) located on the fixed glass plate (4) can be recorded by a miniature camera (1) in a manner similar to that of the aforementioned feature points. Let r be the reference feature points. L，j and r R，j Let Q represent the positions of the j-th feature point in the left and right halves of the camera's field of view, respectively. Let Q be the three-dimensional coordinates of this feature point in the world coordinate system. j It can be determined by the known P I P E1 P E2 r L，j and r R，j The result is calculated using triangulation, as shown in Equation 6:

[0126] Q j =G(P I P E1 P E2 r L，j r R,j (6)

[0127] Since the reference point is a set of m reference feature points, all reference feature points can be determined according to their three-dimensional coordinates Q in the world coordinate system. j Use a column vector Q A As shown in Equation 7:

[0128] Q A =(Q1) T , ...,Q m T ) T (7)

[0129] Among them, from Q1 to Q m These m three-dimensional coordinates can each be represented by a 3×1 column vector. Concatenating these n 3×1 column vectors results in a longer column vector Q. A .

[0130] When the miniature camera (1) generates heat due to being powered on, the internal reference P... Iand extrinsic parameter matrix P E A slight change occurs, internal reference P I and extrinsic parameter matrix P E The change can be represented by ΔP I and ΔP E As shown in Equations 8 and 9:

[0131] ΔP I =(Δf x , Δf y , Δc x , Δc y ) T (8)

[0132]

[0133] Where, Δf x , Δf y Δc x Δc y f x f y c x and c y The change in ΔR c The rotation matrix is ​​used to represent the attitude change caused by the heating of the miniature camera (1). Let α... x α y and α z Let ΔR be the small rotation angle of the miniature camera (1) in the x, y, and z axes of the camera coordinate system. c (Δα x ,Δα y ,Δα z ) is the rotation matrix ΔR c Regarding Δα x , Δα y , Δα z function

[0134] L c This is the change in position of the optical center of the miniature camera in the world coordinate system, as shown in Equation 10:

[0135] ΔL c =(Δx) c Δy c Δz c ) T (10)

[0136] Where, Δx c It is the change in the x-axis coordinate of the optical center of the miniature camera in the world coordinate system, Δy. c It is the change in the y-axis coordinate of the optical center of the miniature camera in the world coordinate system, Δz. cIt is the change in the z-axis coordinate of the optical center of the miniature camera in the world coordinate system.

[0137] At this point, the three-dimensional position deviation ΔS of the feature point is calculated using the original intrinsic and extrinsic parameter matrices of the miniature camera. i As shown in Equation 11:

[0138] ΔS i =F(P I +ΔP I P E1 ΔP E P E2 ΔP E p L，i p R,i )-F(P I P E1 P E2 p L，i p R,i (11)

[0139] The three-dimensional position deviation ΔQ of the reference point was calculated using the original intrinsic and extrinsic parameter matrices of the miniature camera. j As shown in Equation 12:

[0140] ΔQ j =G(P I +ΔP I P E1 ΔP E P E2 ΔP E r L，j r R,j )-G(P I P E1 P E2 r L，j r R,j (12)

[0141] The parameter deviation of the miniature camera is represented by a vector e. c As shown in Equation 13:

[0142] e c =(Δf x , Δf y , Δc x , Δc y ,Δα x ,Δα y ,Δα z Δx c Δy c Δz c ) T (13)

[0143] e c With ΔSi The relationship can be approximated as linear, as shown in Equation 14:

[0144]

[0145] e c With ΔQ j The relationship can be approximated as linear, as shown in Equation 15:

[0146]

[0147] Among them, matrix sum matrix It can be calculated numerically from functions F and G. Since the sensor has n feature points and m reference feature points arranged in a moving manner, the n matrices will be distinguished here. and m The matrices are concatenated into two matrices, denoted as M. S and M Q As shown in Equations 16 and 17:

[0148]

[0149]

[0150] At this time, ΔS A and ΔQ A The approximate relationship between them can be obtained through M S and M Q As shown in Equation 18:

[0151] ΔS A =M S M Q + ΔQ A (18)

[0152] Considering M Q The matrix may contain extremely small eigenvalues, and directly calculating the pseudo-inverse and substituting it may result in significant numerical computation errors. Therefore, a regularization method is introduced, as shown in Equation 19 in practical calculations:

[0153] ΔS A =M S (M Q T M Q +ωI) -1 M Q T ΔQ A (19)

[0154] Where ω is a regularization parameter greater than zero, ensuring... and ||M S (M Q T M Q +ωI) -1 ||2 Under the premise of the same order of magnitude, select the smallest possible ω. The specific value can be determined by the user according to their own needs, and this application does not impose any restrictions.

[0155] The second embodiment of this application relates to a visual-tactile sensor detection device, such as... Figure 11 As shown, it includes:

[0156] The feature point imaging acquisition module 201 is used to acquire feature point imaging information of feature points through a miniature camera, wherein the feature points are distributed on the lower surface of the elastic body of the visual-touch sensor.

[0157] The reference point imaging acquisition module 202 is used to acquire reference point imaging information of reference points through the miniature camera, wherein the reference feature points are distributed on the surface of the light-transmitting plate of the visual-tactile sensor, and the light-transmitting plate is fixed to the upper surface of the elastomer.

[0158] The reference point deviation calculation module 203 is used to perform position deviation calculation based on the reference point imaging information and the preset reference point imaging information, and obtain the deviation calculation result.

[0159] The feature point information correction module 204 is used to correct the feature point imaging information according to the deviation calculation result, so as to obtain the corrected feature point imaging information.

[0160] The detection result generation module 205 is used to substitute the corrected feature point imaging information into a pre-set mechanical model to obtain the detection result of the visual-tactile sensor.

[0161] Based on the above embodiments, there are multiple reference feature points, and the reference point imaging information includes first reference point imaging information and second reference point imaging information. The preset reference point imaging information includes first preset reference point imaging information corresponding to the first reference point imaging information and second preset reference point imaging information corresponding to the second reference point imaging information. In the visual-tactile sensor detection device provided in this application, the reference point deviation calculation module 203 includes:

[0162] The first reference deviation calculation unit is used to perform position deviation calculation based on the first reference point imaging information and the first preset reference point imaging information to obtain a first intermediate result, wherein the reference point corresponding to the first reference point imaging information is located on the lower surface of the light-transmitting plate.

[0163] The second reference deviation calculation unit is used to perform position deviation calculation based on the second reference point imaging information and the second preset reference point imaging information to obtain a second intermediate result, wherein the reference point corresponding to the second reference point imaging information is located on the upper surface of the light-transmitting plate.

[0164] The first summary calculation unit is used to calculate the deviation calculation result based on the first intermediate result and the second intermediate result.

[0165] Based on the above implementation, there are multiple reference feature points, and the reference point imaging information includes third reference point imaging information and fourth reference point imaging information. The preset reference point imaging information includes third preset reference point imaging information corresponding to the third reference point imaging information and fourth preset reference point imaging information corresponding to the fourth reference point imaging information. In the visual-tactile sensor detection device provided in this application, the reference point deviation calculation module 203 includes:

[0166] The third reference deviation calculation unit is used to perform position deviation calculation based on the third reference point imaging information and the third preset reference point imaging information to obtain a third intermediate result, wherein the horizontal distance from the reference point corresponding to the third reference point imaging information to the first side of the light-transmitting plate is less than the horizontal distance from any feature point to the first side of the light-transmitting plate.

[0167] The fourth reference deviation calculation unit is used to perform position deviation calculation based on the fourth reference point imaging information and the fourth preset reference point imaging information to obtain a fourth intermediate result. The horizontal distance from the reference point corresponding to the fourth reference point imaging information to the second side of the light-transmitting plate is less than the horizontal distance from any feature point to the second side of the light-transmitting plate. The second side is different from the first side.

[0168] The second summary calculation unit is used to calculate the deviation calculation result based on the third intermediate result and the fourth intermediate result.

[0169] Based on the above embodiments, the feature point imaging acquisition module 201 in the visual-tactile sensor detection device provided in this application includes:

[0170] The first mirror acquisition unit is used to acquire the first mirror information formed by the feature points on the first mirror surface through the miniature camera;

[0171] The second mirror acquisition unit is used to acquire the second mirror information formed by the feature points on the second mirror surface through the miniature camera, wherein the first mirror surface and the second mirror surface are both located within the lens range of the miniature camera and the angles between the first mirror surface and the second mirror surface and the horizontal plane are different.

[0172] The feature point imaging stitching unit is used to synthesize the feature point imaging information based on the first mirror information and the second mirror information.

[0173] Based on the above embodiments, the feature point information correction module 204 in the visual-tactile sensor detection device provided in this application includes:

[0174] The regularization processing unit is used to perform regularization processing on the deviation calculation result according to the pre-set regularization parameters to obtain the regularization processing result;

[0175] The regularized feature point calculation unit is used to calculate the offset calculation result of the feature point based on the regularization processing result;

[0176] The regularized imaging information correction unit is used to correct the feature point imaging information according to the deviation calculation result of the feature point, so as to obtain the corrected feature point imaging information.

[0177] The third embodiment of this application relates to a visual-tactile sensor, which performs tactile detection using any of the visual-tactile sensor detection methods described in the first embodiment, including:

[0178] Miniature camera, first mirror, second mirror, elastomer, and light-transmitting sheet;

[0179] The elastomer is fixed to the lower surface of the light-transmitting plate, and the lower surface of the elastomer is provided with feature points, while the surface of the light-transmitting plate is provided with reference points.

[0180] The first mirror and the second mirror are positioned directly above the light-transmitting plate and within the lens range of the miniature camera. The first mirror and the second mirror have different angles relative to the horizontal plane, which are used to form the first mirror image information corresponding to the first mirror and the second mirror image information corresponding to the second mirror, based on the feature point and the reference point.

[0181] The miniature camera is used to collect the first mirror image information formed by the feature point and the reference point through the first mirror surface, and to collect the second mirror image information formed by the feature point and the reference point through the second mirror surface.

[0182] Based on the above embodiments, in the visual-tactile sensor provided in this application, a plurality of the feature points form a feature point array, and a plurality of reference points surround the feature point array in the horizontal direction and are respectively disposed on the upper and lower surfaces of the light-transmitting plate.

[0183] The miniature camera is used to simultaneously acquire first mirror information corresponding to the feature point array and reference point formed by the first mirror, and second mirror information corresponding to the feature point array and reference point formed by the second mirror.

[0184] The fourth embodiment of this application relates to an electronic device, such as... Figure 12 As shown, it includes:

[0185] At least one processor 301; and,

[0186] The memory 302 is communicatively connected to the at least one processor 301; wherein,

[0187] The memory 302 stores instructions that can be executed by the at least one processor, which are executed by the at least one processor 301 to enable the at least one processor 301 to implement the visual-tactile sensor detection method according to the first embodiment of this application.

[0188] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0189] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0190] The fifth embodiment of this application relates to a non-volatile computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the visual-tactile sensor detection method described in the first embodiment of this application.

[0191] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0192] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0193] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for detecting visual and tactile sensors, characterized in that, The method includes: Feature point imaging information of feature points is acquired by a miniature camera, wherein the feature points are distributed on the lower surface of the elastic body of the visual-tactile sensor; The reference point imaging information of the reference point is acquired by the miniature camera, wherein the reference point is distributed on the surface of the light-transmitting plate of the visual-tactile sensor, and the light-transmitting plate is fixed to the upper surface of the elastomer; Position deviation is calculated based on the imaging information of the reference point and the imaging information of the preset reference point, and the deviation calculation result is obtained. The feature point imaging information is corrected based on the deviation calculation result to obtain the corrected feature point imaging information; Substitute the corrected feature point imaging information into a pre-set mechanical model to obtain the detection results of the visual-touch sensor. The reference points include at least a first reference point and a second reference point, and the reference point imaging information includes at least first reference point imaging information and second reference point imaging information. The reference point corresponding to the first reference point imaging information is located on the lower surface of the light-transmitting plate, and the reference point corresponding to the second reference point imaging information is located on the upper surface of the light-transmitting plate.

2. The method according to claim 1, characterized in that, There are multiple reference points, and the preset reference point imaging information includes first preset reference point imaging information corresponding to the first reference point imaging information and second preset reference point imaging information corresponding to the second reference point imaging information. The step of calculating the positional deviation based on the imaging information of the reference points and the preset reference point imaging information, and obtaining the deviation calculation result, includes: Based on the imaging information of the first reference point and the imaging information of the first preset reference point, the positional deviation is calculated to obtain the first intermediate result; The positional deviation is calculated based on the imaging information of the second reference point and the imaging information of the second preset reference point to obtain the second intermediate result; The deviation calculation result is obtained based on the first intermediate result and the second intermediate result.

3. The method according to claim 1, characterized in that, There are multiple reference points. The imaging information of the reference points includes imaging information of a third reference point and imaging information of a fourth reference point. The preset reference point imaging information includes imaging information of a third preset reference point corresponding to the imaging information of the third reference point and imaging information of a fourth preset reference point corresponding to the imaging information of the fourth reference point. The position deviation calculation is performed based on the imaging information of the reference points and the imaging information of the preset reference points to obtain the deviation calculation result, which includes: The position deviation is calculated based on the imaging information of the third reference point and the imaging information of the third preset reference point to obtain a third intermediate result. The horizontal distance from the reference point corresponding to the imaging information of the third reference point to the first side of the light-transmitting plate is less than the horizontal distance from any feature point to the first side of the light-transmitting plate. The position deviation is calculated based on the imaging information of the fourth reference point and the imaging information of the fourth preset reference point to obtain a fourth intermediate result. The horizontal distance from the reference point corresponding to the imaging information of the fourth reference point to the second side of the light-transmitting plate is less than the horizontal distance from any feature point to the second side of the light-transmitting plate. The second side is different from the first side. The deviation calculation result is obtained based on the third intermediate result and the fourth intermediate result.

4. The method according to claim 1, characterized in that, The feature point imaging information acquired by the miniature camera includes: The miniature camera captures the first mirror image information of the feature points formed on the first mirror surface; The feature points are captured by the miniature camera to form a second mirror image on the second mirror surface, wherein both the first mirror surface and the second mirror surface are within the lens range of the miniature camera and the angles between the first mirror surface and the second mirror surface and the horizontal plane are different. The feature point imaging information is synthesized based on the first mirror information and the second mirror information.

5. The method according to claim 1, characterized in that, The step of correcting the feature point imaging information based on the deviation calculation result to obtain the corrected feature point imaging information includes: The deviation calculation result is regularized according to the pre-set regularization parameters to obtain the regularization result; The offset calculation result of the feature point is obtained based on the regularization processing result; The feature point imaging information is corrected based on the deviation calculation result of the feature point to obtain the corrected feature point imaging information.

6. A visual-tactile sensor detection device, characterized in that, include: The feature point imaging acquisition module is used to acquire feature point imaging information through a miniature camera, wherein the feature points are distributed on the lower surface of the elastic body of the visual-tactile sensor; The reference point imaging acquisition module is used to acquire reference point imaging information through the miniature camera, wherein the reference points are distributed on the surface of the light-transmitting plate of the visual-tactile sensor, and the light-transmitting plate is fixed to the upper surface of the elastomer. The reference point deviation calculation module is used to calculate the position deviation based on the reference point imaging information and the preset reference point imaging information, and obtain the deviation calculation result. The feature point information correction module is used to correct the feature point imaging information according to the deviation calculation result, so as to obtain the corrected feature point imaging information. The detection result generation module is used to substitute the corrected feature point imaging information into a pre-set mechanical model to obtain the detection result of the visual-touch sensor. The reference points include at least a first reference point and a second reference point, and the reference point imaging information includes at least first reference point imaging information and second reference point imaging information. The reference point corresponding to the first reference point imaging information is located on the lower surface of the light-transmitting plate, and the reference point corresponding to the second reference point imaging information is located on the upper surface of the light-transmitting plate.

7. A visual-tactile sensor, characterized in that, Touch detection is performed using the visual-tactile sensor detection method as described in any one of claims 1-5, including: Miniature camera, first mirror, second mirror, elastomer, and light-transmitting sheet; The elastomer is fixed to the lower surface of the light-transmitting plate, and the lower surface of the elastomer is provided with feature points, while the surface of the light-transmitting plate is provided with reference points. The first mirror and the second mirror are positioned directly above the light-transmitting plate and within the lens range of the miniature camera. The first mirror and the second mirror have different angles relative to the horizontal plane, which are used to form the first mirror image information corresponding to the first mirror and the second mirror image information corresponding to the second mirror, based on the feature point and the reference point. The miniature camera is used to collect the first mirror image information formed by the feature point and the reference point through the first mirror surface, and to collect the second mirror image information formed by the feature point and the reference point through the second mirror surface.

8. The visual-tactile sensor according to claim 7, characterized in that, The plurality of feature points form a feature point array, and the plurality of reference points surround the feature point array in the horizontal direction and are respectively disposed on the upper and lower surfaces of the light-transmitting plate. The miniature camera is used to simultaneously acquire first mirror information corresponding to the feature point array and reference point formed by the first mirror, and second mirror information corresponding to the feature point array and reference point formed by the second mirror.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the visual-tactile sensor detection method as described in any one of claims 1-5.

10. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the visual-tactile sensor detection method as described in any one of claims 1-5.

Images