A visual tactile sensor based on grating structure and its preparation method

Abstract

The embodiment of the present application provides a visual-tactile sensor based on a grating structure and a preparation method thereof, which belongs to the technical field of optical measuring instruments. The scheme of the present invention designs a grating layer; according to the grating layer, a flexible contact layer is set; a mapping relationship between the diffraction image and the contact information is established; according to the flexible contact layer and the mapping relationship, visual-tactile sensing information is obtained. The present invention can enable the sensor to realize high-precision recognition of contact information such as multi-axis force (normal force, tangential force, torque, etc.), contact position, contact posture, contact object, etc. on the basis of having high-resolution texture recognition capability.

Description

Technical Field

[0001] The present application relates to the technical field of optical measuring instruments, and in particular to a visual-tactile sensor based on a grating structure and a preparation method thereof. Background Art

[0002] Nowadays, robots are interacting with the external environment more and more frequently, and touch is one of the most important ways to interact with the outside world. For example, in a variety of work scenarios such as virtual reality, medical health, wearable devices, automotive industry, and robotics, the perception and feedback of tactile information are indispensable. As an intelligent device that can simulate human touch, tactile sensors need to have the ability to detect the force, texture and other information of external objects. Related technologies enable visual tactile sensors to have the ability to recognize multi-axis forces by designing the structure of a flexible contact interface, but the currently widely reported structure will interfere with the collection of texture details, and there is a problem of difficulty in balancing multi-axis force and texture recognition accuracy. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide a visual-tactile sensor based on a grating structure and a method for preparing the same.

[0004] On the one hand, an embodiment of the present invention provides a method for preparing a visual-tactile sensor based on a grating structure, the method comprising:

[0005] Design grating layer;

[0006] According to the grating layer, a flexible contact layer is provided;

[0007] Establishing a mapping relationship between the diffraction image and the contact information;

[0008] Visual-tactile sensing information is obtained according to the flexible contact layer and the mapping relationship.

[0009] Furthermore, the designing of the grating layer includes:

[0010] Setting the elasticity of the grating and the compressibility of the grating;

[0011] Setting a periodic structure of the grating;

[0012] Arrange the diffraction units of the grating;

[0013] A deformable material is provided for the grating; the deformable material includes flexible plastic and silica gel;

[0014] Set the size of the grating and the shape of the grating.

[0015] Further, the setting of the periodic structure of the grating includes:

[0016] The grating is arranged into a periodic groove or micro-column structure by periodic arrangement;

[0017] The grating is arranged into a periodic micro-protrusion structure.

[0018] Furthermore, the diffraction unit of the grating layout includes:

[0019] Using arrayed slits as the diffraction unit;

[0020] Using arrayed grooves as the diffraction unit;

[0021] The diffraction elements are arranged on the surface of the grating.

[0022] Furthermore, the formula used in designing the grating layer includes:

[0023] d(sini±siniθ)=kλ

[0024] Where d is the grating constant, θ is the diffraction angle, k is the diffraction order, λ is the wavelength, and i is the incident angle;

[0025] When the diffracted light and the incident light are on the same side of the normal, the sign in the brackets on the left side of the above formula is positive; when the diffracted light and the incident light are on different sides of the normal, the sign in the brackets on the left side of the above formula is negative.

[0026] Furthermore, the step of providing a flexible contact layer according to the grating layer comprises:

[0027] According to the microscopic texture of the grating layer, a flexible contact surface is manufactured using a flexible material;

[0028] matching the surface properties of the flexible contact surface with the grating layer;

[0029] The flexible contact surface is covered on the surface of the grating layer and coated and pressed to complete the flexible contact layer.

[0030] Further, establishing a mapping relationship between the diffraction image and the contact information includes:

[0031] Acquire contact information; the contact information includes multi-axis contact force, posture, and position information; the contact information is measured by a sensor; the multi-axis contact force includes normal force, tangential force, and torque;

[0032] Acquire a diffraction image;

[0033] Extracting characteristic information of the grating layer according to the contact information and the diffraction image; the characteristic information includes a contact boundary, brightness characteristics and chromaticity characteristics in the image;

[0034] The mapping relationship between the grating layer and the contact information is established by using the characteristic information.

[0035] Further, obtaining visual-tactile sensing information according to the flexible contact layer and the mapping relationship includes:

[0036] Acquiring transmission information of the flexible contact layer;

[0037] The transmitted information is processed using the mapping relationship to obtain visual-tactile sensing information.

[0038] Furthermore, the method further comprises:

[0039] Setting components of a visual-tactile sensor system; the components of the visual-tactile sensor system include a grating layer, a reflection layer, a flexible packaging layer, an acrylic substrate, a light source, an imaging system, a mechanical gripper, an outer frame, and a host computer;

[0040] Using organic silicon compounds to prepare elastomers; the elastomers include polydimethylsiloxane elastomers, styrene-based thermoplastic elastomers, polyurethane-based thermoplastic elastomers, polyolefin-based thermoplastic elastomers, and polyamide-based thermoplastic elastomers;

[0041] Adding the grating structure inside the elastic layer to use the change of the diffraction image to represent the tactile information;

[0042] Placing the acrylic substrate below the flexible packaging layer; the acrylic substrate comprises a high-transparency acrylic substrate;

[0043] The acrylic substrate supports the flexible packaging layer;

[0044] The outer frame is manufactured using a model preparation process; the model preparation process includes 3D printing;

[0045] The light source and the imaging system are fixed on the outer frame by screws; the imaging system comprises a COMS camera, a CCD camera and a spectrometer.

[0046] On the other hand, an embodiment of the present invention further provides a visual-tactile sensor based on a grating structure, the visual-tactile sensor comprising a grating layer, a reflective layer, a flexible packaging layer, an acrylic substrate, a light source, an imaging system, a mechanical gripper, an outer frame, and a host computer;

[0047] The grating layer is located above the reflective layer;

[0048] The reflective layer is located above the flexible packaging layer;

[0049] The flexible packaging layer is located above the acrylic substrate;

[0050] The acrylic substrate is located above the outer frame;

[0051] The light source is located within the outer frame;

[0052] The imaging system is located within the outer frame;

[0053] The host computer is connected to the imaging system network;

[0054] The mechanical gripper is connected to the imaging system network.

[0055] The embodiments of the present application include at least the following beneficial effects: The present application provides a visual tactile sensor based on a grating structure and a method for preparing the same. The present invention designs a grating layer; a flexible contact layer is provided according to the grating layer; a mapping relationship between the diffraction image and the contact information is established; and visual tactile sensing information is obtained according to the flexible contact layer and the mapping relationship. The present invention can enable the sensor to realize high-precision recognition of contact information such as multi-axis force, contact position, contact posture, contact object, etc. on the basis of having high-resolution texture recognition capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 1. It is a diagram of the structural design and identification method of a visual-tactile sensor based on a grating structure provided by an embodiment of the present invention;

[0057] Figure 2 It is a schematic diagram of the overall components of the visual-tactile sensor system provided by an embodiment of the present invention;

[0058] Figure 3 is a complete preparation flow chart of the overall elastic sensing layer provided by an embodiment of the present invention;

[0059] Figure 4 is a schematic diagram of the structure of a flexible contact layer provided by an embodiment of the present invention;

[0060] FIG5( a ) is a schematic diagram of the recognition capability of the model provided by an embodiment of the present invention for the normal force of images in a test set;

[0061] FIG5( b ) is a schematic diagram of the recognition capability of the model provided by an embodiment of the present invention for the tangential force of the images in the test set;

[0062] FIG5( c ) is a schematic diagram of the recognition capability of the model provided by an embodiment of the present invention for the torque of the images in the test set;

[0063] Figure 6 is a diffraction principle diagram of a grating provided by an embodiment of the present invention;

[0064] Figure 7 is an identification flow chart provided by an embodiment of the present invention;

[0065] Notes on the various parts of the structure in the attached drawings:

[0066] Flexible contact layer 100 , grating layer 101 , reflective layer 102 , flexible encapsulation layer 103 , acrylic layer 200 , light source 300 , imaging system 400 , mechanical gripper 500 , outer frame 600 , host computer 700 , white light 800 , and diffraction spectrum 900 . DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0068] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0069] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0071] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0072] 1)PLA, which stands for polylactic acid, is a biodegradable polymer material.

[0073] 2) PCB camera, which is a miniature camera built into a printed circuit board (PCB) for visual inspection, image acquisition, and machine vision applications.

[0074] 3) CCD sensor, which stands for Charge-Coupled Device sensor.

[0075] 4) CMOS sensor, which stands for Complementary Metal-Oxide-Semiconductor sensor.

[0076] 5) PDMS, which stands for polydimethylsiloxane, is a silicon-based polymer also known as silicone rubber.

[0077] 6) COMS camera is a digital camera that uses a complementary metal oxide semiconductor (CMOS) sensor.

[0078] 7)CCD camera is a digital camera that uses a charge coupled device (CCD) sensor.

[0079] 8) Spectrometer is an instrument used to measure the wavelength distribution and intensity of light.

[0080] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0081] The present invention takes into account the fact that the visual tactile sensors widely reported in the related art have the ability to perceive multi-axis forces while interfering with high-resolution texture recognition. In order to alleviate this contradiction, the present invention proposes a new method: adding a grating structure inside the flexible contact interface, introducing wavelength information into the system, establishing a mapping relationship between external stimuli and diffraction images, and achieving high-precision multi-axis force perception.

[0082] On the one hand, an embodiment of the present invention provides a method for preparing a visual tactile sensor based on a grating structure, specifically, referring to Figure 1 Structural design and recognition method diagram of visual tactile sensor based on grating structure,

[0083] Structural design includes:

[0084] Grating layer 101 , reflection layer 102 , flexible packaging layer 103 , acrylic layer 200 , light source 300 , imaging system 400 .

[0085] Methods include:

[0086] S100, designing a grating layer 101;

[0087] S200, providing a flexible contact layer 100 according to the grating layer 101;

[0088] S300, establishing a mapping relationship between the diffraction image and the contact information;

[0089] S400 , obtaining visual-tactile sensing information according to the flexible touch layer 100 and the mapping relationship.

[0090] As an optional implementation, in the step S100 of designing the grating layer 101, the embodiment of the present invention needs to first design a micro-nano scale grating structure to replace the traditional contact interface structure. The design of the grating layer 101 needs to take into account its sensitivity to external stimuli and its ability to produce a specific diffraction image.

[0091] Benefits of step S100: Designing a suitable grating layer 101 can enable the system to have a higher perception accuracy of the external multi-axis force, and can solve the interference problem when recognizing texture details, thereby improving the overall performance of the system.

[0092] In the step S200 of providing the flexible contact layer 100 according to the grating layer 101 in the embodiment of the present invention, the designed grating layer 101 is embedded into the flexible contact layer 100 to construct a new contact interface.

[0093] Benefits of step S200: By embedding the grating layer 101 into the flexible contact layer 100, it not only protects the fine grating structure, but also makes the contact interface more sensitive to changes in external forces, thereby improving the accuracy of perception of contact information such as multi-axis force (normal force, tangential force, torque, etc.), contact position, and posture.

[0094] In the step S300 of collecting diffraction images and establishing a mapping relationship, the present embodiment establishes a mapping relationship between external stimuli and diffraction images, that is, a corresponding relationship is established between the diffraction images generated by the grating and the multi-directional forces and torques applied by the outside world.

[0095] Benefits of step S300: Establishing a mapping relationship allows the system to infer multi-axis forces (normal force, tangential force, torque, etc.) acting on the contact interface, contact position, posture, contact object and other contact information based on the characteristics of the diffraction image, thereby achieving accurate perception of external forces, contact position, posture, contact object category and other contact information.

[0096] In the step S400 of obtaining visual and tactile sensing information according to the flexible contact layer 100 and the mapping relationship, the system obtains visual and tactile sensing information according to the deformation of the flexible contact layer 100 and the established mapping relationship. Through this step, the system can obtain multimodal tactile information on the surface of the object, including information such as contour, posture, micro texture, contact position and force (normal force, tangential force, torque, etc.), thereby achieving high-precision perception of the object.

[0097] The embodiment of the present invention discloses step S100 of designing a grating layer 101, including:

[0098] S110, setting the elasticity of the grating and the compressibility of the grating;

[0099] S120, setting a periodic structure of the grating;

[0100] S130, layout of the diffraction unit of the grating;

[0101] S140, setting a deformable material for the grating; the deformable material includes flexible plastic and silicone;

[0102] S150, setting the size and shape of the grating.

[0103] As an optional implementation, the present invention introduces a grating layer 101 into the system to introduce wavelength information. This can be achieved through the characteristics of the grating, because the grating will deform according to the external force, thereby changing the characteristics of the diffraction image.

[0104] The embodiment of the present invention discloses step S120 of setting the periodic structure of the grating, including:

[0105] S121, setting the grating into a periodic groove or micro-column structure by periodic arrangement;

[0106] S122, setting the grating into a periodic micro-protrusion structure.

[0107] The embodiment of the present invention discloses step S130 of laying out the diffraction unit of the grating, including:

[0108] S131, using the arrayed slits as diffraction units;

[0109] S132, using the arrayed grooves as diffraction units;

[0110] S133, layout the diffraction units on the surface of the grating.

[0111] As an optional implementation, in step S121 of the present invention, the grating is arranged into a structure such as periodic grooves or micro-columns that can realize light field modulation through regular arrangement.

[0112] The embodiment of the present invention discloses step S100 of designing the grating layer 101, and the formula used includes:

[0113] d(sini±siniθ)=kλ

[0114] Where d is the grating constant, θ is the diffraction angle, k is the diffraction order, λ is the wavelength, and i is the incident angle;

[0115] When the diffracted light and the incident light are on the same side of the normal, the sign in the brackets on the left side of the above formula is positive; when the diffracted light and the incident light are on different sides of the normal, the sign in the brackets on the left side of the above formula is negative.

[0116] The embodiment of the present invention discloses step S200 of setting a flexible contact layer 100 according to the grating layer 101, including:

[0117] S210, manufacturing a flexible contact surface using a flexible material according to the micro texture of the grating layer 101;

[0118] S220, matching the surface characteristics of the flexible contact surface with the grating layer;

[0119] S230 , covering the surface of the grating layer with a flexible contact surface and performing coating and lamination to complete the flexible contact layer 100 .

[0120] The embodiment of the present invention discloses step S300 of establishing a mapping relationship between the diffraction image and the contact information, including:

[0121] S310, obtaining contact information; the contact information includes multi-axis contact force, posture, and position information; the contact information is obtained by measuring the sensor; the multi-axis contact force includes normal force, tangential force, and torque;

[0122] S320, acquiring a diffraction image;

[0123] S330, extracting feature information of the grating layer according to the contact information and the diffraction image; the feature information includes contact boundaries, brightness features, and chromaticity features in the image;

[0124] S340 , using the feature information to complete the establishment of a mapping relationship between the grating layer 101 and the contact information.

[0125] As an optional implementation, the embodiment of the present invention establishes a mapping relationship between external stimuli and diffraction images. This involves establishing a corresponding relationship between the diffraction image generated by the grating and the multi-directional force and torque applied by the outside world, so as to infer contact information such as multi-axis contact force (normal force, tangential force, torque, etc.), contact position, posture, contact object, etc. acting on the contact interface.

[0126] The embodiment of the present invention discloses step S400 of obtaining visual-tactile sensing information according to the flexible contact layer 100 and the mapping relationship, including:

[0127] S410, obtaining transmission information of the flexible contact layer 100;

[0128] S420: Process the transmitted information using the mapping relationship to obtain visual-tactile sensing information.

[0129] As an optional implementation, an embodiment of the present invention collects and processes diffraction image information. The system needs to be able to capture and analyze the chromaticity changes in the diffraction image to infer the magnitude and direction of the force and torque applied by the outside world. Using the diffraction image information, multimodal tactile information such as the contour, posture, micro-texture, contact position and force (normal force, tangential force, torque, etc.) of the object surface can be extracted. The micro-nano scale of the structure of the grating layer makes the sensor very sensitive to tiny deformations, thereby achieving high-precision tactile perception.

[0130] refer to Figure 2 The embodiment of the present invention discloses a method for preparing a visual-tactile sensor based on a grating structure, and further includes:

[0131] S500, setting components of the visual-tactile sensor system; the components of the visual-tactile sensor system include a grating layer 101, a reflective layer 102, a flexible packaging layer 103, an acrylic substrate 200, a light source 300, an imaging system 400, a mechanical gripper 500, an outer frame 600, and a host computer 700;

[0132] S600, preparing an elastomer using an organic silicon compound; the elastomer includes a polydimethylsiloxane elastomer, a styrene-based thermoplastic elastomer, a polyurethane-based thermoplastic elastomer, a polyolefin-based thermoplastic elastomer, and a polyamide-based thermoplastic elastomer;

[0133] S700, adding a grating structure inside the elastic layer, and using the change of the diffraction image to represent tactile information;

[0134] S800, placing an acrylic substrate 200 below the flexible encapsulation layer 103; the acrylic substrate 200 includes a high-transmittance acrylic substrate;

[0135] S900, supporting the flexible packaging layer 103 with the acrylic substrate 200;

[0136] SA00, using a model preparation process to manufacture an outer frame 600; the model preparation process includes 3D printing;

[0137] SB00, fix the light source and imaging system on the outer frame 600 by screws; the imaging system 400 includes a CMOS camera, a CCD camera, and a spectrometer;

[0138] SC00, use the host computer 700 to receive the diffraction image collected by the imaging system 400.

[0139] As an optional implementation, the elastomer in step S600 of the embodiment of the present invention includes polydimethylsiloxane elastomer, styrene-based thermoplastic elastomer, polyurethane-based thermoplastic elastomer, polyolefin-based thermoplastic elastomer, and polyamide-based thermoplastic elastomer.

[0140] In step S800 of the embodiment of the present invention, a substrate having certain light transmittance, including but not limited to high-transmittance acrylic, is placed under the elastomer, and the elastomer includes a grating layer 101 , a reflective layer 102 , and a flexible packaging layer 103 .

[0141] In step SA00 of the embodiment of the present invention, the outer frame 600 is manufactured using a model preparation process including but not limited to 3D printing.

[0142] As an optional embodiment, the elastomer preparation process of the embodiment of the present invention is as follows:

[0143] The main material used to prepare the elastomer is polydimethylsiloxane (PDMS, Dow Corning, Sylgard 184), which has good thermal stability, biocompatibility, corrosion resistance, flexibility, low cost, easy to use, chemical inertness, proliferation and breathability, and is one of the most promising elastomers. It has important uses in wearable devices, tactile sensors and microfluidic systems, biomedical devices and other fields. Therefore, PDMS is an ideal material for making flexible gratings.

[0144] We use the existing grating template as a mold, pour the prepared PDMS solution into the mold to make a flexible grating. The complete preparation process of the overall elastic sensing layer is as follows: Figure 3 shown.

[0145] (1) Casting. First, prepare a mixed solution of PDMS-A liquid and PDMS curing liquid, stir it thoroughly and degas it, then spin-coat it onto the surface of the grating template in the culture dish.

[0146] (2) Curing: The culture dish containing the PDMS mixture and the template was then placed in a forced air oven at 70°C for 3 hours.

[0147] (3) Demolding. The culture dish containing the cured PDMS was taken out of the oven, and the PDMS was demolded from the mold to obtain a flexible grating film with a thickness of about 0.1 mm. Under laboratory lighting, it can be clearly seen that the surface of the film presents a diffraction pattern.

[0148] (4) Plasma treatment. Place the flexible grating film in a culture dish and send it into a plasma cleaning machine for 3 minutes. After plasma treatment, the PDMS film can have a stronger bonding force with the aluminum film.

[0149] (5) Magnetron sputtering to coat the reflective film. The flexible grating film is then placed in a magnetron sputtering device, and a layer of aluminum film is coated on the grating structure. This has two functions: one is to make the grating have better reflective properties; the other is to prevent the second layer of PDMS poured later from filling and covering the grating structure. It is worth noting that the aluminum film layer should not be too thick and should be less than the depth of the grating grooves to avoid covering the entire grating layer. Therefore, after several adjustments, we set the sputtering current parameters to 90mA and the time to 60s. After this step, we obtained a flexible grating film with a thickness of about 0.1mm and an aluminum film on the surface.

[0150] (6) Encapsulation. Considering that the grating structure is very fine and easily damaged, it is not suitable for long-term exposure to the air, let alone direct contact with external objects. Therefore, it is necessary to cast a layer of PDMS elastomer on the surface of the flexible grating film to encapsulate the fine micro-nano structure in the elastomer.

[0151] (7) Curing: The entire culture dish was placed in a forced air oven at 70°C and dried for 3 hours.

[0152] After curing, a piece of elastic body with a length and width of 30 mm and a thickness of 2.5 mm is finally obtained. The schematic diagram of the structure of the flexible contact layer (elastic body) is shown in Figure 4 As shown, the flexible contact layer has a “sandwich” structure, with three layers in total, namely a grating layer 101 , a reflection layer (aluminum film layer) 102 , and a flexible packaging layer 103 .

[0153] As an optional embodiment, the system design of the present invention is:

[0154] First, the design of the system outer frame in the present invention can assemble other parts of the system together and keep the relative positions fixed. In this example, providing a sealed opaque outer frame can effectively avoid the influence of external ambient light on the diffraction image.

[0155] Second, considering that the elastomer may be damaged during long-term operation of the system, an easily removable and replaceable flexible contact interface is designed. This flexible contact interface consists of a flexible contact layer and an acrylic substrate. When the flexible contact interface is damaged, only the flexible contact layer needs to be removed and replaced as a whole.

[0156] Third, in order to reduce the impact of the distortion of the imaging system, the center of the elastic body is aligned with the center of the imaging system in this example. In addition, the light source is located at the lower right of the elastic body and the upper right of the imaging system. The light will be incident obliquely on the surface of the elastic body, and then the diffracted light will be reflected to the imaging plane through the metal reflective film on the surface of the grating structure.

[0157] The implementation process of the present invention can be divided into five parts, namely (1) building the system, fixing the system on the experimental platform, adjusting the relative positions of the elastic body, the imaging system, and the light source to meet actual needs; (2) initializing the system, the imaging system collects diffraction images when there is no contact with external objects; (3) data set acquisition, making the external object contact with the elastic body, the imaging system collects the diffraction image after the contact, constantly adjusting the contact mode of the external object, and repeatedly collecting the diffraction image until all the required feature images are collected; (4) image feature recognition, annotating each image in the data set, and the annotated content consists of two parts: 1. Which category of tactile information belongs to (including but not limited to normal force, tangential force, texture, shape, posture, etc.), 2. The specific value for this type of tactile feature. Then send it to the neural network for training all the data set images. (5) feature recognition, the external object contacts the elastic body again, and the trained neural network is used to judge the tactile information, and the judgment result will be displayed by the host computer.

[0158] The model's ability to recognize the normal force of the images in the test set is shown in Figure 5(a), the model's ability to recognize the tangential force of the images in the test set is shown in Figure 5(b), and the model's ability to recognize the torque of the images in the test set is shown in Figure 5(c). In Figures 5(a), 5(b), and 5(c), the vertical axis represents the standard multi-axis force and torque values ​​measured by the ATI six-axis sensor, and the horizontal axis represents the predicted multi-axis force and torque values. The balls represent the data distribution of the predicted images in the test set, and the line segments represent the data distribution when the judgment accuracy is 100%. It can be found that the balls are closely distributed on both sides of the line segments, indicating that the prediction accuracy is high. In this example, the system's recognition accuracy for the normal force in the range of 0-3N is 0.04N; for the static friction tangential displacement range of 0-0.4mm (the tangential force range is about 0-1.5N), the recognition accuracy of the tangential force reaches 0.017N; the system's recognition accuracy for the torque in the range of -13-13N-mm is 0.13N-mm. In addition, although this example only tests the recognition accuracy of the system within a specific force value and a specific torque range, in principle, the present invention can perceive tactile information such as multi-axis force (normal force, tangential force, torque, etc.), contact posture, contact position, object shape, object category, etc. within any range.

[0159] On the other hand, a visual tactile sensor based on a grating structure includes a grating layer 101, a reflective layer 102, a flexible packaging layer 103, an acrylic substrate 200, a light source 300, an imaging system 400, a mechanical gripper 500, an outer frame 600, and a host computer 700;

[0160] The grating layer 101 is located above the reflective layer 102;

[0161] The reflective layer 102 is located above the flexible packaging layer 103;

[0162] The flexible encapsulation layer 103 is located above the acrylic substrate 200;

[0163] The acrylic substrate 200 is located above the outer frame 600;

[0164] The light source 300 is located inside the outer frame 600;

[0165] The imaging system 400 is located within the outer frame 600;

[0166] The host computer 700 is connected to the imaging system 400 via a network;

[0167] The mechanical gripper 500 is connected to the imaging system 400 via a network.

[0168] The present invention takes into account the existing work that designs the structure of a flexible contact interface to enable the visual-tactile sensor to have the ability to recognize multi-axis forces (normal force, tangential force, torque, etc.), but the currently widely reported structure will interfere with the collection of texture details, and there is a problem of difficulty in balancing the multi-axis force and texture recognition accuracy. Therefore, an embodiment of the present invention designs a new structure for the visual-tactile sensor to achieve the perception of multi-axis forces (normal force, tangential force, torque, etc.).

[0169] In view of the above problems, the present invention adds a grating structure at a micro-nano scale inside the elastic contact interface of the visual touch sensor to replace the traditional structure, introduces wavelength information into the system, and can decouple the multi-axis force (normal force, tangential force, torque, etc.) from the diffraction image generated by the grating, while still having the ability to identify contact information such as contact posture, contact position, contact shape, contact category, etc. The recognition principle is that when the flexible contact interface based on the grating undergoes different deformations when subjected to different contacts, it can be known from the grating equation that a diffraction spectrum with different characteristics will be generated. The diffraction spectrum is captured by the imaging system and reflected as a change in chromaticity in the image. Therefore, a mapping relationship between the chromaticity characteristics of the diffraction image and the multi-axis force (normal force, tangential force, torque, etc.), contact posture, contact position, contact shape, contact category, etc. can be established.

[0170] Since the data is collected in the form of image information, and different modes of contact behavior have different characteristics in the image, this method can be used to obtain multimodal tactile information such as the contour, posture, micro-texture, contact position, and force (normal force, tangential force, torque, etc.) of the object surface. In addition, the micro-nano-scale flexible grating structure is very sensitive to force, and some tiny deformations will also change the characteristic distribution of the diffraction image, so the sensor can achieve high-precision tactile perception. Therefore, the proposed scheme has a high practical value. For example, in some grasping scenarios, it can judge the grasping state of the object, so as to complete the grasping task more smoothly.

[0171] According to the design expectations, the system of the present invention will serve as a multimodal, highly sensitive, highly robust, and durable flexible tactile sensor suitable for multiple practical scenarios.

[0172] The identification principle and process of the present invention:

[0173] Based on the characteristic that the diffraction image generated by the flexible grating is sensitive to stress and strain, the present invention adds it to the elastic contact interface of the visual touch sensor, so that the sensor can realize high-precision multi-axis force (normal force, tangential force, torque, etc.) recognition on the basis of high-resolution texture recognition capability. The recognition principle of multi-axis force (normal force, tangential force, torque, etc.) of this invention will be explained below.

[0174] Based on the fact that the diffraction image generated by the flexible grating is sensitive to stress and strain, the present invention adds it to the elastic contact interface of the visual tactile sensor, so that the sensor can realize high-precision multi-axis force recognition on the basis of high-resolution texture recognition capability. The recognition principle of multi-axis force (normal force, tangential force, torque, etc.) by this invention will be explained below.

[0175] An optical element composed of a large number (thousands or even tens of thousands) of equally wide and equally spaced slits is usually called a diffraction grating. However, the diffraction units of some gratings may not be slits in the usual sense. Therefore, a grating can be defined as an optical element that can produce periodic spatial modulation of the amplitude and phase of the incident light or one of the two. The grating's spectroscopic principle is based on the diffraction and interference of light, and the incident light is dispersed into light beams of different wavelengths or directions through a periodic structure. When a beam of multi-color light is obliquely incident on the grating plane, the outgoing light will form a series of colorful diffraction spectra after passing through the grating, and the fringes are distributed according to the wavelength law.

[0176] The principle of grating light splitting can be obtained from the formula of the bright line position in the multi-slit Fraunhofer diffraction image. In grating theory, this formula is also called the grating equation:

[0177] dsinθ=kλ (1)

[0178] Where d is the grating constant, θ is the diffraction angle, k is the diffraction order, and λ is the wavelength. From the grating equation, we know that the diffraction angle of the bright line is related to the wavelength and the grating constant. However, formula (1) is only applicable to the case where the incident light is incident vertically on the grating surface. For the more common oblique incidence, the formula should be modified:

[0179] d(sini±sinθ)=kλ (2)

[0180] Where i is the angle of incidence. When the observed diffracted light and the incident light are on the same side of the normal, the above formula takes a positive sign; when the observed diffracted light and the incident light are on different sides of the normal, the above formula takes a negative sign.

[0181] The diffraction principle of the grating used in this scheme is as follows Figure 6 As shown. When white light 800 is incident on the plane of the grating layer 101, it will be split, and the diffraction spectrum 900 will be distributed in the order of wavelength. According to the grating equation, it can be obtained that the shorter the wavelength, the smaller the diffraction angle; the longer the wavelength, the larger the diffraction angle. In addition, it can be seen from formula (2) that the distribution of the diffraction spectrum is also related to the grating constant and the angle of the incident light. In summary, for the flexible grating in the present invention, when the physical form of the flexible grating changes due to contact with an external object, the distribution of the diffraction spectrum will also change. The information of different diffraction images generated by the elastomer under different external stimuli is collected and stored in the host computer. By identifying the image features, a mapping relationship between the diffraction image and the contact situation can be established. Finally, the collected diffraction images are packaged into a data set, and the data set is trained and tested with the help of a deep learning model including but not limited to a deep residual network (ResNet), so as to achieve the purpose of contact information recognition such as multi-axis force (normal force, tangential force, torque, etc.), contact posture, contact position, contact shape, contact category, etc. The main recognition process is as follows Figure 7 shown.

[0182] Components of the present invention:

[0183] like Figure 2 This is a schematic diagram of the overall components of the visual tactile sensor system. The entire system includes:

[0184] Grating layer 101, reflective layer 102, flexible encapsulation layer 103, acrylic substrate (acrylic layer) 200, light source 300, imaging system 400, mechanical gripper 500, outer frame 600, host computer 700.

[0185] The outer frame 600 can be made of PLA material by 3D printing, and the light source 300 and the imaging system 400 are both fixed to the outer frame 600 by screws through customized 3D printed parts as connectors. The functions of each part are described as follows:

[0186] The flexible contact layer 100 composed of the grating layer 101, the reflective layer 102, and the flexible encapsulation layer 103 is used as the main sensing unit in the system to transmit the contact information of the outside world to the system; the acrylic substrate 200 is placed under the flexible encapsulation layer 103 to support the flexible encapsulation layer 103 and limit the deformation range of the elastomer (the elastomer includes the grating layer 101, the reflective layer 102, and the flexible encapsulation layer 103) to prevent the external load from being too large and damaging the elastomer; the light source 300 provides a light source for the system; the imaging system 400 is used to collect the diffraction image generated by the flexible contact layer; the host computer 700 is used to store and analyze the diffraction image, as well as classify and evaluate the external contact information to realize the contact six-axis force (F x,F y ,F z ,R x ,R y ,R z ), recognition of multimodal tactile information such as fine texture, contour, displacement, etc., among which F x ,F y ,F z Indicates Figure 2 The components of force in each direction shown in the coordinate system, R x ,R y ,R z Indicates Figure 2 The torque in each direction shown in the coordinate system.

[0187] In this invention, there is no restriction on the types of light source 300 and imaging system 400, and the most suitable scheme can be selected according to the actual scene and needs, and can be flexibly replaced. For example, the light source 300 can be a laser, collimated polychromatic light, divergent polychromatic light, etc.; the imaging system 400 can be a small PCB camera, an industrial camera, a CCD and a CMOS sensor, etc. The size and shape of the outer frame 600 are determined by the size and relative position of the first four parts, and are not restricted in this invention. In addition, the type of material for 3D printing can be selected according to the actual situation, and can be replaced with a transparent or opaque type according to the needs, and is not restricted here.

[0188] The analytical method of the present invention:

[0189] The imaging system 400 is responsible for collecting the diffraction image of the elastic body surface and sending the diffraction image to the host computer 700. The host computer 700 extracts the feature information of the diffraction image by running software, and completes the recognition of tactile information after comparing the feature information of the previous and next frames. Since the image contains very rich information, the features that can be extracted include but are not limited to color, brightness and other information.

[0190] Beneficial effects of the present invention:

[0191] The present invention designs a visual tactile sensor based on a grating structure, which can realize multi-axis force perception of external stimuli. The proposed method for making a "sandwich" structure elastomer can add a micro-nanoscale grating structure inside the elastomer, avoiding exposing the delicate grating structure to the contact surface, and effectively extending the service life of the sensor. The easy-to-detach structure designed on the outer frame facilitates the replacement of the damaged elastomer, making the sensor convenient and long-term usable.

[0192] The key innovation point of this solution is the structural design and manufacturing process of this system.

[0193] First, using optical means to achieve tactile perception can achieve high-resolution recognition and avoid problems such as complex wiring.

[0194] Second, unlike other visual tactile sensors, this solution uses the diffraction image generated by the grating to encode the external multi-axis force. The imaging system records and collects image information, establishes the relationship between tactile information and diffraction images, and realizes the tactile perception function.

[0195] Third, by adding a grating structure inside the elastic body, the change of the diffraction image is used to represent the tactile information. This method solves the problem that traditional sensors are difficult to decouple multimodal tactile. The image is rich in information and has high resolution. The changes of different feature quantities in the image can be extracted, and the contact position, contact six-axis force (F x ,F y ,F z ,R x ,R y ,R z ), recognition of multimodal tactile information such as fine texture, contour, displacement, etc.

[0196] Fourthly, in order to protect the fine grating structure, the present invention proposes and prepares a flexible contact layer with a "sandwich" structure. This "sandwich" structure can add the grating inside the elastomer, while utilizing the grating characteristics while maximally protecting the integrity of the grating structure, thereby increasing the robustness and wear resistance of the sensor.

[0197] Fifth, each part of the sensor adopts a separate structure, which can meet different scenarios and needs by changing the relative positions of the imaging system and the light source, and it is convenient to replace damaged parts, making the sensor highly adaptable, creative and long-term practical.

[0198] As an optional embodiment, the present invention also has the following scheme:

[0199] 1. There is no restriction on the shape and size of the outer frame. If it is not designed to be square, it can also be any other shape such as oval, triangle, etc.

[0200] 2. There is no restriction on the shape and material of the flexible contact layer. The shape of the flexible contact layer can be square, circular, triangular or any other shape. This solution uses PDMS material, and other elastomers such as Eco-Flecx or any other material can also be selected according to actual conditions.

[0201] 3. No restrictions are set on the structure of the grating layer. The grating layer can be of any shape, and the grating constant is adjustable. As long as the grating can produce a diffraction image, it is within the protection scope of this patent.

[0202] 4. There is no restriction on the material of the reflective layer on the grating structure. The material selected in the present invention is aluminum, and any other material with reflective properties may also be selected.

[0203] 5. No restrictions are set on the incident angle and diffraction angle of the system. The incident angle and diffraction angle can be changed by changing the relative position of the light source and the imaging system.

[0204] 6. No restrictions are set on image feature analysis methods. Any image analysis method except deep learning method is a variant of this solution.

[0205] 7. Any addition of a structure that has no actual function is a variation of this solution.

[0206] 8. There is no restriction on the image analysis method. In this example, it is proposed to judge the intensity and chromaticity characteristics of the image, but light information also includes polarization and phase, so other analysis methods are all variations of this solution.

[0207] 9. No restrictions are set on the material of the grating layer. The grating layer can be made of any material, as long as the grating structure has the ability to deform, it is within the protection scope of this patent.

[0208] 10. No restrictions are set on the incident light source of the system. The light source selected in the present invention is an LED white light source, and light sources of different colors, intensities, and categories can also be selected.

[0209] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for preparing a visual-tactile sensor based on a grating structure, the method comprising: Design grating layer; According to the grating layer, a flexible contact layer is provided; Establishing a mapping relationship between the diffraction image and the contact information; Obtaining visual-tactile sensing information according to the flexible contact layer and the mapping relationship; The step of establishing a mapping relationship between the diffraction image and the contact information includes: Acquire contact information; the contact information includes multi-axis contact force, posture, and position information; the contact information is measured by a sensor; the multi-axis contact force includes normal force, tangential force, and torque; Acquire a diffraction image; Extracting characteristic information of the grating layer according to the contact information and the diffraction image; the characteristic information includes a contact boundary, brightness characteristics and chromaticity characteristics in the image; The mapping relationship between the grating layer and the contact information is established by using the characteristic information.

2. The method according to claim 1, characterized in that The designed grating layer comprises: Setting the elasticity of the grating and the compressibility of the grating; Setting a periodic structure of the grating; Arrange the diffraction units of the grating; A deformable material is provided for the grating; the deformable material includes flexible plastic and silica gel; Set the size of the grating and the shape of the grating.

3. The method according to claim 2, characterized in that The step of setting the periodic structure of the grating comprises: The grating is arranged into a periodic groove or micro-column structure by periodic arrangement; The grating is arranged into a periodic micro-protrusion structure.

4. The method according to claim 2, characterized in that: The diffraction unit of the grating layout includes: Using arrayed slits as the diffraction unit; Using arrayed grooves as the diffraction unit; The diffraction elements are arranged on the surface of the grating.

5. The method according to claim 1, characterized in that The formulas used in designing the grating layer include: d(sini±sinθ)=kλ Where d is the grating constant, θ is the diffraction angle, k is the diffraction order, λ is the wavelength, and i is the incident angle; When the diffracted light and the incident light are on the same side of the normal, the sign in the brackets on the left side of the above formula is positive; when the diffracted light and the incident light are on different sides of the normal, the sign in the brackets on the left side of the above formula is negative.

6. The method according to claim 1, characterized in that The step of providing a flexible contact layer according to the grating layer comprises: According to the microscopic texture of the grating layer, a flexible contact surface is manufactured using a flexible material; matching the surface properties of the flexible contact surface with the grating layer; The flexible contact surface is covered on the surface of the grating layer and coated and pressed to complete the flexible contact layer.

7. The method according to claim 1, characterized in that The obtaining of visual-tactile sensing information according to the flexible contact layer and the mapping relationship includes: Acquiring transmission information of the flexible contact layer; The transmitted information is processed using the mapping relationship to obtain visual-tactile sensing information.

8. The method according to claim 1, characterized in that The method further comprises: Setting components of a visual-tactile sensor system; the components of the visual-tactile sensor system include a grating layer, a reflection layer, a flexible packaging layer, an acrylic substrate, a light source, an imaging system, a mechanical gripper, an outer frame, and a host computer; Using organic silicon compounds to prepare elastomers; the elastomers include polydimethylsiloxane elastomers, styrene-based thermoplastic elastomers, polyurethane-based thermoplastic elastomers, polyolefin-based thermoplastic elastomers, and polyamide-based thermoplastic elastomers; Adding the grating structure inside the elastic body and using the change of the diffraction image to represent the tactile information; Placing the acrylic substrate below the flexible packaging layer; the acrylic substrate comprises a high-transparency acrylic substrate; The acrylic substrate supports the flexible packaging layer; The outer frame is manufactured using a model preparation process; the model preparation process includes 3D printing; The light source and the imaging system are fixed on the outer frame by screws; the imaging system includes a CMOS camera, a CCD camera, and a spectrometer.

9. A visual-tactile sensor based on a grating structure, obtained by the method for preparing a visual-tactile sensor based on a grating structure according to any one of claims 1 to 8, characterized in that: The visual-tactile sensor includes a grating layer, a reflective layer, a flexible packaging layer, an acrylic substrate, a light source, an imaging system, a mechanical gripper, an outer frame, and a host computer; The grating layer is located above the reflective layer; The reflective layer is located above the flexible packaging layer; The flexible packaging layer is located above the acrylic substrate; The acrylic substrate is located above the outer frame; The light source is located within the outer frame; The imaging system is located within the outer frame; The host computer is connected to the imaging system network; The mechanical gripper is connected to the imaging system network.

Images