Tactile sensing method and device based on optical fiber conduction

Abstract

The invention discloses a tactile sensing method and device based on optical fiber conduction. The tactile sensing method comprises the following steps: S1, designing and manufacturing a contact end of a tactile sensing device; s2, designing and manufacturing an acquisition end of the tactile sensing device; s3, collecting a mechanical analysis training data set and training the model; and S4, observing the acquisition end by using an event camera to obtain event flow data, and reasoning mechanical information. According to the invention, the movement condition of the mark points at the contact end of the flexible tactile sensor is converted into the brightness change of light in each optical fiber of each mark point corresponding to the data acquisition end by using the optical fiber bundle, so that the volume of the tactile sensor is compressed, and the tactile sensor is more compact. In addition, the characteristic of low-data-bandwidth high-speed dynamic observation of the event camera is combined with a traditional flexible touch sensing device and a machine learning technology, and continuous, stable and high-speed mechanical quantity output is achieved.

Description

Technical Field

[0001] The present invention relates to the field of tactile sensors, and in particular to a tactile sensing method and device based on optical fiber conduction. Background Art

[0002] Touch is one of the key ways for humans to perceive the world, and together with vision, hearing and smell, it forms a powerful perceptual network. When robots perform complex operations and other tasks that require close contact, touch can provide more direct, accurate and rich proprioceptive information than vision, which is crucial for generating more reliable operation and control strategies. Despite this, existing tactile sensors still have some limitations in terms of compactness and rapid response to mechanical changes on the contact surface.

[0003] In the past few decades, various types of tactile sensors with different principles have emerged in large numbers, including capacitive, piezoelectric, piezoresistive, magnetic field, photoelectric, etc. However, the above traditional pressure array tactile sensors often find it difficult to obtain relatively rich sensing information. In response to the above problems, vision-based optical tactile sensors have gradually become a hot topic in the field of tactile sensor research. They usually consist of three main parts: a soft elastomer that deforms into the shape of an object when in contact; a camera on the back to observe the deformation of the elastomer; and an LED to illuminate the space between the elastomer and the webcam. Due to the use of a camera, visual tactile sensors can provide high-resolution tactile images of the deformation of the contacted object, which contain a large amount of tactile information. With its advantages of easy manufacturing, high resolution, and multimodal sensing capabilities, it has appeared in various robotic systems, such as GelSight, GelSlim, TacTip, Soft-bubble, FingerVision, DIGIT, and GelStereo.

[0004] When faced with long-term, high-frame-rate dynamic target measurement scenarios, traditional CMOS or CCD-based cameras will generate a large amount of data during the measurement process, complicating the measurement process and making it difficult to achieve high-speed real-time measurement requirements. Therefore, tactile sensors based on event cameras came into being.

[0005] An event camera is a new type of biologically inspired visual sensor, sometimes also called a dynamic vision sensor (DVS). Compared with standard cameras that capture images at a fixed frame rate, it has the characteristics of low bandwidth, low latency, high dynamic range, low power consumption, and high temporal resolution. Today, event-based cameras are gradually being applied to computer vision tasks related to motion estimation, such as feature point tracking, spatial super-resolution, and object / gesture recognition. They can also be used in fields such as 3D reconstruction, optical flow estimation, video frame interpolation, and simultaneous localization and mapping (SLAM). For a single pixel, an event camera outputs event information only when the received light intensity changes, which makes its data bandwidth extremely low and very suitable for scenarios where high-speed mechanical estimation is performed based on marker displacement. Each data output includes the address of the event (corresponding to the pixel position), the nature (brightening or darkening), and the time when the event occurred.

[0006] However, due to factors such as the focal length and field of view of various cameras, the above-mentioned visual-tactile sensors are often large in size and difficult to integrate into various compact manipulators. Summary of the invention

[0007] In response to the above problems, the present application provides a tactile sensing method and device based on optical fiber conduction. The device uses an optical fiber bundle to convert the movement of the marking point at the contact end of the flexible tactile sensor into the brightness change of the light in the optical fiber corresponding to the marking point at the data acquisition end, which significantly compresses the volume of the tactile sensor and makes it more compact and portable.

[0008] In addition, the present invention combines the low data bandwidth and high-speed dynamic observation capabilities of the event camera with traditional flexible tactile sensing devices and advanced machine learning technology to achieve continuous, stable and high-speed output of mechanical quantities. It not only solves the problem of large size of existing flexible tactile sensors, but also significantly improves the ability to quickly respond to mechanical changes on the contact surface.

[0009] The present invention provides a tactile sensing method based on optical fiber conduction, the method comprising the following steps: Step 1: Design and manufacture of the contact end of the tactile sensor device; Step 2: Design and manufacture of the acquisition end of the tactile sensor device; Step 3: Acquisition of mechanical analysis training data set and training of the model: Before training the convolutional neural network, a large amount of event stream data and the corresponding three-dimensional force and torque data of the contact surface are collected by using a calibrated six-dimensional force sensor to collect event stream data; Normalize the collected data to ensure that the data is consistent in different scales and directions, so that the convolutional neural network can learn effective features from the data, thereby improving its performance in practical applications; Step 4: Use the event camera to observe the acquisition end to obtain event stream data and infer mechanical information.

[0010] Preferably, the event stream data acquisition process in step 3 is: installing contact probes with different surface areas and surface shapes on the ATINano17 six-axis force sensor, and then pressing the probes on the contact end surface, while using an event camera to capture event stream data and record the readings of the force sensor, which will be used as a baseline value for training.

[0011] Preferably, in step 4, the event camera captures event stream data for mechanical information reasoning, and the specific steps are: 1. When pressure is applied to the contact end, the marking point on the flexible contact part will be displaced. The displacement changes the light received by the optical fiber at the contact end, and the change in light is transmitted to the collection end of the optical fiber. Second, the high-resolution event camera carefully observes the cross section of the optical fiber at the collection end, captures the event stream data at each point, and records the changes in light intensity on the cross section of the optical fiber; 3. The captured event stream data is then converted into tensor form, and the tensor is input into a pre-trained force estimation model. The model can parse and learn complex patterns in the data, including the relationship between light changes and mechanical information. The model can accurately infer the mechanical information of the contact surface, including the magnitude and distribution of the applied force.

[0012] The present invention also provides a tactile sensing device applied to the above-mentioned tactile sensing method based on optical fiber conduction, comprising a tactile sensing device contact end and a tactile sensing device collection end, wherein the tactile sensing device collection end is connected to the tactile sensing device contact end, wherein: The contact end of the tactile sensing device includes a housing, a coaxial optical fiber, an optical fiber mounting plate, an ultraviolet shielding film, an ultraviolet lighting LED, a transparent elastic gel, a marking point array, an ultraviolet fluorescent background layer and a contact layer, a plurality of coaxial optical fibers are mounted on the optical fiber mounting plate, and the optical fiber mounting plate is mounted in the housing; A layer of ultraviolet shielding film is fixed on one side of the transparent elastic gel, a black marking point array made of iron powder is sprayed on the surface of the other side of the transparent elastic gel, and a layer of ultraviolet fluorescent powder is sprayed on the transparent elastic gel on the same side as the marking point array to form an ultraviolet fluorescent background layer, and a contact layer composed of black color paste is sprayed on the ultraviolet fluorescent background layer, and the ultraviolet lighting LED board is installed on the side of the transparent elastic gel.

[0013] Preferably, the multiple coaxial optical fibers are mounted on the optical fiber mounting plate and then polished into a smooth surface.

[0014] Preferably, one side of the ultraviolet shielding film is adhered to the smooth surface of the optical fiber mounting plate.

[0015] Preferably, the optical fiber mounting plate is made of an opaque black material, and an 8x12 array of through holes is provided on the surface of the optical fiber mounting plate.

[0016] Preferably, a layer of black ultraviolet light-curing glue is injected into the interlayer between the back side of the optical fiber installation plate and the outer shell.

[0017] Preferably, the collection end of the tactile sensing device includes a housing, a fiber optic hub and an event camera. The event camera is installed on the top of the housing, and the fiber optic hub is installed on the bottom of the housing. There are four groups of 8x12 array through holes on the fiber optic hub. The other end of the optical fiber located at the corresponding hole position of the contact end is installed at the collection end.

[0018] Preferably, LED lights for assisting the camera in locating each hole are installed on the four sides of the top surface of the optical fiber hub plate.

[0019] Advantages of the present invention: The present invention utilizes an optical fiber bundle to convert the movement of the marking point at the contact end of the flexible tactile sensor into the brightness change of the light in each optical fiber corresponding to each marking point at the data acquisition end, thereby compressing the volume of the tactile sensor and making it more compact; in addition, the present invention combines the characteristics of low data bandwidth and high-speed dynamic observation of the event camera with traditional flexible tactile sensing devices and machine learning technology to achieve continuous, stable and high-speed output of mechanical quantities, solving the problem of large volume of existing flexible tactile sensors and the deficiency of rapid response to mechanical changes on the contact surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a tactile sensing method based on optical fiber conduction of the present invention.

[0021] Figure 2 It is a structural schematic diagram of the contact end of the tactile sensor device of the present invention.

[0022] Figure 3 It is a structural schematic diagram of the acquisition end of the tactile sensing device of the present invention.

[0023] Figure numerals: 1-coaxial optical fiber, 2-housing, 3-optical fiber mounting plate, 4-ultraviolet shielding film, 5-transparent elastic gel, 6-marking point array, 7-ultraviolet fluorescent background layer, 8-contact layer, 9-ultraviolet lighting LED board, 10-optical fiber hub board, 11-event camera. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] like Figure 1 , Figure 2 and Figure 3 As shown, this embodiment provides a tactile sensing method based on optical fiber conduction, which solves the problem of large volume of existing flexible tactile sensors and the deficiency of rapid response to mechanical changes of the contact surface, and includes the following steps: Step 1: Design and manufacture of the contact end of the tactile sensor device; Step 2: Design and manufacture of the acquisition end of the tactile sensor device; Step 3: Acquisition of mechanical analysis training data set and training of the model: Before training the convolutional neural network, a large amount of event stream data and the corresponding three-dimensional force and torque data of the contact surface are collected by using a calibrated six-dimensional force sensor to collect event stream data; Normalize the collected data to ensure that the data is consistent in different scales and directions, so that the convolutional neural network can learn effective features from the data, thereby improving its performance in practical applications; Step 4: Use the event camera 11 to observe the acquisition end to obtain event stream data and infer mechanical information.

[0026] The event stream data acquisition process in step 3 is: install contact probes with different surface areas and surface shapes on the ATINano17 six-axis force sensor, then press the probes on the contact end surface, and use the event camera 11 to capture the event stream data and record the readings of the force sensor. The readings will be used as the baseline value for training.

[0027] Step 4: Use the event camera 10 to observe the acquisition end to obtain event stream data and infer mechanical information.

[0028] In this example, the Prophesee EVK4 event camera 11 is used, and the nominal resolution of this camera is 1280*720.

[0029] In actual applications, when pressure is applied to the contact end, the marking point on the flexible contact part will move accordingly. This displacement changes the light received by the optical fiber at the contact end, and the change in light is transmitted to the collection end of the optical fiber.

[0030] At this time, the high-resolution event camera 11 carefully observes the cross section of the optical fiber at the collection end and captures the event stream data at each point. These data record the changes in light intensity on the cross section of the optical fiber, which are directly related to the pressure applied to the contact end.

[0031] The captured event stream data is then converted into tensor form, a standard format for processing data by machine learning models. These tensors are input into a pre-trained force estimation model that is able to parse and learn complex patterns in the data, including the relationship between light changes and mechanical information.

[0032] Through this process, the model is able to accurately infer the mechanical information of the contact surface, including the magnitude and distribution of the applied forces.

[0033] The present invention also provides a tactile sensing device based on the tactile sensing method of optical fiber conduction, comprising a tactile sensing device contact end and a tactile sensing device collection end, wherein the tactile sensing device collection end is connected to the tactile sensing device contact end, wherein: The contact end of the tactile sensing device comprises a housing 2, a coaxial optical fiber 1, an optical fiber mounting plate 3, an ultraviolet shielding film 4, an ultraviolet lighting LED 9, a transparent elastic gel 5, a marking point array 6, an ultraviolet fluorescent background layer 7 and a contact layer 8, a plurality of coaxial optical fibers 1 are mounted on the optical fiber mounting plate 3, and the optical fiber mounting plate 3 is mounted in the housing 2; A layer of ultraviolet shielding film 4 is fixed on one side of the transparent elastic gel 5, and a black marking point array 6 made of iron powder is sprayed on the surface of the other side of the transparent elastic gel 5, and a layer of ultraviolet fluorescent powder is sprayed on the transparent elastic gel 5 on the same side as the marking point array 6 to form an ultraviolet fluorescent background layer 7, and a contact layer 8 composed of black color paste is sprayed on the ultraviolet fluorescent background layer 7, and the ultraviolet lighting LED board 9 is installed on the side of the transparent elastic gel 5.

[0034] The contact end body includes a housing 2, a coaxial optical fiber 1, an optical fiber mounting plate 3, an ultraviolet shielding film 4, an ultraviolet lighting LED 9, a transparent elastic gel 5, a marking point array 6, an ultraviolet fluorescent background layer 7 and a contact layer 8.

[0035] Specifically, each contact end is equipped with three coaxial optical fibers 1, each with a diameter of 2.2 mm; in each optical fiber bundle, there are 32 evenly distributed smaller optical fibers with a diameter of 0.25 mm. These 96 optical fibers are carefully arranged and evenly distributed at intervals of 1 mm on a fiber optic mounting plate 3 that is 12 mm long and 8 mm wide.

[0036] The fiber optic mounting plate 3 is 0.8 mm thick and is made of opaque black material. On its surface, there is an 8×12 array, and each hole has a diameter of 0.30 mm. These holes are specially tailored for the 96 optical fibers so that they can be inserted accurately. After the optical fiber is inserted, the surface of the contact end will be polished to be smooth to ensure that the optical fiber can receive sufficient light.

[0037] After the installation of the optical fiber is completed, the optical fiber installation plate 3 is combined with the housing 2.

[0038] In order to further strengthen the structure and prevent mutual interference between optical fiber bundles, a layer of black UV light-curing glue is injected into the interlayer between the back of the optical fiber mounting plate 3 and the outer shell 2; this glue cures rapidly under ultraviolet light, which not only provides a solid support for the optical fiber, but also ensures the clear and accurate transmission of the optical fiber signal. This design not only ensures the stability of the structure, but also optimizes the performance of the optical fiber, providing a solid foundation for accurate mechanical information collection.

[0039] Specifically, when making the flexible contact part, you first need to make a mold with a size of 8mm×12mm×1.5mm to accommodate the transparent elastic gel 5. A layer of ultraviolet shielding film 4 is fixed at the bottom of the mold to prevent ultraviolet rays from affecting the gel. Then, pour AB glue into the mold and wait for it to completely solidify.

[0040] After curing, a mold with an 8×12 array of 0.6mm diameter holes was used to spray black marking dots made of iron powder on the surface of the gel. These marking dots formed a clear pattern on the gel, providing a benchmark for subsequent displacement detection.

[0041] After the marking point is made, a layer of ultraviolet fluorescent powder is sprayed on the surface of the gel to form an ultraviolet fluorescent background layer 7. This background layer appears white under visible light, but emits red fluorescence under ultraviolet light, which is in sharp contrast with the black marking point. When the marking point is no longer directly above the optical fiber due to displacement, the red fluorescence received by the optical fiber will increase significantly, so that slight changes in the contact surface can be detected.

[0042] A contact layer 8 composed of a black paste is sprayed on the ultraviolet fluorescent background layer 7. This contact layer 8 can not only shield the external ambient light and protect the gel and coating from damage, but also ensure that the light received by the optical fiber mainly comes from the fluorescent background layer inside the gel.

[0043] Finally, the finished flexible contact part is taken out from the mold, with a UV shielding film on the bottom and a coated transparent elastic gel 5 on the top, thus completing the production of the flexible contact part. This delicate production process ensures that the flexible contact part can provide accurate mechanical information in practical applications, while also protecting the internal structure from the influence of the external environment.

[0044] After the flexible contact portion is manufactured, the next step is to adhere one side of the ultraviolet shielding film 4 to the smooth surface of the optical fiber mounting plate 3 .

[0045] After the pasting is completed, the ultraviolet lighting LED board 9 is installed on the side of the transparent elastic gel 5, which will become the lower shell of the tactile sensing device. The installation position of the ultraviolet lighting LED board 9 needs to be precise to ensure that the ultraviolet light can be evenly irradiated on the gel to activate the ultraviolet fluorescent background layer 7 to emit red fluorescence under ultraviolet light. The contact end of the entire tactile sensing device is completed.

[0046] The collection end includes a housing 2, a fiber optic hub 10 and an event camera 11. The fiber optic hub 10 is 30 mm long and 20 mm wide and is installed at the bottom of the housing 2. It has four groups of 8×12 arrays and 0.3 mm diameter through holes. Each fiber optic hub 10 can collect data from four contact ends at the same time. The installation method of the optical fiber here is the same as that of the contact end. The other end of the optical fiber located at the corresponding hole position of the contact end can be installed at the collection end. The four sides of the top surface of the fiber optic hub 10 are also equipped with LED lights for assisting the camera to locate each hole position.

[0047] The event camera 11 is installed on the top of the housing 2. It monitors the brightness changes of light in the optical fiber from the inside, thereby collecting event flow information about the displacement of the marking point. In order to ensure that the event camera 11 is not disturbed by external light changes, the event camera 11 and the optical fiber hub 10 are both encapsulated in an opaque housing 2 to ensure the accuracy and stability of data collection. This design aims to improve the data collection efficiency and accuracy of the tactile sensor while taking into account the cost-effectiveness and portability of the equipment.

[0048] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A tactile sensing method based on optical fiber conduction, characterized in that: The method comprises the following steps: Step 1: Design and manufacture of the contact end of the tactile sensor device; Step 2: Design and manufacture of the acquisition end of the tactile sensor device; Step 3: Acquisition of mechanical analysis training data set and training of the model: Before training the convolutional neural network, a large amount of event stream data and the corresponding three-dimensional force and torque data of the contact surface are collected by using a calibrated six-dimensional force sensor to collect event stream data; Normalize the collected data to ensure that the data is consistent in different scales and directions, so that the convolutional neural network can learn effective features from the data, thereby improving its performance in practical applications; Step 4: Use the event camera (11) to observe the acquisition end to obtain event stream data and infer mechanical information.

2. A tactile sensing method based on optical fiber transmission as claimed in claim 1, characterized in that: The process of acquiring the event stream data in step 3 is as follows: installing contact probes with different surface areas and surface shapes on the ATINano17 six-axis force sensor, then pressing the probes onto the contact end surface, and simultaneously using an event camera (11) to capture the event stream data and record the readings of the force sensor, which will serve as the baseline value for training.

3. A tactile sensing method based on optical fiber conduction as claimed in claim 2, characterized in that: Step 4: The event camera (11) captures the event stream data for mechanical information reasoning. The specific steps are as follows:

1. When pressure is applied to the contact end, the marking point on the flexible contact part will be displaced. The displacement changes the light received by the optical fiber at the contact end, and the change in light is transmitted to the collection end of the optical fiber. Second, a high-resolution event camera (11) conducts detailed observations of the cross section of the optical fiber at the collection end, captures event stream data at each point, and records changes in light intensity on the cross section of the optical fiber; 3. The captured event stream data is then converted into tensor form, and the tensor is input into a pre-trained force estimation model. The model can parse and learn complex patterns in the data, including the relationship between light changes and mechanical information. The model can accurately infer the mechanical information of the contact surface, including the magnitude and distribution of the applied force.

4. A tactile sensing device applied to a tactile sensing method based on optical fiber conduction as claimed in any one of claims 1 to 3, characterized in that: It comprises a tactile sensor device contact end and a tactile sensor device collection end, wherein the tactile sensor device collection end is connected to the tactile sensor device contact end, wherein: The contact end of the tactile sensing device comprises a housing (2), a coaxial optical fiber (1), an optical fiber mounting plate (3), an ultraviolet shielding film (4), an ultraviolet lighting LED (9), a transparent elastic gel (5), a marking point array (6), an ultraviolet fluorescent background layer (7) and a contact layer (8), wherein a plurality of coaxial optical fibers (1) are mounted on the optical fiber mounting plate (3), and the optical fiber mounting plate (3) is mounted in the housing (2); A layer of ultraviolet shielding film (4) is fixed on one side of the transparent elastic gel (5), a black marking point array (6) made of iron powder is sprayed on the surface of the other side of the transparent elastic gel (5), and a layer of ultraviolet fluorescent powder is sprayed on the transparent elastic gel (5) on the same side as the marking point array (6) to form an ultraviolet fluorescent background layer (7), and a contact layer (8) composed of black color paste is sprayed on the ultraviolet fluorescent background layer (7), and the ultraviolet lighting LED board (9) is installed on the side of the transparent elastic gel (5).

5. A tactile sensing device as claimed in claim 4, characterized in that: A plurality of coaxial optical fibers (1) are mounted on an optical fiber mounting plate (3) and then polished into a smooth surface.

6. A tactile sensing device as claimed in claim 5, characterized in that: One side of the ultraviolet shielding film (4) is adhered to the smooth surface of the optical fiber installation plate (3).

7. A tactile sensing device as claimed in claim 6, characterized in that: The optical fiber installation plate (3) is made of an opaque black material, and an 8x12 array through hole is provided on the surface of the optical fiber installation plate (3).

8. A tactile sensing device as claimed in claim 7, characterized in that: A layer of black ultraviolet light-curing glue is injected into the interlayer between the back side of the optical fiber installation plate (3) and the outer shell (2).

9. A tactile sensing device as claimed in claim 8, characterized in that: The collection end of the tactile sensing device comprises a housing (2), an optical fiber hub (10) and an event camera (11), wherein the event camera (11) is mounted on the top of the housing (2), and the optical fiber hub (10) is mounted on the bottom of the housing (2). The optical fiber hub (10) has four groups of through holes in an 8x12 array, and the other end of the optical fiber located at the hole position corresponding to the contact end is mounted on the collection end.

10. A tactile sensing device as claimed in claim 9, characterized in that: LED lights for assisting the camera in locating each hole position are also installed on the four sides of the top surface of the optical fiber hub plate (10).

Images