Three-point tactile fusion optical fiber sensing system based on multilayer LSTM (Long Short Term Memory)
Through a three-point tactile fusion fiber sensing system based on multi-layer LSTM, the fiber grating sensor is used to collect vibration and stress signals and train the network model, the limitations of tactile information acquisition and sensor complexity in the existing technology are solved, and efficient Accurate object surface recognition.
Patent Information
- Application Number
- CN202410952822.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-16
AI Technical Summary
Existing tactile sensors have limitations when obtaining multiple tactile information of objects, especially difficult to obtain multiple tactile information at the same time. Multi-tactile perception of human-like skin is difficult to achieve. Traditional sensors are complex in packaging, many wires and large in weight, resulting in Difficulty in identification.
Design a three-point tactile fusion fiber sensing system based on multi-layer LSTM, collect vibration signals and stress signals through fiber grating sensors, train multi-layer LSTM network models to integrate multi-point stress and vibration information, simulate human multi-point tactile perception, and obtain Rich tactile information for object surface recognition.
It realizes high recognition efficiency that is anti-electromagnetic interference, small size and light weight, and can detect static stress and dynamic vibration at the same time, imitate human multi-touch perception, promote up to multi-point perception, and improve the accuracy of object surface recognition.
Smart Images

Figure CN118936529A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of optical fiber sensing and is a three-point tactile fusion optical fiber sensing system based on multi-layer LSTM. Background Art
[0002] With the continuous development of the times and the continuous growth of the economy, intelligent mechanization has become a new era development strategy driven by the inherent needs of industries such as industry and medical care. Its development involves the exploration of sensors, further processing and research of sensor-collected data, etc. Touch is a basic function of human beings to perceive the surrounding environment, and it is also an important means for sensors to imitate human perception of objects. The development of sensor tactile perception function can enable sensors to better detect and identify objects and help people complete more complex content.
[0003] For tactile sensing, when in contact with an object, it can sense information such as hardness and temperature, while more subtle information such as roughness and texture can only be sensed by sliding relative to the object. Although current tactile sensing can achieve certain object recognition using technologies such as resistance and piezoelectricity, it has certain limitations in the following aspects: it is difficult to obtain multiple tactile information of an object at the same time; it is difficult to implement multi-point tactile perception similar to human skin; there is insufficient progress in the algorithm for fusion recognition of multiple tactile features; traditional sensors have complex packaging, many wires, and heavy weight, which makes recognition difficult. Summary of the invention
[0004] The present invention proposes a three-point tactile fusion fiber optic sensing system based on multi-layer LSTM. The tactile sensor used includes a system optical path and a fiber grating part. The tactile information is collected by the fiber grating sensor to collect vibration signals and stress signals (vibration signals identify roughness, and stress signals identify softness and hardness). The network model that can fuse multi-point stress and vibration information is trained to simulate human multi-point tactile perception, obtain rich tactile information, and identify the surface of objects. The scheme provided by the present invention is as follows:
[0005] A three-point tactile fusion optical fiber sensing system based on multi-layer LSTM includes the following steps:
[0006] The first step is to collect tactile data
[0007] Design a tactile sensing model, build a tactile information sensing optical path based on fiber grating, and realize synchronous three-point sensing of the object surface; transmit the collected tactile signals to the host computer; use the robotic arm to drive the tactile sensing model to collect tactile data;
[0008] Step 2: Build a tactile dataset
[0009] The object to be tested is a silicone block with different surface conditions, and its surface tactile information includes hardness and roughness. The multi-touch point tactile data collected in the first step is demodulated and processed, and static stress information and dynamic vibration information are obtained at the same time. The hardness is reflected in the static stress information, and the roughness is reflected in the dynamic vibration information. The obtained static stress information and dynamic vibration information are spliced, aligned, and normalized to form a three-point tactile time series data set for input into the network model for recognition.
[0010] The third step is to build and train a tactile information recognition network model
[0011] The data in the processed three-point tactile time series data set are input into the multi-layer LSTM network model for fusion processing: the static stress and dynamic vibration of each point are regarded as a mode respectively, and feature extraction is performed within and between modes. At the same time, the corresponding timing characteristics of each mode and between modes are grasped, and the stress and vibration information of a single point are extracted and fused; the stress and vibration fusion information of each point are fused again between channels, and each layer of the fusion network uses an LSTM network; after all tactile information is fused, it is input into the gated memory network and the network output layer to obtain the final output result of the network model.
[0012] Furthermore, the first step includes the following steps: the bottom of the tactile sensing model encapsulated with the fiber grating is in direct contact with the surface of the object to be tested, the data is collected by the robot arm, the robot arm is controlled to move, and the three contact point data are collected according to the cycle of moving, pressing down, sliding, and lifting. The tactile sensing model collects light signals after contacting the surface of the object. When the bottom of the tactile sensing model contacts the surface of the object, the pressure sensed causes the fiber grating to deform tangentially and causes the central wavelength of the fiber grating to change. The change information is modulated in the form of signal envelope offset in the interference signal of each scanning frame, and the static stress information can be obtained by demodulation; when the tactile sensing model slides and rubs on the surface of the object, the vibration information sensed by the contact surface is transmitted to the fiber grating, and is coupled in the form of instantaneous phase change in the interference signal of each scanning frame, and the dynamic vibration information can be obtained by demodulation. Data are collected on different object surfaces respectively, and the same object surface is repeatedly collected.
[0013] Furthermore, the second step includes the following steps: for static stress information, the envelope offset calculated for each frame is a scalar, and the envelope offsets of consecutive frames are spliced into a 1×T tensor, where T is the sequence length of the data sample, as a feature vector sequence representing the static stress information; for dynamic vibration information, the vibration waveform demodulated from each frame is taken as a high-dimensional vector, which is downsampled to S dimensions, and the vibration waveforms of consecutive frames are spliced into an S×T tensor as a feature vector sequence representing the dynamic vibration information. The two types of information are strictly aligned in time with the frame as the timestamp; the information of the three touch points is processed in the same way, thereby obtaining the three-point tactile data of a sample, and constructing a three-point tactile time series data set.
[0014] Furthermore, the first step includes the following steps:
[0015] (1) Single-point bimodal feature fusion: For the feature input of each modality, C at time t-1 and x at time t are combined as the LSTM input at time t, which are:
[0016]
[0017] Where s represents stress, v represents vibration, and denote the stress and vibration mode input at the tth moment, respectively. and Represent the memory unit information of stress and vibration mode at the t-1th moment respectively. and The outputs of the two modalities are fused and input into the corresponding dual-modal fusion network, as shown below:
[0018]
[0019] in The stress and vibration modal fusion information is input into the dual-modal fusion network and encoded by LSTM:
[0020]
[0021] is the output of the bimodal fusion network. It is the final memory unit of the LSTM network. The stress and vibration information of the three points are fused in the same way to obtain different outputs of the three points. Then the stress output of each point is Vibration Output Dual-modal fusion output To combine:
[0022]
[0023] According to this combined method, we can obtain Represents the final output results of three single points.
[0024] (2) Three-point feature fusion: The feature fusion information of the three points is combined in pairs and input into the two-point feature fusion layer:
[0025]
[0026] The temporal characteristics between channels are captured by the LSTM network, and the output is obtained:
[0027]
[0028] Where k∈N2, N2={(1,2),(1,3),(2,3)}, represents the two sequences currently being fused. is the final output of the LSTM network, It is the final memory unit of the LSTM network. Finally, the two-point fusion output result and the single-point output result are combined as the input of the three-point feature fusion layer:
[0029]
[0030] Where n = {1,2,3}, k = {(1,2),(1,3),(2,3)}. The output is:
[0031]
[0032] Finally, the output results of all single points, two points, and three points are integrated as the final output of the multi-layer LSTM fusion network to enhance the feature performance of modal fusion:
[0033]
[0034] Where n = {1,2,3}, k = {(1,2),(1,3),(2,3)}.
[0035] (3) Gated memory network: Output of multi-layer LSTM fusion network O t After three neural networks, we get g1, g2, Among them, g1 and g2 are two gate control units, which are controlled by neural networks D1 and D2 respectively. is the cross-modal update unit, u t It is the output of the gated memory network at time t. The gated memory network can fuse the feature information of the previous moment and the current moment.
[0036] (4) Output layer: The input of the output layer is the final state u of the gated memory network tThe output of stress and vibration characteristics in each sensing point is cascaded together to Represents vector concatenation:
[0037]
[0038] The cascaded input is mapped to the output layer using the weight matrix D0, and then converted to a probability distribution o through the softmax function. The output probability distribution o is a probability vector, in which each dimension is the probability that the sample belongs to the corresponding category, thereby generating the corresponding classification recognition result.
[0039] Compared with the prior art, the advantages of the present invention are as follows:
[0040] 1) Compared with the traditional electrical tactile sensor, the fiber Bragg grating tactile sensor used in the present invention has the characteristics of anti-electromagnetic interference, small size, light weight, high sensitivity, etc., can simultaneously realize the detection of static stress and dynamic vibration, has high recognition efficiency, and is more applicable;
[0041] 2) Compared with a single-point tactile sensor, the three-point tactile sensor designed and used in the present invention can simultaneously sense three points of an object, and can be further extended to multi-point perception, imitating the multi-point tactile perception function of humans, and is closer to anthropomorphic goals;
[0042] 3) Compared with the traditional time series modal fusion algorithm, the multi-layer LSTM fusion network model adopted in the present invention can more deeply grasp the feature information of each modality and the interactive feature information between modalities, thereby stably extracting and fully fusing multi-point dual-modal tactile information. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 The following are the design drawings of the sensor bracket used in the present invention. (a) is a schematic diagram of the appearance of the sensor bracket. (b) is a schematic diagram of the design of a separate movable branch.
[0044] Figure 2 This is the fiber grating packaging diagram and complete sensing model diagram used in the present invention
[0045] Figure 3 The light path diagram of the present invention is
[0046] Figure 4 This is the sample data processing result diagram of the present invention
[0047] Figure 5 The network model diagram used in the present invention is
[0048] Figure 6 This is the verification result diagram of the present invention
[0049] Figure 7 It is the system flow chart of the present invention
[0050] Description of Figure Numbers:
[0051] 1,2,3- movable branches; 4- sensor bracket body; 5- rectangular protrusion installed at the end of the robotic arm; 6- bracket cap; 7- sensor bracket protective shell; 11- branch tube body; 12- rectangular protrusion for controlling the downward pressure distance; 13- branch sensor front end. DETAILED DESCRIPTION
[0052] The following is a further detailed description of the embodiments of the present invention in conjunction with the accompanying drawings and formulas, and is specifically implemented in the following steps:
[0053] Step 1: Collect three-point tactile data
[0054] Designing a sensing model for contact objects: The present invention uses Figure 2 The complete sensor model shown is used for data acquisition. First, design Figure 1 The three-head sensor bracket shown in the figure: the left figure numbers 1, 2, and 3 are three identical movable branches (the lifting and falling are controlled by independent springs, and the activities of the three do not affect each other). When sliding on the surface of an object, three-point synchronous sensing can be performed, which can obtain richer tactile information on the surface of the object than single-point contact. The right figure numbers 11, 12, and 13 describe the branch design separately. 11 is the branch tube body, 12 is used to control the downward distance of the branch, and 13 has a built-in spring and a sensor at the bottom; number 4 is the main body of the sensor bracket; the rectangular protrusion shown by number 5 is installed at the end of the robot arm, and the robot arm drives the sensor bracket to slide; number 6 is the bracket cap, which is used to fix the three movable branches; number 7 is the protective shell of the sensor bracket. The fiber grating is as shown in FIG. Figure 2 The package is shown in the silicone finger pad. According to this packaging method, three identical silicone finger pads are made and respectively installed in the depressions at the bottom of each branch of the sensor bracket with adhesive glue to form a complete sensor model. Finally, the complete sensor model is fixed on the robotic arm, and the robotic arm activity is set by the program, so that the robotic arm drives the sensor model to collect data on the surface of the object.
[0055] Build the tactile information sensing optical path: Figure 3The three-point tactile information sensing optical path is shown, including a sweeping light source module, a 2*2 coupler, a fiber Bragg grating with a reflection bandwidth of 1nm, a Faraday rotating mirror, a photodetector, a data acquisition card, and a host computer data processing. Among them, the sweeping light source module provides detection light for the sensing system, which enters the three middle couplers through the left coupler. Because the three middle paths are the same, one of them is used to illustrate: the detection light is divided into two beams through the coupler, and enters the fiber Bragg grating and the Faraday rotating mirror respectively; the Faraday rotating mirror and the fiber Bragg grating form a Michelson interferometer through the coupler, the fiber Bragg grating is the detection arm, and the Faraday rotating mirror is the reference arm, and the two paths of light interfere in the Michelson interferometer; the interference light returns to the optical coupler and enters the photodetector; the photodetector converts the optical signal into an analog electrical signal and transmits it to the data acquisition card; the data acquisition card further converts the analog electrical signal into a digital signal and transmits it to the host computer, thereby obtaining the original tactile interference signal.
[0056] Tactile data acquisition process: The bottom of the sensing model encapsulated with fiber grating is in direct contact with the surface of the object to be tested, and the data is collected by the robot arm. A computer is used to write a program to control the movement of the robot arm, and data is collected according to the cycle of moving, pressing, sliding, and lifting. During the collection, the movement is based on the parameters of 10mm pressing distance and 160mm / s sliding speed. After the sensing model contacts the surface of the object, the optical signal is collected and transmitted to the host computer through the sensing optical path. The specific tactile information obtained is: when the bottom of the sensing model contacts the surface of the object, the sensed pressure causes the fiber grating to deform tangentially and causes the central wavelength of the fiber grating to change. The change information is modulated in the interference signal of each scanning frame in the form of signal envelope offset, and the static stress information can be obtained by demodulation; when the sensing model slides and rubs on the surface of the object, the vibration information sensed by the contact surface is transmitted to the fiber grating, and is coupled in the interference signal of each scanning frame in the form of instantaneous phase change, and the dynamic vibration information can be obtained by demodulation. Data were collected on different object surfaces respectively, and the same object surface was repeatedly collected 30 times to obtain comprehensive dynamic and static tactile data.
[0057] The parameters of each module set in the above tactile data acquisition process are: the scanning rate of the sweeping light source module is 50hz, the optical path difference between the two arms is set to 5cm, and the acquisition rate of the data acquisition card is 100KSa / s. The corresponding scanning period of each scanning frame is 20ms, the number of interference signal waves in each scanning frame is about 80, and the interference signal length of each scanning frame is about 2000 sampling points. The maximum frequency of static stress information detection is 25hz, and the maximum frequency of dynamic vibration information detection is 1KHz.
[0058] Step 2: Build a three-point tactile dataset
[0059] Tactile data preprocessing: In the previous step, the three-point signals are collected simultaneously, so the three-way information obtained in this step can be processed synchronously. The data processing process of one way is described below. The processing method refers to patent 202311316484.6. In step 1, the wavelength of the detection light output by the sweeping light source module is linearly related to time, and the wavelength is scanned at a fixed repetition period. In one scanning cycle, when the wavelength of the sweeping light source reaches the reflection wavelength of the fiber grating, the detection light and the reference light interfere with each other on the coupler. After one scanning cycle, an interference signal of a scanning frame is formed, which is transmitted into an electrical signal through a photoelectric detector. Finally, the interference signal g(λ) is collected by the data acquisition card, which is expressed as formula (1):
[0060]
[0061] Where R(λ) is the reflection spectrum of the fiber Bragg grating, which reflects the outer envelope characteristics of the interference signal and contains strain and temperature information. n is the effective refractive index of the fiber, L is the optical path difference of the grating, is the phase modulation of the coherent signal caused by vibration during optical transmission. Therefore, R(λ) contains the required static stress information. The required dynamic vibration information is contained in it.
[0062] The collected interference signal g(λ) needs to be processed by signal demodulation. When the fiber Bragg grating senses strain, its reflection spectrum will change, causing the envelope of the interference signal waveform of each scanning frame to move, so the static stress information is reflected in the interference signal envelope offset (time domain); when the fiber Bragg grating senses vibration, the instantaneous phase of the interference signal will change, so the dynamic vibration information is reflected in the instantaneous phase change of the interference signal (frequency domain). Because the two types of information are in different domains at the same time, they can be demodulated and processed synchronously.
[0063] Demodulate the static stress information: First, the interference signal corresponding to the scanning frame when the robot arm is not in contact with the object is used as the reference signal. To prevent the influence of noise, Gaussian smoothing is used to denoise the obtained interference signal, and the envelope information of each frame signal is extracted by Hilbert transform. Finally, the cross-correlation method is used to calculate the offset between the signal envelope extracted from each frame and the reference signal envelope. The offset is the static stress information in the scanning frame.
[0064] Demodulate the dynamic vibration information: First, perform Gaussian smoothing to remove noise, perform Hilbert transform on the interference signal to extract the envelope, and then use the extracted envelope to normalize the original interference signal. Perform synchronous compression wavelet transform on the normalized interference signal to demodulate the instantaneous frequency, and then perform integral restoration to obtain the instantaneous phase change, whose waveform is the dynamic vibration information in a scan frame.
[0065] Through the above demodulation process, the dynamic and static tactile information of the three points can be obtained synchronously, that is, the static stress and dynamic vibration information of the three points. The objects to be tested in the experiment are silicone blocks with different surface conditions, and their surface tactile information includes hardness and roughness. Hardness is reflected in static stress, and roughness is reflected in dynamic vibration.
[0066] Preparation of tactile data set: In order to input two different data into the network model for training, consider processing the above-obtained data into a feature vector sequence for use. For static stress information, the envelope offset calculated for each frame is a scalar, and the envelope offsets of consecutive frames need to be spliced into a 1×T tensor (the sequence length of the data sample, 70 in this experiment) as a feature vector sequence representing static stress information; for dynamic vibration information, the vibration waveform demodulated from each frame is a 600-dimensional vector. In order to save storage space and computational efficiency, it is downsampled to 150 dimensions, and the vibration waveforms of consecutive frames are spliced into a 150×T tensor as a feature vector sequence representing dynamic vibration information. The two types of information are strictly aligned in time with the frame as the timestamp. The information of the three points is processed according to the above method to obtain the three-point tactile data of a sample.
[0067] Finally, all samples are processed according to the above process to obtain the data of three-point tactile time series data set, which is described as D i ={s 1i ,v 1i ,s 2i ,v 2i ,s 3i ,v 3i ,L i}, where i∈(1,N), represents the i-th sample, and N is the number of samples. k represents the stress mode, v k Represents the vibration mode, k represents the kth channel, and L is a label representing the softness, hardness and roughness of the object surface, which is obtained from the silicone blocks with different surface conditions used in the experiment. Figure 4 Visualization results of tactile data of one of the sample surfaces are shown, with vibration on the left and stress on the right.
[0068] Step 3: Use the prepared data set to train the three-point dynamic and static tactile fusion network model
[0069] Build as Figure 5The multi-layer LSTM fusion network (MLFN) shown in the figure is used for data processing. If the static stress and dynamic vibration of each point are regarded as a mode, the network model can extract features within and between modes, while grasping the corresponding time series characteristics of each mode and between modes. It mainly includes four parts: (1) single-point bimodal feature fusion, including two layers of LSTM; (2) three-point feature fusion, including two layers of LSTM; (3) gated memory network, a multimodal joint time series storage structure, whose storage changes over time and performs cross-modal interaction; (4) output layer, the recognition result is obtained based on the fusion information. The input data is a three-point tactile time series dataset D i ={s 1i ,v 1i ,s 2i ,v 2i ,s 3i ,v 3i ,L i}. First, process the single-point bimodal information fusion, and then process the three-point information fusion.
[0070] (1) Single-point bimodal feature fusion: For the feature input of each modality, C at time t-1 and x at time t are combined as the LSTM input at time t, which are:
[0071]
[0072]
[0073] Where s represents stress, v represents vibration, and denote the stress and vibration mode input at the tth moment, respectively. and Represent the memory unit information of stress and vibration mode at the t-1th moment respectively. and Input into the single-point bimodal feature fusion network, taking stress as an example, as shown below:
[0074]
[0075] The superscript s(v) symbol indicates that the current parameter is the stress (vibration) mode. represents the corresponding network weight, Represents the corresponding network bias, σ is set to the sigmoid activation function, represents the input gate at time t, represents the forget gate at time t, represents the output gate at time t, represents the new memory unit at time t, is the final memory unit at time t. is the final output value at time t. Both stress and vibration are input into the LSTM network in this way to obtain the feature information within the mode. Then the outputs of the two modes are fused and input into the corresponding dual-modal fusion network, as shown below:
[0076]
[0077] in The stress and vibration modal fusion information is input into the dual-modal fusion network and encoded by LSTM:
[0078]
[0079] is the output of the bimodal fusion network. It is the final memory unit of the LSTM network. The stress and vibration information of the three points are fused as above to obtain different outputs of the three points. Then the stress output of each point is Vibration Output Dual-modal fusion output To combine:
[0080]
[0081] According to this combined method, we can obtain Represents the final output results of three single points.
[0082] (2) Three-point feature fusion: First, the feature fusion information of the three points is combined in pairs and input into the two-point feature fusion layer:
[0083]
[0084] The temporal characteristics between channels are captured by the LSTM network, and the output is obtained:
[0085]
[0086] Where k∈N2, N2={(1,2),(1,3),(2,3)}, represents the two sequences currently being fused. is the final output of the LSTM network, It is the final memory unit of the LSTM network. Finally, the two-point fusion output result and the single-point output result are combined as the input of the three-point feature fusion layer:
[0087]
[0088] Where n = {1,2,3}, k = {(1,2),(1,3),(2,3)}. The output is:
[0089]
[0090] Finally, all single-point, two-point, and three-point output results are integrated as the final output of feature fusion to enhance the feature performance of modal fusion:
[0091]
[0092] Where n = {1,2,3}, k = {(1,2),(1,3),(2,3)}.
[0093] (3) Gated Memory Network: It is composed of multiple neural networks and is a unified memory. Its specific components are:
[0094] g1=σ(D1(O t )) (20)
[0095] g2=σ(D2(O t )) (twenty one)
[0096]
[0097]
[0098] Output of multi-layer LSTM fusion network O t After three neural networks, we get g1, g2, Among them, g1 and g2 are two gate control units, which are controlled by neural networks D1 and D2 respectively. is the cross-modal update unit, u t is the output of the gated memory network at time t. The gated memory network can fuse the feature information of the previous moment and the current moment.
[0099] (4) Output layer: The input of the output layer is the final state u of the gated memory network t The output of stress and vibration characteristics in each sensing point is cascaded together to Represents vector concatenation:
[0100]
[0101] The cascaded input is mapped to the output layer using the weight matrix D0, and then converted to a probability distribution o through the softmax function. The output probability distribution o is a probability vector, in which each dimension is the probability that the sample belongs to the corresponding category, thereby generating the corresponding classification recognition result.
[0102] The essential feature of the present invention is: first, the stress and vibration information of a single point are feature extracted and fused, and then the stress and vibration fusion information of the three points are fused again between channels, wherein each layer of the fusion network uses an LSTM network. After all the tactile information is fused, it is input into the gated memory network and the network output layer to obtain the final output result of the network model. If each channel of stress and vibration is regarded as a mode, the model used in the present invention is different from early fusion, because it is not a simple fusion of the input data characteristics, but feature extraction within and between modes; at the same time, it is also different from late fusion, but each specific mode is explicitly modeled to better and more stably extract cross-modal information between modes. The network model adopted by the present invention is characterized by the use of a multi-layer LSTM network for modal fusion, and the main architecture of the network model is as follows: Figure 5 .
[0103] Step 4: Verify the performance of the solution
[0104] According to step 1 and step 2, data of twelve object surfaces with three types of roughness and four types of hardness are collected and processed, and a complete data set is prepared. After being divided into a training set and a validation set, it is input into the network model described in step 3 for training and validation. The classification accuracy on the training set is about 87.9%. The classification performance of the model is evaluated using the precision p, recall r, and F1 score on the validation set. The calculation formula is as follows:
[0105]
[0106] Among them, true positive (TP), false positive (FP), and false negative (FN) represent the correct classification value of the positive label, the misclassification value of the positive label, and the misclassification value of the negative label, respectively, and the F1 score takes the above two values into consideration. The results are shown in Figure 6 , it can be seen that the performance of this solution is good.
Claims
1. A three-point tactile fusion optical fiber sensing system based on multi-layer LSTM, comprising the following steps: The first step is to collect tactile data Design a tactile sensing model, build a tactile information sensing optical path based on fiber grating, and realize synchronous three-point sensing of the object surface; transmit the collected tactile signals to the host computer; use the robotic arm to drive the tactile sensing model to collect tactile data; Step 2: Build a tactile dataset The object to be tested is a silicone block with different surface conditions, and its surface tactile information includes hardness and roughness. The multi-touch point tactile data collected in the first step is demodulated and processed, and static stress information and dynamic vibration information are obtained at the same time. The hardness is reflected in the static stress information, and the roughness is reflected in the dynamic vibration information. The obtained static stress information and dynamic vibration information are spliced, aligned, and normalized to form a three-point tactile time series data set for input into the network model for recognition. The third step is to build and train a tactile information recognition network model The data in the processed three-point tactile time series data set are input into the multi-layer LSTM network model for fusion processing: the static stress and dynamic vibration of each point are regarded as a mode respectively, and feature extraction is performed within and between modes. At the same time, the corresponding timing characteristics of each mode and between modes are grasped, and the stress and vibration information of a single point are extracted and fused; the stress and vibration fusion information of each point are fused again between channels, and each layer of the fusion network uses an LSTM network; after all tactile information is fused, it is input into the gated memory network and the network output layer to obtain the final output result of the network model.
2. The three-point tactile fusion optical fiber sensing system according to claim 1, characterized in that: The first step includes the following steps: the bottom of the tactile sensing model encapsulated with the fiber Bragg grating is in direct contact with the surface of the object to be measured, and the data is collected by the robot arm, and the robot arm is controlled to move, and three contact point data are collected according to the cycle process of moving, pressing down, sliding, and lifting. After the tactile sensing model comes into contact with the surface of the object, an optical signal is collected. When the bottom of the tactile sensing model comes into contact with the surface of the object, the sensed pressure causes the fiber Bragg grating to undergo tangential deformation and causes the central wavelength of the fiber Bragg grating to change. The change information is modulated in the interference signal of each scanning frame in the form of signal envelope offset, and the static stress information can be obtained by demodulation; When the tactile sensing model slides on the surface of an object, the vibration information sensed by the contact surface is transmitted to the fiber Bragg grating and coupled in the interference signal of each scanning frame in the form of instantaneous phase change. The dynamic vibration information can be obtained by demodulation. Data is collected on different object surfaces and repeatedly collected on the same object surface.
3. The three-point tactile fusion optical fiber sensing system according to claim 1, characterized in that: The second step includes the following steps: for static stress information, the envelope offset calculated for each frame is a scalar, and the envelope offsets of consecutive frames are spliced into a 1×T tensor, where T is the sequence length of the data sample, as a feature vector sequence representing the static stress information; for dynamic vibration information, the vibration waveform demodulated from each frame is taken as a high-dimensional vector, which is downsampled to S dimensions, and the vibration waveforms of consecutive frames are spliced into an S×T tensor as a feature vector sequence representing the dynamic vibration information. The two types of information are strictly aligned in time with the frame as the timestamp; the information of the three touch points is processed in the same way to obtain the three-point tactile data of a sample, and construct a three-point tactile time series data set.
4. The three-point tactile fusion optical fiber sensing system according to claim 1, characterized in that: The third step includes the following steps: (1) Single-point bimodal feature fusion: For the feature input of each modality, C at time t-1 and x at time t are combined as the LSTM input at time t, which are: Where s represents stress, v represents vibration, and denote the stress and vibration mode input at the tth moment, respectively. and Represent the memory unit information of stress and vibration mode at the t-1th moment respectively. and The outputs of the two modalities are fused and input into the corresponding dual-modal fusion network, as shown below: in The stress and vibration modal fusion information is input into the dual-modal fusion network and encoded by LSTM: is the output of the bimodal fusion network. It is the final memory unit of the LSTM network. The stress and vibration information of the three points are fused in the same way to obtain different outputs of the three points. Then the stress output of each point is Vibration Output Dual-modal fusion output To combine: According to this combined method, we can obtain Represents the final output results of three single points. (2) Three-point feature fusion: The feature fusion information of the three points is combined in pairs and input into the two-point feature fusion layer: The temporal characteristics between channels are captured by the LSTM network, and the output is obtained: Where k∈N2, N2={(1,2),(1,3),(2,3)}, represents the two sequences currently being fused. is the final output of the LSTM network, It is the final memory unit of the LSTM network. Finally, the two-point fusion output result and the single-point output result are combined as the input of the three-point feature fusion layer: Where n = {1,2,3}, k = {(1,2),(1,3),(2,3)}. The output is: Finally, the output results of all single points, two points, and three points are integrated as the final output of the multi-layer LSTM fusion network to enhance the feature performance of modal fusion: Where n = {1,2,3}, k = {(1,2),(1,3),(2,3)}. (3) Gated memory network: Output of multi-layer LSTM fusion network O t After three neural networks, we get g1, g2, Among them, g1 and g2 are two gate control units, which are controlled by neural networks D1 and D2 respectively. is the cross-modal update unit, u t It is the output of the gated memory network at time t. The gated memory network can fuse the feature information of the previous moment and the current moment. (4) Output layer: The input of the output layer is the final state u of the gated memory network t The output of stress and vibration characteristics in each sensing point is cascaded together to Represents vector concatenation: The cascaded input is mapped to the output layer using the weight matrix D0, and then converted to a probability distribution o through the softmax function. The output probability distribution o is a probability vector, in which each dimension is the probability that the sample belongs to the corresponding category, thereby generating the corresponding classification recognition result.
Citation Information
Patent Citations
Visual and tactile multi-mode object recognition method based on deep learning
CN117435973A
Flexible fingertip tactile-slip sensor based on fiber bragg grating and detection method of flexible fingertip tactile-slip sensor
CN116046031A
Perimeter security system based on optical fiber fine identification
CN116486543A
MNF flexible tactile sensing system simulating fingertip skin microstructure and manufacturing method
CN117451228A
Multi-mode optical fiber sensing system based on attention mechanism
CN117520791A
