Three-point tactile fusion optical fiber sensing system based on multi-layer lstm

Abstract

The application relates to a three-point tactile fusion optical fiber sensing system based on a multi-layer LSTM, comprising the following steps: collecting tactile data; constructing a tactile data set, splicing, aligning and normalizing static stress information and dynamic vibration information obtained, so as to form a three-point tactile time sequence data set for inputting a network model for identification; establishing and training a tactile information identification network model; inputting data in the processed three-point tactile time sequence data set into a multi-layer LSTM network model for fusion processing, regarding static stress and dynamic vibration of each point as a kind of mode respectively, performing feature extraction within the mode and between the modes, simultaneously grasping mutual corresponding time sequence characteristics of each mode and between the modes, and performing feature extraction and fusion on the two kinds of information of stress and vibration of a single point; after all the tactile information is fused, being inputted into a gate memory network and a network output layer, so as to obtain a final output result of the network model.

Description

Technical Field

[0001] This invention belongs to the field of fiber optic sensing, and is a three-point tactile fusion fiber optic sensing system based on multilayer LSTM. Background Technology

[0002] With the continuous development of the times and the continuous growth of the economy, intelligent mechanization has become a new era development strategy driven by the inherent needs of industries such as manufacturing and healthcare. Its development involves the exploration of sensors, further processing of sensor-collected data, and research. Touch is a basic function for humans to perceive their surroundings and an important means for sensors to mimic human perception of objects. Developing the tactile perception function of sensors can enable sensors to better detect and identify objects, helping people complete more complex tasks.

[0003] For tactile sensing, information such as hardness and temperature can be sensed upon contact with an object, while more subtle information such as roughness and texture requires relative sliding with the object to be perceived. While current tactile sensing technologies such as resistance and piezoelectricity can achieve some object recognition, they have limitations in several aspects: it is difficult to simultaneously acquire multiple tactile information about an object; multi-point tactile sensing similar to human skin is difficult to implement practically; there is insufficient progress in algorithms for fusing and recognizing multiple tactile features; and traditional sensors are complex to package, have many wires, and are heavy, all of which contribute to recognition difficulties. Summary of the Invention

[0004] This invention proposes a three-point tactile fusion fiber optic sensing system based on a multilayer LSTM. The tactile sensor used includes the system's optical path and fiber Bragg grating components. Tactile information is acquired by the fiber Bragg grating sensor, which collects vibration and stress signals (vibration signals identify roughness, and stress signals identify hardness). A network model capable of fusing multi-point stress and vibration information is trained to simulate human multi-point tactile perception, acquire rich tactile information, and perform object surface recognition. The solution provided by this invention is as follows.

[0005] A three-point tactile fusion fiber optic sensing system based on multilayer LSTM includes the following steps:

[0006] The first step is to collect tactile data.

[0007] Design a tactile sensing model, build a tactile information sensing optical path based on fiber optic grating to realize synchronous three-point sensing on the surface of an object; transmit the collected tactile signals to the host computer; and use a robotic arm to drive the tactile sensing model to collect tactile data.

[0008] The second step is to construct a tactile dataset.

[0009] The test object is a silicone block with different surface conditions. Its surface tactile information includes hardness and roughness. The multi-touch tactile data collected in the first step is demodulated, and static stress information and dynamic vibration information are obtained at the same time. Hardness is reflected in the static stress information, and roughness is reflected in the dynamic vibration information. The obtained static stress information and dynamic vibration information are spliced, aligned and normalized to form a three-point tactile time series dataset, which is then input into the network model for recognition.

[0010] The third step is to establish and train a tactile information recognition network model.

[0011] The processed three-point tactile time series dataset is input into a multi-layer LSTM network model for fusion processing: the static stress and dynamic vibration of each point are treated as a mode, and features are extracted within and between modes. At the same time, the temporal characteristics of each mode and the inter-mode correspondence are grasped, and the stress and vibration information of a single point are extracted and fused. The stress and vibration fused information of each of the three points is then fused between channels, with each fusion network using an LSTM network. After all the tactile information is fused, it is input into a gated memory network and the network output layer to obtain the final output result of the network model.

[0012] Further, the first step includes the following steps: The bottom of the tactile sensing model encapsulated with a fiber Bragg grating is in direct contact with the surface of the object under test. Data is collected by a robotic arm controlled to move the arm, collecting data at three contact points in a cyclical process of moving, pressing, sliding, and lifting. After the tactile sensing model contacts the object surface, it collects optical signals. When the bottom of the tactile sensing model contacts the object surface, the sensed pressure causes tangential deformation of the fiber Bragg grating, resulting in a change in the center wavelength of the fiber Bragg grating. This change is modulated into the interference signal of each scan frame in the form of a signal envelope shift. Demodulation yields the static stress information. When the tactile sensing model slides and rubs against the object surface, the vibration information sensed by the contact surface is transmitted to the fiber Bragg grating and coupled into the interference signal of each scan frame in the form of an instantaneous phase change. Demodulation yields the dynamic vibration information. Data is collected from different object surfaces, and repeated for the same object surface.

[0013] Furthermore, the second step includes the following steps: For static stress information, the envelope offset calculated for each frame is a scalar. The envelope offsets of consecutive frames are concatenated into a 1×T tensor, where T is the sequence length of the data samples, serving as a feature vector sequence representing the static stress information. For dynamic vibration information, the demodulated vibration waveform of each frame is used as a high-dimensional vector. After downsampling it to S dimensions, the vibration waveforms of consecutive frames are concatenated into an S×T tensor, serving as a feature vector sequence representing the dynamic vibration information. Both types of information are strictly aligned in time using frames as timestamps. The information from the three touch points is processed in the same way to obtain the three-point tactile data of a sample, constructing a three-point tactile time series dataset.

[0014] Furthermore, the first step includes the following steps:

[0015] (1) Single-point dual-modal feature fusion: For the feature input of each modality, the C at time t-1 and the x at time t are combined as the LSTM input at time t, as follows:

[0016]

[0017] Where s represents stress and v represents vibration. and Let these represent the stress and vibration mode inputs at time t, respectively. and These represent the memory cell information for stress and vibration modes at time t-1, respectively. and The outputs of the two modalities are fused and input into the corresponding bimodal fusion network, as shown below:

[0018]

[0019] in The stress and vibration modal fusion information is input into the dual-modal fusion network and encoded using LSTM:

[0020]

[0021] This is the output of the dual-modal fusion network. This is the final memory unit of the LSTM network. The stress and vibration information at the three points are fused using the same method to obtain different outputs at the three points. Then stress output at each point Vibration output Dual-modal fusion output Combine:

[0022]

[0023] According to this combination method, the following were obtained respectively: This represents the final output result of three single points.

[0024] (2) Three-point feature fusion: The feature fusion information of the three points is combined in pairs and input into the two-point feature fusion layer:

[0025]

[0026] The output is obtained by capturing the time characteristics between channels using an LSTM network:

[0027]

[0028] Where k∈N2, N2={(1,2),(1,3),(2,3)}, represents the two sequences currently being merged. It is the final output of the LSTM network. This is the final memory unit of the LSTM network. Finally, the two-point fusion output and the single-point output are combined as the input to the three-point feature fusion layer:

[0029]

[0030] Where n = {1, 2, 3}, k = {(1, 2), (1, 3), (2, 3)}. The output is:

[0031]

[0032] Finally, all single-point, two-point, and three-point outputs are integrated as the final output of the multi-layer LSTM fusion network to enhance the feature representation of modality fusion.

[0033]

[0034] Where n = {1, 2, 3}, k = {(1, 2), (1, 3), (2, 3)}.

[0035] (3) Gated memory network: Output O of multi-layer LSTM fusion network t After passing through three neural networks, we obtain g1, g2, and g3. g1 and g2 are two gating units, controlled by neural networks D1 and D2 respectively. It is a cross-modal update unit, u t It is the output of the gated memory network at time t. The gated memory network can integrate the feature information of the previous time step and the current time step.

[0036] (4) Output layer: The input to the output layer is the final state u of the gated memory network. tCascaded with the outputs of stress and vibration characteristics at each sensing point, using Represents vector concatenation:

[0037]

[0038] The cascaded input is mapped to the output layer using a weight matrix D0, and then transformed into a probability distribution o using a softmax function. The output probability distribution o is a probability vector, where each dimension represents the probability that a sample belongs to the corresponding category, thus producing the corresponding classification and recognition result.

[0039] The advantages of this invention compared to existing technologies are as follows:

[0040] 1) Compared with traditional electrical tactile sensors, the fiber optic grating tactile sensor used in this invention has the characteristics of anti-electromagnetic interference, small size, light weight, and high sensitivity. It can simultaneously detect static stress and dynamic vibration, has higher recognition efficiency, and is more applicable.

[0041] 2) Compared to single-point tactile sensors, the three-point tactile sensor designed and used in this invention can simultaneously perceive an object from three points, and can be further extended to multi-point perception, mimicking the multi-point tactile perception function of humans, and is closer to an anthropomorphic target.

[0042] 3) Compared with traditional time series modality fusion algorithms, the multi-layer LSTM fusion network model used in this invention can grasp the feature information of each modality and the interaction feature information between modalities more deeply, thereby stably extracting and fully fusing multi-point bimodal tactile information. Attached Figure Description

[0043] Figure 1 This is a design drawing of the sensor bracket used in this invention.

[0044] Figure 2 This invention includes a fiber optic grating packaging diagram and a complete sensing model diagram.

[0045] Figure 3 This is the optical path diagram for data acquisition in this invention.

[0046] Figure 4 This is a diagram showing the sample data processing results of the present invention.

[0047] Figure 5 This is the network model diagram used in this invention.

[0048] Figure 6 This is a verification result diagram of the present invention.

[0049] Figure 7 This is the system flowchart of the present invention.

[0050] Explanation of icon numbers:

[0051] 4-Sensor bracket body; 5-Rectangular protrusion installed at the end of the robotic arm; 6-Bracket cap; 11-Branch tube body; 12-Rectangular protrusion used to control the downward pressure distance; 13-Branch sensor front end. Detailed Implementation

[0052] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and formulas, and will be implemented in specific steps:

[0053] Step 1: Collect three-point tactile data

[0054] Design of a sensing model for contacting objects: This invention uses Figure 2 The complete sensing model shown is used for data acquisition. First, design... Figure 1 The three-head sensor bracket shown consists of three identical movable branches (each controlled by an independent spring to raise and lower itself, with no interference between their movements). When sliding on an object's surface, it allows for simultaneous three-point sensing, providing richer tactile information compared to single-point contact. Labels 11, 12, and 13 describe the branch design individually: 11 is the branch tube, 12 controls the downward pressure distance of the branch, and 13 has a built-in spring and a sensor at its bottom. Label 4 is the main body of the sensor bracket. Label 5 shows a rectangular protrusion mounted at the end of a robotic arm, which drives the sensor bracket to slide. Label 6 is the bracket cap, used to fix the three movable branches. The fiber optic grating is then... Figure 2 The sensor is encapsulated in silicone finger pads. Three identical silicone finger pads are made using this encapsulation method and then glued to the recessed areas at the bottom of each branch of the sensor bracket, assembling them into a complete sensor model. Finally, the complete sensor model is fixed to a robotic arm, and the robotic arm's movements are programmed to allow it to move and collect data from the object's surface.

[0055] Constructing the tactile information sensing optical path: Press Figure 3The diagram illustrates a three-point tactile information sensing optical path, comprising a swept-wave light source module, a 2x2 coupler, a fiber grating with a reflection bandwidth of 1 nm, a Faraday rotator mirror, a photodetector, a data acquisition card, and a host computer for data processing. The swept-wave light source module provides the detection light for the sensing system, which enters the three middle couplers via the left coupler. Since the three middle paths are identical, only one path is used for explanation: the detection light is split into two beams by the coupler, entering the fiber grating and the Faraday rotator mirror respectively; the Faraday rotator mirror and the fiber grating form a Michelson interferometer through the coupler, with the fiber grating serving as the probe arm and the Faraday rotator mirror as the reference arm. The two beams interfere within the Michelson interferometer; the interfering light returns to the optical coupler and enters the photodetector; the photodetector converts the optical signal into an analog electrical signal and transmits it to the data acquisition card; the data acquisition card further converts the analog electrical signal into a digital signal and transmits it to the host computer, thus obtaining the original tactile interference signal.

[0056] Tactile data acquisition process: The bottom of the sensing model encapsulated with a fiber Bragg grating is in direct contact with the surface of the object under test, and data is acquired by a robotic arm controlled by the arm. A computer program is used to control the movement of the robotic arm, which acquires data according to a cycle of moving, pressing, sliding, and lifting. During acquisition, the arm moves at a pressing distance of 10 mm and a sliding speed of 160 mm / s. After the sensing model comes into contact with the object surface, it collects optical signals, which are then transmitted to the host computer via the sensing optical path. The specific tactile information obtained is as follows: When the bottom of the sensing model comes into contact with the object surface, the sensed pressure causes tangential deformation of the fiber Bragg grating, resulting in a change in the center wavelength of the fiber Bragg grating. This change is modulated into the interference signal of each scan frame in the form of a signal envelope shift, and demodulation yields the static stress information. When the sensing model slides and rubs against the object surface, the vibration information sensed by the contact surface is transmitted to the fiber Bragg grating and coupled into the interference signal of each scan frame in the form of an instantaneous phase change. Demodulation yields the dynamic vibration information. Data was collected from different object surfaces, and the same object surface was collected 30 times to obtain comprehensive dynamic and static tactile data.

[0057] The parameters set for each module in the above tactile data acquisition process are as follows: the scanning rate of the swept wave light source module is 50 Hz, the optical path difference between the two arms is set to 5 cm, and the acquisition rate of the data acquisition card is 100 KSa / s. Correspondingly, the scanning period for each scan frame is 20 ms, the interference signal wavenumber in each scan frame is approximately 80, and the interference signal length in each scan frame is approximately 2000 sampling points. The highest frequency for static stress information detection is 25 Hz, and the highest frequency for dynamic vibration information detection is 1 kHz.

[0058] Step 2: Construct a three-point tactile dataset

[0059] Tactile data preprocessing: The three signals were collected simultaneously in the previous step, so the three channels of information obtained in this step can be processed synchronously. The following describes the data processing process of one channel. The processing method refers to patent 202311316484.6. In step one, the wavelength of the probe light output by the sweeping light source module is linearly related to time and performs wavelength scanning with a fixed repetition period. Within one scanning period, when the wavelength of the sweeping light source reaches the reflection wavelength of the fiber optic grating, the probe light and the reference light interfere on the coupler. After one scanning period, an interference signal of one scanning frame is formed and transmitted as an electrical signal through the photodetector. Finally, the interference signal g(λ) is collected by the data acquisition card and expressed as formula (1):

[0060]

[0061] Where R(λ) is the reflection spectrum of the fiber grating, reflecting the outer envelope characteristics of the interference signal and containing strain and temperature information. n is the effective refractive index of the fiber, and L is the optical path difference of the grating. This represents the phase modulation of the coherent signal caused by vibration during light transmission. Therefore, R(λ) contains the required static stress information. It contains the required dynamic vibration information.

[0062] The acquired interference signal g(λ) needs to undergo signal demodulation processing. When the fiber grating senses strain, its reflection spectrum changes, causing a shift in the envelope of the interference signal waveform in each scan frame. Therefore, static stress information is reflected in the interference signal envelope shift (time domain). When the fiber grating senses vibration, the instantaneous phase of the interference signal changes. Therefore, dynamic vibration information is reflected in the instantaneous phase change of the interference signal (frequency domain). Since these two types of information are in different domains simultaneously, they can be demodulated synchronously.

[0063] The static stress information is demodulated as follows: First, the interference signal corresponding to the scan frame when the robotic arm and the object are not in contact is used as the reference signal. To prevent noise interference, Gaussian smoothing is used to denoise the acquired interference signal, and the envelope information of each frame is extracted using Hilbert transform. Finally, the offset between the signal envelope extracted from each frame and the reference signal envelope is calculated using the cross-correlation method. This offset is the static stress information in that scan frame.

[0064] The dynamic vibration information is demodulated as follows: First, Gaussian smoothing is performed for noise reduction. Then, the Hilbert transform is applied to the interference signal to extract the envelope, and the extracted envelope is used to normalize the original interference signal. The normalized interference signal is then subjected to synchronous compressed wavelet transform to demodulate the instantaneous frequency. After integration and restoration, the instantaneous phase change is obtained, and its waveform represents the dynamic vibration information in one scan frame.

[0065] Through the demodulation process described above, dynamic and static tactile information at three points can be acquired simultaneously, namely, the static stress and dynamic vibration information of each of the three points. The test object in the experiment is a silicone block with different surface conditions; its surface tactile information includes hardness and roughness. Hardness is reflected in static stress, and roughness is reflected in dynamic vibration.

[0066] Creating a tactile dataset: To input two different types of data into the network model for training, consider processing the obtained data into a feature vector sequence. For static stress information, the envelope offset calculated for each frame is a scalar, and the envelope offsets of consecutive frames need to be concatenated into a 1×T tensor. The sequence length of the data samples is given in this experiment. For static stress information, a 600-dimensional feature vector sequence is used. For dynamic vibration information, the demodulated vibration waveform of each frame is a 600-dimensional vector. To save storage space and improve computational efficiency, it is downsampled to 150 dimensions, and the vibration waveforms of consecutive frames are concatenated into a 150×T tensor, which serves as the feature vector sequence representing dynamic vibration information. Both types of information are strictly aligned in time using frames as timestamps. The information from all three points is processed using the above method to obtain the three-point tactile data of a sample.

[0067] Finally, after processing all samples according to the above procedure, the resulting data is compiled into a three-point tactile time-series dataset, described as D. i ={s 1i ,v 1i ,s 2i ,v 2i ,s 3i ,v 3i ,L i}, where i∈(1,N), represents the i-th sample, and N is the number of samples. k Represents stress modes, v k The symbol represents the vibration mode, k represents the kth channel, and L represents the label representing the hardness or roughness of the object's surface, derived from silicone blocks with different surface conditions used in the experiment. Figure 4 The visualization results of the surface tactile data of one of the samples are shown, with vibration on the left and stress on the right.

[0068] Step 3: Train a three-point dynamic and static tactile fusion network model using the prepared dataset.

[0069] Build as Figure 5The multi-layer LSTM fusion network (MLFN) shown is used for data processing. If the static stress and dynamic vibration of each point are regarded as a mode, the network model can extract features within and between modes, and grasp the temporal characteristics of each mode and the mutual correspondence between modes. It mainly includes four parts: (1) single-point dual-modal feature fusion, which includes two layers of LSTM; (2) three-point feature fusion, which includes two layers of LSTM; (3) gated memory network, a multi-modal joint temporal storage structure, whose storage changes with time and performs cross-modal interaction; (4) output layer, which obtains the recognition result based on the fusion information. The input data is a three-point tactile time series dataset D. i ={S 1i ,v 1i ,S 2i ,v 2i ,s 3i ,v 3i ,L i First, process the dual-modal information fusion of a single point, then process the information fusion of three points.

[0070] (1) Single-point dual-modal feature fusion: For the feature input of each modality, the C at time t-1 and the x at time t are combined as the LSTM input at time t, as follows:

[0071]

[0072] Where s represents stress and v represents vibration. and Let these represent the stress and vibration mode inputs at time t, respectively. and These represent the memory cell information for stress and vibration modes at time t-1, respectively. and The input is fed into a single-point dual-modal feature fusion network, taking stress as an example, as shown below:

[0073]

[0074] Wherein, the superscript s(v) indicates that the current parameter is a stress (vibration) mode. Represents the corresponding network weights. This indicates the network bias, with σ set to the sigmoid activation function. This represents the input gate at time t. This represents the forget gate at time t. This represents the output gate at time t. This represents the new memory unit at time t. It is the final memory unit at time t. This represents the final output value at time t. Stress and vibration are both input into the LSTM network in this manner to obtain intramodal feature information. The outputs of the two modes are then fused and input into the corresponding dual-modal fusion network, as shown below:

[0075]

[0076] in The stress and vibration modal fusion information is input into the dual-modal fusion network and encoded using LSTM:

[0077]

[0078] This is the output of the dual-modal fusion network. This is the final memory unit of the LSTM network. The stress and vibration information at the three points are fused using the above process to obtain different outputs at the three points. Then stress output at each point Vibration output Dual-modal fusion output Combine:

[0079]

[0080] According to this combination method, the following were obtained respectively: This represents the final output result of three single points.

[0081] (2) Three-point feature fusion: First, the feature fusion information of the three points is combined in pairs and then input into the two-point feature fusion layer:

[0082]

[0083] The output is obtained by capturing the time characteristics between channels using an LSTM network:

[0084]

[0085] Where k∈N2, N2={(1,2),(1,3),(2,3)}, represents the two sequences currently being merged. It is the final output of the LSTM network. This is the final memory unit of the LSTM network. Finally, the two-point fusion output and the single-point output are combined as the input to the three-point feature fusion layer:

[0086]

[0087] Where n = {1, 2, 3}, k = {(1, 2), (1, 3), (2, 3)}. The output is:

[0088]

[0089] Finally, all single-point, two-point, and three-point outputs are integrated as the final output of feature fusion, enhancing the feature representation of modality fusion.

[0090]

[0091] Where n = {1, 2, 3}, k = {(1, 2), (1, 3), (2, 3)}.

[0092] (3) Gated memory network: Composed of multiple neural networks, it is a unified memory, and its specific components are as follows:

[0093] g1=σ(D1(O t (20)

[0094] g2=σ(D2(O t )) (twenty one)

[0095]

[0096] Output O of multi-layer LSTM fusion network t After passing through three neural networks, we obtain g1, g2, and g3. g1 and g2 are two gating units, controlled by neural networks D1 and D2 respectively. It is a cross-modal update unit, u t This is the output of the gated memory network at time t. The gated memory network can fuse feature information from the previous time step and the current time step.

[0097] (4) Output layer: The input to the output layer is the final state u of the gated memory network. t Cascaded with the outputs of stress and vibration characteristics at each sensing point, using Represents vector concatenation:

[0098]

[0099] The cascaded input is mapped to the output layer using a weight matrix D0, and then transformed into a probability distribution o using a softmax function. The output probability distribution o is a probability vector, where each dimension represents the probability that a sample belongs to the corresponding category, thus producing the corresponding classification and recognition result.

[0100] The essential features of this invention are: firstly, feature extraction and fusion of stress and vibration information at a single point are performed; then, the stress and vibration fused information of each of the three points is further fused across channels, with each fusion network using an LSTM network. After all tactile information is fused, it is input into a gated memory network and the network output layer to obtain the final output of the network model. If each stress and vibration path is considered as a modality, the model used in this invention differs from earlier fusion methods because it is not a simple fusion of input data characteristics, but rather feature extraction within and between modalities; it also differs from later fusion methods, as it explicitly models each specific modality, extracting cross-modal information better and more stably between modalities. The network model used in this invention is characterized by the use of a multi-layer LSTM network for modal fusion, and the main architecture of the network model is as follows: Figure 5 .

[0101] Step 4: Verify the performance of the solution.

[0102] Data was collected and processed for twelve object surfaces with three roughnesses and four hardnesses according to steps one and two, and a complete dataset was created. This dataset was then divided into training and validation sets and input into the network model described in step three for training and validation. The classification accuracy on the training set was approximately 87.9%. On the validation set, the model's classification performance was evaluated using precision p, recall r, and F1 score, calculated using the following formulas:

[0103]

[0104] True positive (TP), false positive (FP), and false negative (FN) represent the correct classification value for the positive label, the incorrect classification value for the positive label, and the incorrect classification value for the negative label, respectively. The F1 score takes both of these values ​​into account. The results are shown in [link to results]. Figure 6 This indicates that the solution performs well.

Claims

1. A three-point tactile fusion fiber optic sensing system based on multilayer LSTM, comprising the following steps: The first step is to collect tactile data. Design a tactile sensing model, build a tactile information sensing optical path based on fiber optic grating to realize synchronous three-point sensing on the surface of an object; transmit the collected tactile signals to the host computer; and use a robotic arm to drive the tactile sensing model to collect tactile data. The second step is to construct a tactile dataset. The test object is a silicone block with different surface conditions. Its surface tactile information includes hardness and roughness. The multi-touch tactile data collected in the first step is demodulated, and static stress information and dynamic vibration information are obtained at the same time. Hardness is reflected in the static stress information, and roughness is reflected in the dynamic vibration information. The obtained static stress information and dynamic vibration information are spliced, aligned and normalized to form a three-point tactile time series dataset, which is then input into the network model for recognition. The third step is to establish and train a tactile information recognition network model. The processed three-point tactile time series dataset is input into a multi-layer LSTM network model for fusion processing: the static stress and dynamic vibration of each point are treated as a mode, and features are extracted within and between modes. At the same time, the temporal characteristics of each mode and the inter-mode correspondence are grasped, and the stress and vibration information of a single point are extracted and fused. The stress and vibration fused information of each of the three points is then fused between channels, with each fusion network using an LSTM network. After all the tactile information is fused, it is input into a gated memory network and the network output layer to obtain the final output result of the network model.

2. The three-point tactile fusion fiber optic sensing system according to claim 1, characterized in that, The first step includes the following steps: The bottom of the tactile sensing model encapsulated with fiber Bragg gratings is in direct contact with the surface of the object to be tested. Data is collected by a robotic arm. The robotic arm is controlled to move and collect data at three contact points in a cycle of moving, pressing down, sliding, and lifting. After the tactile sensing model comes into contact with the object surface, light signals are collected. When the bottom of the tactile sensing model comes into contact with the object surface, the sensed pressure causes the fiber Bragg grating to undergo tangential deformation and causes a change in the center wavelength of the fiber Bragg grating. The change information is modulated into the interference signal of each scanning frame in the form of signal envelope offset. The static stress information can be obtained by demodulation. When the tactile sensing model slides and rubs on the surface of an object, the vibration information sensed by the contact surface is transmitted to the fiber optic grating and coupled into the interference signal of each scan frame in the form of instantaneous phase change. The dynamic vibration information can be obtained by demodulation. Data is collected from different object surfaces and the same object surface is collected repeatedly.

3. The three-point tactile fusion fiber optic sensing system according to claim 1, characterized in that, The second step includes the following steps: For static stress information, the envelope offset calculated for each frame is a scalar. The envelope offsets of consecutive frames are concatenated into a 1×T tensor, where T is the sequence length of the data sample, which serves as the feature vector sequence representing the static stress information. For dynamic vibration information, the demodulated vibration waveform of each frame is used as a high-dimensional vector. After downsampling it to S dimensions, the vibration waveforms of consecutive frames are concatenated into an S×T tensor, which serves as the feature vector sequence representing the dynamic vibration information. The two types of information are strictly aligned in time with the frame as the timestamp. The information of the three touch points is processed in the same way to obtain the three-point tactile data of a sample, and a three-point tactile time series dataset is constructed.

4. The three-point tactile fusion fiber optic sensing system according to claim 1, characterized in that, The third step includes the following steps: (1) Single-point dual-modal feature fusion: For the feature input of each modality, the C at time t-1 and the x at time t are combined as the LSTM input at time t, as follows: Where s represents stress and v represents vibration. and Let these represent the stress and vibration mode inputs at time t, respectively. and These represent the memory cell information of stress and vibration mode at time t-1, respectively; and The outputs of the two modalities are fused and input into the corresponding bimodal fusion network, as shown below: in The stress and vibration modal fusion information is input into the dual-modal fusion network and encoded using LSTM: This is the output of the dual-modal fusion network. It is the final memory unit of the LSTM network; the stress and vibration information at the three points are fused in the same way to obtain different outputs at the three points. Then stress output at each point Vibration output Dual-modal fusion output Combine: According to this combination method, the following were obtained respectively: This represents the final output result of three single points; (2) Three-point feature fusion: The feature fusion information of the three points is combined in pairs and input into the two-point feature fusion layer: The output is obtained by capturing the time characteristics between channels using an LSTM network: Where k∈N2, N2={(1,2),(1,3),(2,3)}, represents the two sequences currently being merged. It is the final output of the LSTM network. This is the final memory unit of the LSTM network; finally, the two-point fusion output and the single-point output are combined as the input to the three-point feature fusion layer: Where n = {1, 2, 3}, k = {(1, 2), (1, 3), (2, 3)}; the output is: Finally, all single-point, two-point, and three-point outputs are integrated as the final output of the multi-layer LSTM fusion network to enhance the feature representation of modality fusion. Where n = {1, 2, 3}, k = {(1, 2), (1, 3), (2, 3)}; (3) Gated memory network: Output O of multi-layer LSTM fusion network t After passing through three neural networks, we obtain g1, g2, and g3. g1 and g2 are two gating units, controlled by neural networks D1 and D2 respectively. It is a cross-modal update unit, u t It is the output of the gated memory network at time t. The gated memory network can fuse feature information from the previous time step and the current time step. (4) Output layer: The input to the output layer is the final state u of the gated memory network. t The outputs of stress and vibration characteristics at each sensing point are cascaded together, and ⊕ represents vector cascading: The weight matrix D0 is used to map the cascaded input to the output layer, and then the softmax function is used to convert it into a probability distribution o. The output probability distribution o is a probability vector, where each dimension is the probability that the sample belongs to the corresponding category, thus producing the corresponding classification and recognition result.

Images