Multi-channel tactile-slip optical fiber sensing identification method based on multi-mode mutual fusion
By using a multi-channel fiber optic grating sensor and a multi-modal fusion model, the limitations of single-point, single-channel measurement are overcome, enabling high-precision multi-modal information measurement and identification of complex surfaces, thus improving identification accuracy.
Patent Information
- Application Number
- CN202411120471.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-15
AI Technical Summary
Existing tactile fiber optic sensors mostly perform single-point, single-channel measurements, which cannot effectively handle complex surface objects. Furthermore, the multimodal fusion algorithm does not fully integrate features, resulting in a decline in recognition performance.
A multi-channel fiber Bragg grating sensor combined with a multimodal fusion model is used to collect tactile and slip information by encapsulating the fiber Bragg grating at the end of a robotic arm, constructing a multi-channel dataset. Information fusion is performed using a single-channel multimodal mutual attention fusion module and a multi-channel fusion module, including a multimodal decoder and an iterative interaction module, combined with a GRU attention mechanism for object surface recognition.
It achieves high-precision multimodal information measurement and recognition of complex surfaces, improves recognition accuracy, and can accurately identify information such as the hardness, texture, and curvature of object surfaces.
Smart Images

Figure CN118981747A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of flexible electronic technology, and in particular relates to an identification system for obtaining tactile information on the surface of an object through a multi-channel fiber grating sensor and fusing multi-modal information of the tactile information. Background Art
[0002] With the integration of informatization and industrialization, the intelligent industry continues to rise, robot bionic perception technology has been developed, and intelligent robots are widely used in industries, services and other fields. Touch and slip are the most important link between humans and the world, and are also important means for robotic hands to perceive external information. Touch can sense the stress signal when human hands touch objects, and slip can sense the sliding signal when human hands touch objects. Combining stress information and vibration information and applying them to intelligent bionic robots can improve the robot's perception performance, enabling it to complete more complex tasks such as accurate identification and detection of targets.
[0003] The multi-channel touch-slip optical fiber sensing and recognition system based on multi-modal mutual fusion forms an overall touch-slip perception of the surface of the object by fusing the acquired touch-slip multi-modal information. Traditional electrical signal tactile sensors are susceptible to electromagnetic interference. Although they can sense vibration, stress, and deformation, most of them can only accurately measure one physical quantity. The optical fiber sensor used in the present invention can better avoid electromagnetic interference, and also has the advantages of small size, light weight, good stability, and good multiplexing. However, most of the touch-slip optical fiber grating sensors currently used only perform single-point single-channel data measurement, and the corresponding multi-modal fusion algorithm only achieves the fusion of single-point touch-slip information data. When facing some objects with complex surfaces, such as objects with different shapes or surfaces with different curvatures, the existing sensor recognition system has certain limitations. In addition, the network structure used for the multi-modal fusion algorithm often only integrates part of the features of a certain mode into another mode, which leads to insufficient fusion of multi-modal features. As the network depth increases, the recognition performance of the sensor system decreases. Summary of the invention
[0004] The present invention proposes a multi-channel touch-slip optical fiber sensing system based on multi-modal mutual fusion. The data acquisition part includes a touch-slip sensor optical path and a fiber grating package. The data set construction part demodulates the multi-channel touch-slip information collected by the fiber grating sensor to obtain vibration signals and stress signals, and constructs a multi-channel touch-slip data set with the results. The multi-channel stress signal mode and vibration signal mode fusion part obtains a multi-channel touch-slip multi-modal mutual fusion model through training to identify the surface of the object. The scheme provided by the present invention is as follows:
[0005] A multi-channel touch-slip optical fiber sensing recognition method based on multi-modal mutual fusion includes the following steps:
[0006] The first step is to collect multimodal information of touch and slip
[0007] Build a multi-channel fiber optic sensing system based on fiber Bragg grating (FBG) for touch and slip information. Encapsulate the fiber Bragg grating sensors into different silicone sleeves. The silicone sleeves are placed at the end of the robotic arm to collect multi-modal information of touch and slip.
[0008] The second step is to build a multi-channel touch and slip dataset
[0009] A robotic arm is used to contact silicone blocks with different hardness, texture, shape, and curvature to collect multi-channel interference signals, where the central wavelength offset of each interference signal is extracted as stress information, and the instantaneous frequency of the interference signal is demodulated to obtain vibration information. The multi-channel stress information, vibration information, and labels form a multi-channel touch and slip data set; the touch and slip information of each channel has different manifestations at the corresponding position, and the data of each channel are combined to obtain the hardness, texture, shape, and curvature information of the object surface;
[0010] The third step is to build and train a multi-channel tactile multimodal mutual fusion model. The network structure of the multi-channel tactile multimodal mutual fusion model consists of two parts. The first part is a single-channel multimodal mutual attention fusion module, and the second part is a multi-channel fusion module.
[0011] (1) The single-channel multimodal mutual attention fusion module performs preliminary fusion on the single-channel touch and slip information to obtain fusion features. The module includes an N-layer multimodal mutual decoder module and N-1 iterative multimodal interaction modules. The multimodal mutual decoder module fuses the vibration information and the stress information, and introduces the iterative multimodal interaction module to enhance the stress information. The structure of each layer of the multimodal mutual decoder module is the same, and the output of each layer is the stress information feature fused with the vibration information feature. The output result of each layer is input into the iterative multimodal interaction module to obtain the fusion feature of the vibration information and the stress information of the layer. After iteration, the fusion feature of the vibration information and the stress information is finally obtained.
[0012] (2) The multi-channel fusion module, based on the GRU attention mechanism, further fuses the fusion features of each channel to form a multi-channel touch-slip multimodal fusion feature, which is normalized to achieve the recognition of object surface information.
[0013] Furthermore, the single-channel multimodal mutual attention fusion module includes:
[0014] The stress information and vibration information are input into the multimodal mutual decoder module; the Transformer encoder is used to capture the dependencies between different positions in the vibration information data, namely the deep vibration feature F D; The input of the first layer of the multimodal mutual decoder module is stress information and deep vibration features F D , stress information generates stress information features F through multi-head self-attention S , the stress information features are projected onto the bond through the linear layer Sum Similarly, the deep vibration feature F is transformed into D Projection to Key Sum Then the interactive attention matrix is generated and normalized to obtain the vibration information features of the fused strain mode. and the stress information characteristics of the fusion vibration mode
[0015] Will Each element in is considered as a weight of the linear layer and applied to On top, we get the initial fusion features
[0016] Will Perform linear mapping and then project it to the key of multi-head cross attention With value F S Projected to the weight of multi-head cross attention d k for The dimension of the multimodal mutual decoder module is obtained by outputting the first layer of the multimodal mutual decoder module, that is, the stress information feature of the first layer fused with the vibration information feature
[0017] The stress information features of the first layer are fused with the vibration information features and stress information feature F S Input iterative multimodal interaction module, through the linear layer mapping F S , generate the stress characteristics of the current layer Then use the linear layer to project the multimodal fusion features of touch and slip First, we map F with a linear layer 1 S Generate the current layer output of the iterative multimodal interaction module Next, the linear layer 3 is used to project the multimodal fusion features of touch and slip to obtain Calculate the attention matrix of the reorganized stress feature and normalize it to obtain
[0018]
[0019] Then through and mapped with a linear layer 2 Generated Obtaining new stress characteristics
[0020]
[0021] Will Inject into middle:
[0022]
[0023] in, is the learnable weight. As each layer is iterated, BN stands for batch normalization and the output is is the fusion feature of the first layer of vibration information and stress information, which is combined with the deep vibration feature F D Together they serve as the input to the second layer of the multimodal mutual decoder module; Generate stress information feature F through multi-head self-attention dec1 , through the linear layer to transform F dec1 Projection to Key Sum Similarly, project the vibration information characteristics to the bond Sum Generate an interactive attention matrix; normalize the interactive attention matrix to obtain the vibration information characteristics of the fused strain mode and multiple fusion features of fused vibration modes
[0024] Will Each element in is considered as a weight of the linear layer and applied to Above, that is, the initial fusion feature Will Perform linear mapping and then project it to the key of multi-head cross attention With value F S Projected to the weight of multi-head cross attention d k for The dimension of the second layer is obtained by multimodal mutual decoder module, that is, the stress feature of the second layer fused with vibration feature
[0025] The stress features of the second layer are fused with the vibration features and the first layer Input iterative multimodal interaction module; first use linear layer 1 to map Generate the current layer output of the iterative multimodal interaction module Next, the linear layer 3 is used to project the multimodal fusion features of touch and slip to obtain Calculate the attention matrix of the reorganized stress feature and normalize it to obtain Then through and mapped with a linear layer 2 Generated Obtaining new stress characteristics
[0026] Will Inject into In the process, the fusion features of the vibration information and stress information of the second layer are obtained.
[0027] Repeat the above steps to iterate and obtain the fusion features of the vibration information and stress information of the N-1th layer.
[0028] Input it into the last layer of the single-channel multimodal mutual decoder module to obtain the final fusion features of vibration information and stress information
[0029] Furthermore, the multi-channel fusion module includes:
[0030] 1) The fusion features of the vibration information and stress information obtained from each channel are finally concatenated into a multi-channel fusion feature sequence F;
[0031] 2) Input the sequence F into the multi-channel fusion module to obtain the output and hidden state of each dimension through GRU.
[0032] 3) Extract vibration information and stress information fusion feature v through attention mechanism:
[0033] 4) Perform normalization calculation to obtain the attention weight matrix:
[0034] A = softmax(v)
[0035] The label corresponding to the feature vector is determined according to the attention weight matrix to achieve overall recognition of the hardness, texture, and curvature information of the object surface.
[0036] Furthermore, assuming that the number of channels is 3, the multi-channel fusion module includes:
[0037] The fusion features of the three channels Splicing to generate sequence Input the sequence F into the multi-channel fusion module to obtain the output h 1 ,h 2 and h 3 : The first element of sequence F Get the output and hidden state h through GRU 1 ; The second element of sequence F Get the output and hidden state h through GRU 2 ; The third element of sequence F Get the output and hidden state h through GRU 3 ;
[0038] Extract vibration information and stress information fusion feature v through attention mechanism:
[0039] e i =w i tanh(W i h i +b i ),1≤i≤3
[0040]
[0041] Among them, e i Indicates the relevant information h corresponding to the i-th eigenvector i The determined attention probability distribution value, w i and W i represents the weight coefficient matrix of the i-th fusion feature vector, b i Represents the offset corresponding to the i-th fusion feature.
[0042] Compared with the prior art, the present invention has the following advantages:
[0043] (1) Compared with traditional touch and slip sensors based on resistance, voltage, and piezoelectricity, the present invention adopts a fiber Bragg grating sensor with high anti-electromagnetic interference capability, which can achieve high-precision measurement of both vibration and stress modes, and can obtain spatial information such as hardness, texture, shape, and curvature through multi-channel acquisition points located at different positions.
[0044] (2) Compared with the traditional single-mode single-channel object surface recognition algorithm, the model adopted by the present invention integrates the vibration signals and stress signals obtained by the three channels, thereby capturing more comprehensive information about the object surface. At the same time, it has higher recognition accuracy and can achieve more accurate object surface recognition function. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 It is a flow chart of the specific operation of the present invention.
[0046] Figure 2 It is a schematic diagram of the optical path of the touch and slip sensor of the present invention.
[0047] Figure 3 It is a schematic diagram of optical fiber packaging of the present invention.
[0048] Figure 4 It is a schematic diagram of the process of constructing a multi-channel touch and slide data set according to the present invention.
[0049] Figure 5 This is a visualization example of multi-channel tactile information obtained by the present invention.
[0050] Figure 6 It is a schematic diagram of the structure of the single-channel multi-modal mutual attention fusion module adopted in the present invention.
[0051] Figure 7 It is a schematic diagram of the structure of the multi-channel fusion module adopted in the present invention.
[0052] Figure 8 It is the result of the five-fold cross validation of the multi-channel tactile and multi-modal mutual fusion model adopted by the present invention. DETAILED DESCRIPTION
[0053] The following further illustrates how to use the multi-channel optical fiber sensing system proposed in the present invention to perform object surface recognition and perception in conjunction with the accompanying drawings and specific implementation examples.
[0054] The present invention is achieved through the following technical solutions, the flow chart is as follows Figure 1 As shown, the specific steps of the present invention are:
[0055] The first step is to collect multimodal information of touch and slip
[0056] The present invention uses a fiber grating sensor with strong anti-electromagnetic interference capability as a touch-slip sensor to obtain touch-slip information. The system optical path is as follows: Figure 2 As shown, it includes a swept wave light source module 1, a 2×2 coupler 2, a 2×2 coupler 6, a 2×2 coupler 7 and a 2×2 coupler 8, a fiber Bragg grating 3, a fiber Bragg grating 4 and a fiber Bragg grating 5 with a reflection bandwidth of 1 nm, a photodetector 9, a photodetector 10 and a photodetector 11, a Faraday rotator mirror 12, a Faraday rotator mirror 13 and a Faraday rotator mirror 14, a data acquisition card 15, and a host computer data processing 16.
[0057] The fiber Bragg grating, Faraday rotator, sweeping light module and photodetector are connected to the four ports of the 2×2 coupler to form a Michelson interferometer structure, in which the optical path where the fiber Bragg grating is located is the detection arm, and the side of the Faraday rotator is the reference arm. In a scanning cycle, when the wavelength of the sweeping light source is equal to the reflection wavelength of the fiber Bragg grating, interference will occur in the 2×2 coupler, thereby generating the interference signal required for collection. The three Michael interferometer structures are connected in parallel to form a multi-channel optical path. The sweeping light source module is connected to the 2×2 couplers of the three structures as input through a 2×2 coupler. The three photodetectors are used as multi-channel outputs and are connected to the data acquisition card through SMA wires, and then connected to the host computer through the PCIE interface on the data acquisition card.
[0058] The packaging of the fiber grating used in the present invention is as follows Figure 3As shown. The three fiber Bragg gratings are respectively encapsulated in three silicone sleeves, and the three silicone sleeves are placed on three resin rods at the end of the robotic arm. The stress information and vibration information are obtained by controlling the robotic arm to contact the surface of the object to be collected. The downward pressing distance of the robotic arm is 10mm, and the sliding speed is 160mm / s. The object to be collected is a silicone block with different hardness, texture, shape, and curvature. The hardness is Shaw A30 and Shaw A70, the texture includes cement texture and silicone texture, and the curvature includes two types, curvature 1 is 0.1599, and curvature 2 is 0.4777.
[0059] The parameters used by each module of the system optical path are as follows: the scanning rate of the scanning light source is 50Hz, the acquisition rate of the data acquisition card is 100KSa / s, and the optical path difference between the two arms is set to 5cm. Each corresponding scanning frame is 20ms long, each scanning frame contains approximately 80 interference signal wave numbers, and the interference signal length of each scanning frame is approximately 2000 scanning points. According to the Nyquist theorem, the maximum frequency of the system stress information detection is 25Hz, and the maximum frequency of the vibration information detection is 1KHz.
[0060] The second step is to construct a multi-channel touch and slip dataset.
[0061] The multi-channel interference signal is collected by the data acquisition card, and the interference signal of each channel is expressed as:
[0062]
[0063] The envelope of the interference signal is represented by the reflection spectrum R(λ) of the fiber Bragg grating, L is the optical path difference of the fiber Bragg grating, n is the refractive index of the fiber Bragg grating, is the phase change of the interference signal caused by vibration. Stress information can be extracted from R(λ) The vibration information is extracted from the sensor. The interference signal with multi-channel touch and slip data is transmitted to the host computer through the data acquisition card for demodulation.
[0064] The interference signal of the current frame is subjected to wavelet denoising, the interference signal envelope is extracted by spline difference, and then Gaussian smoothing is performed on the envelope. Next, the vibration information and stress information are obtained by performing different processing on the envelope: the offset of the envelope peak in the scanning frame is calculated, and the strain waveform is fitted according to the offset, and the waveform is used as stress information; the 3dB area in the envelope is extracted, and the envelope in the area is subjected to SWT time-frequency analysis, and then the wavelet ridge is extracted, and the vibration waveform is obtained by integral restoration of the instantaneous phase, and the vibration waveform is downsampled to obtain the vibration information. Figure 5As shown, the touch and slip information is visualized. The left side is the three-channel vibration information visualization, and the right side is the three-channel stress information visualization. It can be seen that the touch and slip information of the three channels has different performances at different positions. By integrating these differences, the overall hardness, texture, shape, curvature and other information of the object surface can be obtained.
[0065] Multi-channel stress information, vibration information and labels form a data group Data i =(S 1i , S 2i , S 3i , D 1i , D 2i , D 3i , L i 0, where Data i represents the i-th group of data, S 1i , S 2i , S 3i Respectively represent the stress information data corresponding to each channel, D 1i , D 2i , D 3i Indicates the vibration information data corresponding to each channel, L i The label representing the surface condition of the object is obtained through the characteristics of the object to be sampled.
[0066] The third step is to build and train a multi-channel touch and slip multimodal fusion model.
[0067] The multimodal fusion algorithm currently used can only fuse single-point touch and slip information data, and it cannot directly process multi-channel data. At the same time, the network structure used for the multimodal fusion algorithm can often only integrate partial features of a certain modality into another feature, which leads to insufficient feature fusion. As the network depth increases, the recognition performance of the sensor system decreases.
[0068] The multi-channel touch-slip multi-modal mutual fusion model adopted by the present invention can better improve the above problems, can process multi-channel touch-slip information data, and fully integrate the two-modal data in the fusion process. The network structure of the model includes two parts. The first part is as follows Figure 6 The single-channel multimodal mutual attention fusion module shown in the figure is as follows. Figure 7 The multi-channel fusion module shown.
[0069] The single-channel multimodal mutual attention fusion module performs a preliminary fusion of the single-channel touch and slip information to obtain fusion features. The module includes an N-layer multimodal mutual decoder module and N-1 iterative multimodal interaction modules. The multimodal mutual decoder module fuses vibration information and stress information, but stress information is easily lost during the iteration process. The introduction of the iterative multimodal interaction module can enhance stress information. The structure of each layer of the multimodal mutual decoder module is the same. The output of each layer is the stress information feature fused with the vibration information feature. The output result of each layer is input into the iterative multimodal interaction module to obtain the fusion feature of the vibration information and stress information of the layer. After iteration, the fusion feature of the vibration information and stress information is finally obtained.
[0070] In the first step, the stress information and vibration information are input into the multimodal mutual decoder module. The Transformer encoder is used to capture the dependencies between different positions in the vibration information data, namely the deep vibration feature F D The input of the first layer of the multimodal mutual decoder module is stress information and deep vibration features F D , stress information generates stress information features F through multi-head self-attention S , the stress information features are projected onto the bond through the linear layer Sum Similarly, project the vibration information characteristics to the bond Sum Next, the interaction attention matrix is generated:
[0071]
[0072] Normalizing the interaction attention matrix, we get:
[0073]
[0074] in, In order to integrate the vibration information characteristics of the strain mode, To fuse the stress information features of the vibration mode. Each element in is considered as a weight of the linear layer and applied to Above, the initial fusion features:
[0075]
[0076] Will Perform linear mapping and then project it to the key of multi-head cross attention With value F S Projected to the weight of multi-head cross attention d k for The dimension of the multimodal mutual decoder module is obtained by the first layer output result, that is, the stress information feature of the first layer fused with the vibration information feature:
[0077]
[0078] The second step is to integrate the stress information features of the first layer into the fusion vibration information features. and stress information feature F S Input iterative multimodal interaction module. First, use linear layer to map F S , generate the stress characteristics of the current layer Next, use the linear layer to project the multimodal fusion features of touch and slip First, we map F with a linear layer 1 S Generate the current layer output of the iterative multimodal interaction module Next, the linear layer 3 is used to project the multimodal fusion features of touch and slip to obtain Calculate the attention matrix of the reorganized stress features and normalize it:
[0079]
[0080] Then through and mapped with a linear layer 2 Generated Get the new stress feature:
[0081]
[0082] Next, Inject into middle:
[0083]
[0084] in, is the learnable weight. As each layer is iterated, BN stands for batch normalization and the output is is the fusion feature of the first layer of vibration information and stress information, which is combined with the deep vibration feature F D They are jointly used as the input of the second layer of the multimodal mutual decoder module. Generate stress information feature F through multi-head self-attention dec1 , through the linear layer to transform F dec1 Projection to Key Sum Similarly, project the vibration information characteristics to the bond Sum Next, the interaction attention matrix is generated:
[0085]
[0086] Normalizing the interaction attention matrix, we get:
[0087]
[0088] In order to integrate the vibration information characteristics of the strain mode, is the multiple fusion features of the fused vibration modes. Each element in is considered as a weight of the linear layer and applied to Above, the initial fusion features:
[0089]
[0090] Will Perform linear mapping and then project it to the key of multi-head cross attention With value F S Projected to the weight of multi-head cross attention d k for The dimension of the second layer is obtained by multimodal mutual decoder module, that is, the stress feature of the second layer fused with vibration features:
[0091]
[0092] Then, the stress features of the second layer are fused with the vibration features and the first layer Input iterative multimodal interaction module. First, use linear layer to map Generate stress characteristics for the current layer First, we use a linear layer 1 to map Generate the current layer output of the iterative multimodal interaction module Next, the linear layer 3 is used to project the multimodal fusion features of touch and slip to obtain Calculate the attention matrix of the reorganized stress features and normalize it:
[0093]
[0094] Then through and mapped with a linear layer 2 Generated Get the new stress feature:
[0095]
[0096] Next, Inject into In the figure, the fusion features of the vibration information and stress information of the second layer are obtained:
[0097]
[0098] Next, Inject into In the figure, the fusion features of the vibration information and stress information of the second layer are obtained:
[0099]
[0100] Repeat the above steps to iterate and obtain the fusion features of the vibration information and stress information of the N-1th layer.
[0101]
[0102] Input it into the last layer of the multimodal mutual decoder module to obtain the final fusion features of vibration information and stress information
[0103]
[0104] The multi-channel fusion module combines the fusion features of the three channels The multi-channel touch-slip multimodal fusion features are further fused and normalized to realize the recognition of object surface information. The key to the multi-channel fusion module is the GRU-attention mechanism model. Among them, GRU is a recurrent neural network structure similar to LSTM, which can further extract the fusion features of the three channels. The information relationship between the two fusion features before and after the sequence is formed, such as the fusion feature The information given to the feature vector Enhance the relationship between the fusion features of different channels input into the attention mechanism. The attention mechanism further extracts the fusion features of vibration information and stress information, and highlights the key part of the fusion features, that is, the feature vector containing the fusion of multi-channel vibration signal and stress signal information.
[0105] The first step is to combine the fusion features of the three channels Splicing to generate sequence
[0106] The second step is to input the sequence F into the multi-channel fusion module to obtain the output h 1 ,h 2 and h 3 .
[0107] The first element of the sequence F Get the output and hidden state h through GRU 1 . First, get the two gating states:
[0108]
[0109] Among them, r is the gate that controls the reset, and z is the gate that controls the update. Then use the reset gate to get the reset data O*r 1 , and then O*r 1 Fusion feature vector Connect and scale the data to the range of (-1, 1) through a tanh activation function:
[0110]
[0111] Here h 1 Mainly contains the current fusion feature vector It is also necessary to remember the current state, by specifically setting h 1 This is done by adding it to the current hidden state. Next, update gating is used to forget and remember:
[0112] h 1 =z 1 *h 1
[0113] The update gate z is in the range of (0,1). The closer z is to 1, the more data is remembered, and the closer z is to 0, the more data is forgotten.
[0114] The second element of the sequence F Get the output and hidden state h through GRU 2 First, through the previous hidden state and Get two gate states:
[0115]
[0116] Then use the reset gating to get the reset data h 1 *r 2 , and then h 1 *r 2 Fusion feature vector Connect and scale the data to the range of (-1, 1) through a tanh activation function:
[0117]
[0118] Here h 2 Mainly contains the current fusion feature vector It is also necessary to remember the current state, by specifically setting h 2 This is done by adding it to the current hidden state. Next, update gating is used to forget and remember to get the output:
[0119] h 2 =(1-z 2 )*h 1+z 2 *h 2
[0120] The third element of the sequence F Similarly, we get the output h 3
[0121] h 3 =(1-z 3 )*h 2 +z 3 *h 3
[0122] The third step is to extract the fusion features of vibration information and stress information through the attention mechanism:
[0123] e i =w i tanh(W i h i +b i ),1≤i≤3
[0124]
[0125] Among them, e i is a verification model, e i Indicates the relevant information h corresponding to the i-th eigenvector i The determined attention probability distribution value, w i and W i represents the weight coefficient matrix of the i-th fusion feature vector, b i Represents the offset corresponding to the i-th fusion feature.
[0126] The third step is to perform normalization calculation to obtain the attention weight matrix:
[0127] A = softmax(v)
[0128] The label corresponding to the feature vector is determined according to the attention weight matrix, thereby achieving overall recognition of the hardness, texture, curvature and other information on the object surface.
[0129] The multi-channel touch-slip multimodal mutual fusion model was trained and verified by the five-fold crossover method. In the first step, the data set was evenly divided into five subsets; in the second step, four of the subsets were used as training sets and the remaining subset was used as a test set. The above two steps were repeated five times, and a different subset was selected as the test set each time. The recognition accuracy results of each training verification were averaged to obtain the recognition accuracy of the multi-channel touch-slip multimodal mutual fusion model.
[0130] In the process of single-channel touch-slip multimodal fusion, the present invention adopts Figure 6The multimodal mutual attention fusion module shown in FIG. 1 is composed of a multimodal mutual decoder module and an iterative multimodal interaction module. This module fuses the stress and vibration modes to obtain a single-channel fusion feature. Figure 7 The multi-channel touch-slip multimodal fusion model structure shown performs multi-channel touch-slip multimodal fusion on three single-channel fusion features to realize the recognition of the surface of the object. Taking the data set containing 800 data groups obtained by actually collecting touch-slip multimodal information as an example, the number of layers of the multimodal mutual decoder module is set to 8, and the multi-channel touch-slip multimodal mutual fusion model is trained and verified by the five-fold cross method, and the recognition accuracy reaches 87.19%. According to the results, it can be seen that the multi-channel touch-slip multimodal mutual fusion model adopted by the present invention has good recognition accuracy.
Claims
1. A multi-channel touch-slip optical fiber sensing recognition method based on multi-modal mutual fusion, comprising the following steps: The first step is to collect multimodal information of touch and slip Build a multi-channel fiber optic sensing system based on fiber Bragg grating (FBG) for touch and slip information. Encapsulate the fiber Bragg grating sensors into different silicone sleeves. The silicone sleeves are placed at the end of the robotic arm to collect multi-modal information of touch and slip. The second step is to build a multi-channel touch and slip dataset A robotic arm is used to contact silicone blocks with different hardness, texture, shape, and curvature to collect multi-channel interference signals, where the central wavelength offset of each interference signal is extracted as stress information, and the instantaneous frequency of the interference signal is demodulated to obtain vibration information. The multi-channel stress information, vibration information, and labels form a multi-channel touch and slip data set; the touch and slip information of each channel has different manifestations at the corresponding position, and the data of each channel are combined to obtain the hardness, texture, shape, and curvature information of the object surface; The third step is to build and train a multi-channel tactile multimodal mutual fusion model. The network structure of the multi-channel tactile multimodal mutual fusion model consists of two parts. The first part is a single-channel multimodal mutual attention fusion module, and the second part is a multi-channel fusion module. (1) The single-channel multimodal mutual attention fusion module performs preliminary fusion on the single-channel touch and slip information to obtain fusion features. The module includes an N-layer multimodal mutual decoder module and N-1 iterative multimodal interaction modules. The multimodal mutual decoder module fuses the vibration information and the stress information, and introduces the iterative multimodal interaction module to enhance the stress information. The structure of each layer of the multimodal mutual decoder module is the same, and the output of each layer is the stress information feature fused with the vibration information feature. The output result of each layer is input into the iterative multimodal interaction module to obtain the fusion feature of the vibration information and the stress information of the layer. After iteration, the fusion feature of the vibration information and the stress information is finally obtained. (2) The multi-channel fusion module, based on the GRU attention mechanism, further fuses the fusion features of each channel to form a multi-channel touch-slip multimodal fusion feature, which is normalized to achieve the recognition of object surface information.
2. The multi-channel touch-slip optical fiber sensing identification method according to claim 1, characterized in that: The single-channel multimodal mutual attention fusion module includes: The stress information and vibration information are input into the multimodal mutual decoder module; the Transformer encoder is used to capture the dependencies between different positions in the vibration information data, namely the deep vibration feature F D ; The input of the first layer of the multimodal mutual decoder module is stress information and deep vibration features F D , stress information generates stress information features F through multi-head self-attention S , the stress information features are projected onto the bond through the linear layer Sum Similarly, the deep vibration feature F is transformed into D Projection to Key Sum Then the interactive attention matrix is generated and normalized to obtain the vibration information features of the fused strain mode. and the stress information characteristics of the fusion vibration mode Will Each element in is considered as a weight of the linear layer and applied to To obtain the initial fusion features Will Perform linear mapping and then project it to the key of multi-head cross attention With value F S Projected to the weight of multi-head cross attention d k for The dimension of the multimodal mutual decoder module is obtained by outputting the first layer of the multimodal mutual decoder module, that is, the stress information feature of the first layer fused with the vibration information feature The stress information features of the first layer are fused with the vibration information features and stress information feature F S Input iterative multimodal interaction module, through the linear layer mapping F S , generate the stress characteristics of the current layer Then use the linear layer to project the multimodal fusion features of touch and slip First, we map F using a linear layer 1 S Generate the current layer output of the iterative multimodal interaction module Next, the linear layer 3 is used to project the multimodal fusion features of touch and slip to obtain Calculate the attention matrix of the reorganized stress feature and normalize it to obtain Then through and mapped with a linear layer 2 Generated Obtaining new stress characteristics Will Inject into middle: in, is the learnable weight. As each layer is iterated, BN stands for batch normalization and the output is is the fusion feature of the first layer of vibration information and stress information, which is combined with the deep vibration feature F D Together they serve as the input to the second layer of the multimodal mutual decoder module; Generate stress information feature F through multi-head self-attention dec1 , through the linear layer to transform F dec1 Projection to Key Sum Similarly, project the vibration information characteristics to the bond Sum Generate an interactive attention matrix; normalize the interactive attention matrix to obtain the vibration information characteristics of the fused strain mode and multiple fusion features of fused vibration modes Will Each element in is considered as a weight of the linear layer and applied to Above, that is, the initial fusion feature Will Perform linear mapping and then project it to the key of multi-head cross attention With value F S Projected to the weight of multi-head cross attention d k for The dimension of the second layer is obtained by multimodal mutual decoder module, that is, the stress feature of the second layer fused with vibration feature The stress features of the second layer are fused with the vibration features and the first layer Input iterative multimodal interaction module; first use linear layer 1 to map Generate the current layer output of the iterative multimodal interaction module Next, the linear layer 3 is used to project the multimodal fusion features of touch and slip to obtain Calculate the attention matrix of the reorganized stress feature and normalize it to obtain Then through and mapped with a linear layer 2 Generated Obtaining new stress characteristics Will Inject into In the process, the fusion features of the vibration information and stress information of the second layer are obtained. Repeat the above steps to iterate and obtain the fusion features of the vibration information and stress information of the N-1th layer. Input it into the last layer of the single-channel multimodal mutual decoder module to obtain the final fusion features of vibration information and stress information 3. The multi-channel touch-slip optical fiber sensing identification method according to claim 2, characterized in that: The multi-channel fusion module comprises: 1) The fusion features of the vibration information and stress information obtained from each channel are finally concatenated into a multi-channel fusion feature sequence F; 2) Input the sequence F into the multi-channel fusion module to obtain the output and hidden state of each dimension through GRU. 3) Extract vibration information and stress information fusion feature v through attention mechanism: 4) Perform normalization calculation to obtain the attention weight matrix: A = softmax(v) The label corresponding to the feature vector is determined according to the attention weight matrix to achieve overall recognition of the hardness, texture, and curvature information of the object surface.
4. The multi-channel touch-slip optical fiber sensing identification method according to claim 3, characterized in that: Assuming the number of channels is 3, the multi-channel fusion module includes: The fusion features of the three channels Splicing to generate sequence Input sequence F into the multi-channel fusion module to obtain outputs h1, h2 and h3: the first element of sequence F Get the output and hidden state h1 through GRU; the second element of the sequence F Get the output and hidden state h2 through GRU; the third element of the sequence F Get the output and hidden state h3 through GRU; Extract vibration information and stress information fusion feature v through attention mechanism: e i =w i tanh(W i h i +b i ),1≤i≤3 Among them, e i Indicates the relevant information h corresponding to the i-th eigenvector i The determined attention probability distribution value, w i and W i represents the weight coefficient matrix of the i-th fusion feature vector, b i Represents the offset corresponding to the i-th fusion feature.
Citation Information
Patent Citations
Stress, temperature and vibration composite detection optical fiber sensor and signal processing method
CN110132329A
Flexible fingertip tactile-slip sensor based on fiber bragg grating and detection method of flexible fingertip tactile-slip sensor
CN116046031A
Visual and tactile multi-mode object recognition method based on deep learning
CN117435973A
Multi-mode optical fiber sensing system based on attention mechanism
CN117520791A
Multi-modal image classification system and method using attention-based multi-interaction network
US20230377318A1
