A method for generating force tactile information based on stacked auto-encoding

Through the combination of deep convolutional network and stacked automatic encoder, a cross-modal conversion model is established, which solves the problem of high dimensionality extraction of tactile signal features of surface material force, and realizes low-dimensional reconstruction and efficient tactile information acquisition.

CN116736975BActive Publication Date: 2024-11-12JILIN UNIVERSITY
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202310669957.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-06-07
Publication Date
2024-11-12
Estimated Expiration
2043-06-07

AI Technical Summary

Technical Problem

In the prior art, the tactile data feature extraction dimensions of the surface material force tactile signals have high dimensions, resulting in huge amounts of model parameters, difficulty in optimization, and long calculation time.

Method used

The deep convolutional network model VGG16 is used to extract the surface tactile image features, and a cross-modal conversion model is established through a stacked automatic encoder and a gated loop network, and the rectangular weight matrix and bias vector are reduced dimensionality, reconstruct the tactile signal, and pass errors. Optimize and determine model parameters.

Benefits of technology

Cross-modal mapping of visual image features to tactile texture features in low-dimensional space reduces the cost of tactile information acquisition, solves the problems of large amount of parameters and long calculation time, and realizes efficient tactile information reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116736975B_ABST
    Figure CN116736975B_ABST
Patent Text Reader

Abstract

The present invention is applicable to the field of tactile Internet technology, and provides a method for generating force tactile information based on stacked auto-encoding. The present invention is based on a deep learning image feature extraction method, uses a deep convolution model VGG16 to extract the features of the surface tactile image, and uses the encoding part of the stacked auto-encoding to compress the force tactile data obtained from the physical surface in real time, extracts features with dimensions much smaller than itself from the original force tactile signal, and combines them with the image texture features of the surface material extracted above, and then establishes a mapping model from the visual texture features of the surface material to the tactile texture features in a low-dimensional space, and the tactile texture features of the surface material obtained will be restored to the tactile texture signal of the surface material through a decoder. The present invention greatly reduces the cost of tactile information acquisition and solves the problems of directly building a mapping model that will bring about a huge number of parameters, difficulty in optimizing the model, and long operation time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of tactile Internet, and in particular relates to a method for generating force tactile information based on stacked automatic encoding. Background Art

[0002] With the development of tactile reproduction technology and the Internet, Internet users have an increasing demand for immersive remote operation and interaction with the physical world. The tactile Internet has huge potential applications in bilateral remote operation and online shopping. In order to achieve an immersive tactile experience, the tactile Internet has generated additional requirements, including the collection, compression, transmission, reconstruction and display of tactile data. Force tactile signals can display the shape and texture of objects through advanced surface tactile devices, which enables the tactile Internet to improve the interactive immersion with real or virtual environments. The present invention focuses on the acquisition of tactile texture features of surface materials in the tactile Internet, specifically by acquiring tactile texture features from visual texture images of surface materials.

[0003] At present, the methods for acquiring tactile data can be mainly divided into the following three categories: (1) Acquisition of tactile data based on external devices, which is to acquire the vibration signal, acceleration signal, etc. when it contacts the surface of an object through a pen-type or other form of acquisition device. This process requires the use of dedicated acquisition equipment, and the acquired data is not suitable for bare finger tactile reproduction; (2) Acquisition of bare finger tactile data, which is to acquire the force signal, acceleration signal, displacement, etc. when the bare finger interacts with the surface of an object through a measuring device. This method has large differences in the data collected from different subjects, and requires professional measurement equipment, which is expensive; (3) Acquisition of tactile data from image information, which is to acquire tactile data from image information through machine learning, feature extraction, etc. However, the current method for extracting tactile data features of surface material force tactile signals has a high dimension. If the mapping model is directly constructed without processing, it will bring about problems such as huge number of parameters, difficulty in model optimization, and long operation time. Summary of the invention

[0004] The purpose of the present invention is to provide a method for generating force tactile information based on stacked auto-encoding, aiming to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for generating force tactile information based on stacked automatic encoding comprises the following steps:

[0007] Step S1, using the deep convolutional network model VGG16 to extract features of the surface tactile image;

[0008] Step S2, extracting features of tactile texture information of the surface material through an encoder in a stacked autoencoder; applying a rectangular weight matrix and a bias vector to reduce the dimension of the original tactile information to obtain features of the tactile information;

[0009] Step S3, after the tactile texture and image texture features of the surface material are extracted, a cross-modal conversion model from visual information to tactile information is established based on a gated recurrent network GRU by using a lightweight machine learning method for time series processing;

[0010] Step S4, reconstructing the tactile texture features of the surface material into tactile texture information of the surface material through a decoder, applying a rectangular weight matrix and a bias vector to reconstruct the features and restore them to the same dimension as the input signal, thereby obtaining a reconstruction of the tactile signal;

[0011] Step S5, calculating the average relative error between the original tactile signal and the reconstructed tactile signal; taking the derivative of the error and updating the weights and biases in the direction of the decreasing derivative, thereby determining the parameters of the encoder and the decoder;

[0012] Step S6, minimizing the error between the tactile texture feature of the surface material obtained by visual texture feature mapping of the surface material and the tactile texture feature obtained by feature extraction of the tactile texture signal of the surface material, thereby determining the parameters of the gated recurrent network;

[0013] Step S7, retaining the trained model; finally, obtaining the tactile texture features through the mapping model and reconstructing the tactile texture signal through the decoder.

[0014] Furthermore, in step S1, the specific way to obtain the features of the surface tactile image is:

[0015] The deep convolution model VGG16 is used to extract the features of surface tactile images. The convolution kernel in the convolution layer of VGG16 slides across the data matrix of the entire image. The coefficients of the convolution kernel and the data in its receptive field are used to calculate the Hadamard product to form a new hidden layer. After multiple convolutions, pooling operations are performed to reduce the dimension of the image data. The data of the last layer of the convolution network is expanded to obtain the feature vector of the image, which is then connected to the fully connected layer or the Softmax layer to complete the classification task. The accuracy of the classification is used to verify the effectiveness of the feature extraction method.

[0016] Furthermore, the network of the VGG16 consists of 13 convolutional layers and 5 pooling layers, and VGG16 adopts a convolutional filter of size 3×3; the pooling is divided into average pooling and maximum pooling, the average pooling is used to average the values ​​in the scanning area of ​​the pooling window, and the maximum pooling is used to retain the maximum value of the data in the area scanned by the pooling window.

[0017] Furthermore, the specific method of step S3 is:

[0018] Picture feature vector p t As input into GRU, the image feature vector p t With the state vector h t-1 As the input vector into the network, the output end gets a processed visual texture feature vector and a new state vector h t ;

[0019] In the structure of GRU, p t and h t-1 is fed into two gates in parallel: the reset gate and the update gate. The reset gate determines h t-1 To what extent is it retained? The update gate controls the update of the state; the reset gate and update gate output r t and z t , calculated by the following two formulas:

[0020] r t =sigmoid(W r p t +U r h t-1 )

[0021] z t =sigmoid(W z p t +U z h t-1 )

[0022] in, are the parameters of the network; given r t , the historical information retained t ⊙h t-1 First, with p t Combined, then by the following formula:

[0023] h t =tanh(W h p t +U h (r t ⊙h t-1 ))

[0024] in, is a parameter, ⊙ represents element-by-element multiplication; in h t Based on the state vector, the state vector will participate in the upcoming time instant state h t , as shown below:

[0025] h t =(1-zt )⊙h t-1 +z t h t

[0026] The processed image signal p t As shown below:

[0027]

[0028] in, is the parameter matrix.

[0029] Furthermore, the specific method of step S5 is:

[0030] The decoder and encoder together form a stacked autoencoder, where all parameters involved can be expressed as a set:

[0031]

[0032] In order to obtain the optimal parameter Θ (l)* , f (l) 、f (l+1) and The three connected layers together form an autoencoder AE, and the AE parameters Θ (l)* Determined by the following formula:

[0033]

[0034] That is, minimize the reconstruction error of each autoencoder, and use the tactile texture signal of the original surface material as the input and target output of the first autoencoder. When the parameters of the first autoencoder are determined, its hidden layer is extracted as the input and target output of the next autoencoder. The parameter determination is carried out in sequence. When the parameters of each encoder are determined, the parameters of the stacked autoencoder are determined accordingly; update the parameters in the direction of the descending derivative, and retain the weights and biases when the error is minimized as the final parameters of the encoder and decoder.

[0035] Furthermore, the specific method of step S6 is:

[0036] Close all parameters of a gated recurrent neural network into a set:

[0037]

[0038] The best parameter combination Φ * The tactile texture features of the surface material are obtained by minimizing the visual texture feature mapping of the surface material. The error between the tactile texture feature c obtained by extracting the tactile texture signal of the surface material is obtained as follows:

[0039]

[0040] The parameter optimization process uses the Adam algorithm to solve the parameter set through gradient descent with momentum to achieve the purpose of minimizing the error.

[0041] Furthermore, the specific method of step S7 is:

[0042] After the training process is completed, the final updated model parameters are retained, the extracted visual image features are used to obtain the tactile texture features through the mapping model, and the obtained tactile texture features are used to reconstruct the tactile texture signals through the decoder, thereby realizing cross-modal generation from visual images to tactile information.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This method for generating force tactile information based on stacked auto-encoding extracts the features of image information and tactile information, performs cross-modal mapping from visual image features to tactile texture features in a low-dimensional space, and reconstructs the generated tactile texture features back into tactile information. This method greatly reduces the cost of tactile information collection and solves the problems of directly constructing a mapping model, which will result in a huge number of parameters, difficulty in optimizing the model, and long calculation time. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a data flow diagram of the present invention.

[0046] Figure 2 This is a relationship diagram between the average relative reconstruction error of the force tactile signal proposed by the present invention and the change in compression ratio.

[0047] Figure 3 Schematic diagram of original and reconstructed force tactile signals.

[0048] Figure 4 Schematic diagram of the original and acquired force tactile signals. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0050] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0051] An embodiment of the present invention provides a method for generating force tactile information based on stacked auto-encoding, comprising the following steps:

[0052] Step S1, using the deep convolutional network model VGG16 to extract features of the surface tactile image;

[0053] Step S2, extracting features of tactile texture information of the surface material through an encoder in a stacked autoencoder; applying a rectangular weight matrix and a bias vector to reduce the dimension of the original tactile information to obtain features of the tactile information;

[0054] Step S3, after the tactile texture and image texture features of the surface material are extracted, a cross-modal conversion model from visual information to tactile information is established based on a gated recurrent network GRU by using a lightweight machine learning method for time series processing;

[0055] Step S4, reconstructing the tactile texture features of the surface material into tactile texture information of the surface material through a decoder, applying a rectangular weight matrix and a bias vector to reconstruct the features and restore them to the same dimension as the input signal, thereby obtaining a reconstruction of the tactile signal;

[0056] Step S5, calculating the average relative error between the original tactile signal and the reconstructed tactile signal; taking the derivative of the error and updating the weights and biases in the direction of the decreasing derivative, thereby determining the parameters of the encoder and the decoder;

[0057] Step S6, minimizing the error between the tactile texture feature of the surface material obtained by visual texture feature mapping of the surface material and the tactile texture feature obtained by feature extraction of the tactile texture signal of the surface material, thereby determining the parameters of the gated recurrent network;

[0058] Step S7, retaining the trained model; finally, obtaining the tactile texture features through the mapping model and reconstructing the tactile texture signal through the decoder.

[0059] In the embodiment of the present invention, in step S2, the specific way of obtaining the tactile information feature is:

[0060] The original force tactile signal s of length N passes through the first layer encoder and obtains preliminary features through the first set of weight matrices and bias vectors:

[0061] f (1) =σ(W (1) s+b (1) )

[0062] in, is the extracted preliminary feature, M (1) <N represents the length of the preliminary feature, is the weight matrix used to extract preliminary features, is the corresponding bias vector, σ is the activation function, ReLU and Sigmoid can be used, and its form is as follows:

[0063] ReLU:

[0064] Sigmoid:

[0065] Among them, x can be any random variable, and the preliminary features obtained are further compressed to obtain the final features:

[0066] f (l) =σ(W (l) f (l-1) +b (l) )

[0067] Among them, l∈{2,,L} is the number of layers of the neural network used to extract features, f (l-1) represents the features extracted from the l-1th layer, M (l) <M (l-1) is the dimension of the lth layer. In particular, we let M (L) =M, f=f (L) Characteristics of force tactile signals.

[0068] In step S4, the specific way to reconstruct the tactile information is as follows:

[0069] The characteristics of the force tactile signal obtained in the reconstruction step S3 are:

[0070]

[0071] in, is the weight matrix required for reconstruction in the decoder, is the bias vector in reconstruction, To use the weight matrix and bias vector to linearly map the reconstructed data, at the last layer of the decoder, we get a reconstruction of the original signal:

[0072]

[0073] in, Has the same dimensions as the original data.

[0074] As a preferred embodiment of the present invention, in step S1, the specific way of obtaining the features of the surface tactile image is:

[0075] The deep convolution model VGG16 is used to extract the features of surface tactile images. The convolution kernel in the convolution layer of VGG16 slides across the data matrix of the entire image. The coefficients of the convolution kernel and the data in its receptive field are used to calculate the Hadamard product to form a new hidden layer. After multiple convolutions, pooling operations are performed to reduce the dimension of the image data. The data of the last layer of the convolution network is expanded to obtain the feature vector of the image, which is then connected to the fully connected layer or the Softmax layer to complete the classification task. The accuracy of the classification is used to verify the effectiveness of the feature extraction method.

[0076] In the embodiment of the present invention, the coefficients of the convolution kernel and the data in its receptive field are used to calculate the Hadamard product to form a new hidden layer, where the activation function is ReLU, to form a new feature map.

[0077] As a preferred embodiment of the present invention, the network of VGG16 is composed of 13 convolutional layers and 5 pooling layers, and VGG16 adopts a convolution filter of 3×3 size; the pooling is divided into average pooling and maximum pooling, average pooling is used to average the values ​​in the scanning area of ​​the pooling window, and maximum pooling is used to retain the maximum value of the data in the area scanned by the pooling window.

[0078] In the embodiment of the present invention, after several times of convolution and pooling, the image dimension has been greatly reduced.

[0079] As a preferred embodiment of the present invention, the specific approach of step S3 is:

[0080] Picture feature vector p t As input into GRU, the image feature vector p t With the state vector h t-1 As the input vector into the network, the output end gets a processed visual texture feature vector and a new state vector h t ;

[0081] In the structure of GRU, p t and h t-1 is fed into two gates in parallel: the reset gate and the update gate. The reset gate determines h t-1 To what extent is it retained? The update gate controls the update of the state; the reset gate and update gate output r t and z t , calculated by the following two formulas:

[0082] r t =sigmoid(W r p t +U r h t-1 )

[0083] z t =sigmoid(W z p t +U z h t-1 )

[0084] in, are the parameters of the network; given r t , the historical information retained t ⊙h t-1 First, with p t Combined, then by the following formula:

[0085] h t =tanh(W h p t +U h (r t ⊙h t-1 ))

[0086] in, is a parameter, ⊙ represents element-by-element multiplication; in h t Based on the state vector, the state vector will participate in the upcoming time instant state h t , as shown below:

[0087] h t =(1-z t )⊙h t-1 +z t h t

[0088] The processed image signal p t As shown below:

[0089]

[0090] in, is the parameter matrix.

[0091] In the embodiment of the present invention, σ() is a selectable activation function that affects the performance of GRU.

[0092] As a preferred embodiment of the present invention, the specific approach of step S5 is:

[0093] The decoder and encoder together form a stacked autoencoder, where all parameters involved can be expressed as a set:

[0094]

[0095] In order to obtain the optimal parameter Θ (l)* , f (l) 、f(l+1) and The three connected layers together form an autoencoder AE, and the AE parameters Θ (l)* Determined by the following formula:

[0096]

[0097] That is, minimize the reconstruction error of each autoencoder, and use the tactile texture signal of the original surface material as the input and target output of the first autoencoder. When the parameters of the first autoencoder are determined, its hidden layer is extracted as the input and target output of the next autoencoder. The parameter determination is carried out in sequence. When the parameters of each encoder are determined, the parameters of the stacked autoencoder are determined accordingly; update the parameters in the direction of the descending derivative, and retain the weights and biases when the error is minimized as the final parameters of the encoder and decoder.

[0098] As a preferred embodiment of the present invention, the specific approach of step S6 is:

[0099] Close all parameters of a gated recurrent neural network into a set:

[0100]

[0101] The best parameter combination Φ * The tactile texture features of the surface material are obtained by minimizing the visual texture feature mapping of the surface material. The error between the tactile texture feature c obtained by extracting the tactile texture signal of the surface material is obtained as follows:

[0102]

[0103] The parameter optimization process uses the Adam algorithm to solve the parameter set through gradient descent with momentum to achieve the purpose of minimizing the error.

[0104] As a preferred embodiment of the present invention, the specific approach of step S7 is:

[0105] After the training process is completed, the final updated model parameters are retained, the extracted visual image features are used to obtain the tactile texture features through the mapping model, and the obtained tactile texture features are used to reconstruct the tactile texture signals through the decoder, thereby realizing cross-modal generation from visual images to tactile information.

[0106] Example 1

[0107] Simulation experiment: The applicability of the method for generating force tactile information based on stacked automatic encoding proposed in the present invention to force tactile signals is analyzed through simulation experiment data. The software used in the simulation experiment is PyCharm software and Keras framework.

[0108] 1. Image feature extraction based on VGG16

[0109] The proposed method is numerically evaluated using the HapTex dataset. The HapTex dataset contains tactile signals and visual texture images collected from 120 fabrics, where the visual texture image is collected for each material sample, a total of 120 samples, and the force tactile signal measurement process is repeated 4 times, containing a total of 480 data samples. The size of the texture image is 2362*2362 pixels. The input dimension of the visual texture feature extraction method of the surface material cited in this paper is 224*224, so in the simulation experiment, each visual texture image of HapTex is randomly intercepted into a square of 224*224 pixels. The optimizer uses the recently developed stochastic gradient descent solver Adam, and sets its learning rate to 0.001 for parameter optimization. The extracted feature dimension is 512. The Softmax classification layer is connected after the output layer. The Softmax function can convert the output values ​​of multiple classifications into a probability distribution with a range of , and a sum of 1, and the parameters are trained. The effectiveness of the visual texture feature extraction method of the surface material proposed in this paper is verified by the classification accuracy. The classification accuracy was thus 98.25%.

[0110] 2. Tactile texture feature extraction based on SAE

[0111] Before conducting performance testing, the hyperparameters are selected first: For the SAE model, the first thing to determine is the number of layers L in the model, because it determines the model structure and is the basis for all subsequent parameter selection. After balancing the compression effect and the total number of parameters, L was determined to be 2, that is, the stacked autoencoder has a total of 2L+1=5 layers. The length of each hidden layer inside the encoder is set to {1000,512,10} after balancing the model running time and reconstruction error. The parameter determination process of SAE is performed on each training and validation subset. Each training subset iterates 300 epochs and is verified by the validation set, and the batch size is 32.

[0112] In order to obtain the best compression effect of the SAE model on the force tactile texture signal, the present invention considers four different compression ratios, namely N / M∈{25,50,100,200}, that is, the dimensions of the compressed tactile texture signal are {40,20,10,5}. When the parameters and structures other than the related parameters are kept unchanged, the reconstruction error of the force tactile texture signal under different N / M is as follows: Figure 2 It can be seen that when the compression ratio N / M increases from 100 to 200, the reconstruction errors of the normal force and the tangential force increase significantly, so the compression ratio is finally determined to be 100.

[0113] Under the conditions of determining N / M=100 and the activation function as ReLU, the average running time for parameter determination of the tactile texture feature extraction model of the surface material is 61.95s. Once the parameters are stable, the parameters in the model will be regarded as constants and will no longer change with the input, and the test subset will be used to verify the performance of the tactile texture feature extraction model based on the stacked autoencoder. At a 95% confidence interval, the relative percentage error of the test set is: 1.94 (±0.12) for normal force and 4.64 (±0.05) for tangential force. Here, the average execution running time of the stacked encoder feature extraction is 81.3ms. One set of original normal force signals and reconstructed normal force signals, one set of original tangential force signals and reconstructed tangential force signals are as follows Figure 3 As shown, it can be seen that the reconstructed signal is very similar to the original signal.

[0114] 3. GRU-based cross-modal model mapping

[0115] First, the encoder in the stacked autoencoder is used to extract the surface tactile texture signal of length 1000 into the features of the surface material tactile texture signal of length 10; then VGG16 is used to reduce the dimension of the surface material visual texture image of dimension 224*224*3 to feature vectors of length 512, 2048 and 1024; then the visual texture features of the surface material are sent to the neural network composed of GRU and fully connected layers to complete the conversion from visual texture features to tactile texture features; then the tactile texture features of the surface material obtained are reconstructed using the decoder obtained above; the visual texture features are sent to the conversion model as the input of GRU to complete the acquisition of the tactile texture features of the surface material. At the same time, in order to prevent overfitting, L2 regularization is introduced in each layer of the conversion model, and the regularization coefficient is selected as 0.001, and the Dropout layer is introduced in the GRU, and the Dropout ratio is 0.5, that is, in each gradient update, 50% of the parameters are randomly selected and do not participate in the model update, thereby avoiding overfitting to a certain extent. After balancing the running time and model accuracy, the number of iterations was set to 500. The model was now fitted and the batch size was set to 32.

[0116] A set of tactile texture signals of surface materials obtained by the tactile texture feature acquisition model of surface materials constructed in the above manner are as follows: Figure 4 As shown, from Figure 4It can be seen that the tactile texture signal of the surface material obtained from the image is similar to the original tactile texture signal. The data provided by the above simulation experiments prove that the solution proposed by the present invention to obtain tactile texture features from the visual texture image of the surface material is feasible.

[0117] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for generating force tactile information based on stacked automatic encoding, characterized in that: The following steps are involved: Step S1, using the deep convolutional network model VGG16 to extract features of the surface tactile image; Step S2, extracting features of tactile texture information of the surface material through an encoder in a stacked autoencoder; applying a rectangular weight matrix and a bias vector to reduce the dimension of the original tactile information to obtain features of the tactile information; Step S3, after the tactile texture and image texture features of the surface material are extracted, a cross-modal conversion model from visual information to tactile information is established based on a gated recurrent network GRU by using a lightweight machine learning method for time series processing; Step S4, reconstructing the tactile texture features of the surface material into tactile texture information of the surface material through a decoder, applying a rectangular weight matrix and a bias vector to reconstruct the features and restore them to the same dimension as the input signal, thereby obtaining a reconstruction of the tactile signal; Step S5, calculating the average relative error between the original tactile signal and the reconstructed tactile signal; taking the derivative of the error and updating the weights and biases in the direction of the decreasing derivative, thereby determining the parameters of the encoder and the decoder; Step S6, minimizing the error between the tactile texture feature of the surface material obtained by visual texture feature mapping of the surface material and the tactile texture feature obtained by feature extraction of the tactile texture signal of the surface material, thereby determining the parameters of the gated recurrent network; Step S7, retaining the trained model; finally obtaining the tactile texture features through the mapping model and reconstructing the tactile texture signal through the decoder; In step S1, the specific method of obtaining the features of the surface tactile image is: The deep convolution model VGG16 is used to extract the features of the surface tactile image. The convolution kernel in the convolution layer of VGG16 slides across the data matrix of the entire image. The coefficients of the convolution kernel and the data in its receptive field are used to calculate the Hadamard product to form a new hidden layer. After multiple convolutions, pooling operations are performed to reduce the dimension of the image data. The data of the last layer of the convolution network is expanded to obtain the feature vector of the image, which is then connected to the fully connected layer or the Softmax layer to complete the classification task. The accuracy of the classification is used to prove the effectiveness of the feature extraction method. In step S2, the specific way to obtain the tactile information feature is: Length is The original force tactile signal After the first layer of encoder, preliminary features are obtained through the first set of weight matrices and bias vectors: ; in, is the initial feature extracted, represents the length of the preliminary feature, , is the weight matrix used to extract preliminary features, is the corresponding bias vector, is the activation function; The obtained preliminary features are further compressed to obtain the final features: ; in, The number of neural network layers used to extract features, , Representative The features extracted by the layer, For the The dimension of the layer; , is the characteristic of force tactile signal; The specific approach of step S3 is: Image feature vector As input into GRU, the image feature vector With the state vector As the input vector into the network, the output end gets a processed visual texture feature vector and a new state vector ; In the structure of GRU, and is fed into two gates in parallel: the reset gate and the update gate. The reset gate determines To what extent is it retained? The update gate controls the update of the state; the reset gate and update gate output and , calculated by the following two formulas: ; ; in, , , , are the parameters of the network; given , retain historical information First with Combined, then by the following formula: ; in, is a parameter, represents element-wise multiplication; in Based on the state vector, the state vector will participate in the upcoming time instant state , as shown below: ; Processed image signal As shown below: ; in, is the parameter matrix, is the activation function; In step S4, the specific way to reconstruct the tactile information is as follows: The characteristics of the force tactile signal obtained in the reconstruction step S3 are: ; in, is the weight matrix required for reconstruction in the decoder, is the bias vector in reconstruction, In order to use the weight matrix and bias vector to obtain the reconstructed data, in the last layer of the decoder, the reconstruction of the original signal is obtained: ; in, , Have the same dimensions as the original data; The specific approach of step S5 is: The decoder and encoder together form a stacked autoencoder, where all parameters involved can be expressed as a set: ; In order to obtain the optimal parameters , , and The three connected layers together form an autoencoder AE, and the AE parameters Determined by the following formula: ; That is, minimize the reconstruction error of each autoencoder, and use the tactile texture signal of the original surface material as the input and target output of the first autoencoder. When the parameters of the first autoencoder are determined, its hidden layer is extracted as the input and target output of the next autoencoder. The parameters are determined in sequence. When the parameters of each encoder are determined, the parameters of the stacked autoencoder are determined accordingly. Update the parameters in the direction of the descending derivative, and retain the weights and biases with the minimum error as the final parameters of the encoder and decoder. The specific approach of step S6 is: Close all parameters of a gated recurrent neural network into a set: ; The best parameter combination The tactile texture features of the surface material are obtained by minimizing the visual texture feature mapping of the surface material. The error between the tactile texture feature c obtained by extracting the tactile texture signal of the surface material is obtained as follows: ; The parameter optimization process uses the Adam algorithm to solve the parameter set through gradient descent with momentum to achieve the purpose of minimizing the error; The specific approach of step S7 is: After the training process is completed, the final updated model parameters are retained, the extracted visual image features are used to obtain the tactile texture features through the mapping model, and the obtained tactile texture features are used to reconstruct the tactile texture signals through the decoder, thereby realizing cross-modal generation from visual images to tactile information.

2. The method for generating force tactile information based on stacked auto-encoding according to claim 1, characterized in that: The network of VGG16 consists of 13 convolutional layers and 5 pooling layers. VGG16 uses a convolutional filter of 3×3 size. The pooling is divided into average pooling and maximum pooling. Average pooling is used to average the values ​​in the scanning area of ​​the pooling window, and maximum pooling is used to retain the maximum value of the data in the area scanned by the pooling window.

Citation Information

Patent Citations

  • Robot vision SLAM closed-loop detection method based on stack type combined auto-encoder

    CN111753789A

  • Haptic signal processing method based on stack type automatic coding

    CN113627337A

  • Image reconstruction method and device for cross-modal communication system

    CN113628294A

  • Visual-tactile signal adaptive reconstruction method based on transfer learning

    CN115993888A

  • System and method for video captioning

    US10699129B1