A tactile material classification method based on DDQN and generative adversarial network

Abstract

The present invention relates to a tactile material classification method based on DDQN and generative adversarial network, which belongs to a tactile material classification method. The generative adversarial network GAN is trained using pictures and acceleration spectrum diagrams of tactile materials, and sent to the VGG19 network to extract features, the reward function of DDQN is set, the evaluation network of DDQN estimates the Q value, the global optimal solution is found, the trained network is saved, and the pictures are classified according to the action corresponding to the maximum Q value. The advantage is that the tactile information generated by the generative adversarial network GAN is used to classify tactile materials, which improves the noise resistance of the data. DDQN is used for classification problems for the first time, and the convolutional neural network VGG19 is introduced to process pictures, which avoids noise interference, ensures the robustness of the extracted features, avoids the complex tactile information collection process, saves the classification cost, and compared with the traditional classification CNN method, only a small time cost is used, the classification accuracy is improved, and the purchase of expensive tactile information collection equipment is avoided.

Description

Technical Field

[0001] The present invention relates to a tactile material classification method, and in particular to a tactile material classification method based on DDQN and a generative adversarial network. Background Art

[0002] Tactile material classification has become an important and rapidly developing research topic in recent years. Tactile material classification has a wide range of application scenarios, such as robot recognition, virtual reality, etc.

[0003] In recent years, many methods for the classification of tactile materials have been proposed. Among them, classification using visual information is one of the main classification methods. At the beginning of the study of tactile material classification, many methods were used to extract defined features from images for classification. They used random projection to compress images and obtain image features, and then used the K-order nearest neighbor method to classify these features. In addition, calculating the gray level co-occurrence matrix is ​​also a very advanced method. They used the k-order nearest neighbor to classify the features obtained from the gray level co-occurrence matrix. Since these methods have compressed the image information a lot, the differences between materials have become so small that it is difficult to apply them to the classification of tactile materials. With the development of convolutional neural networks (CNNs), many CNN-based tactile material classification methods have been developed. In 2015, Mircea Cimpoi et al. used CNN as a feature extraction tool and defined the Fisher-vector descriptor extracted from the output of the intermediate layer of CNN as a feature. However, a major disadvantage of CNN-based methods is that they cannot distinguish between two pictures with similar textures and close visual information.

[0004] With the rapid update of tactile sensors in recent years, tactile material classification methods based on visual-tactile fusion of tactile information (such as acceleration, friction, etc.) and visual information (pictures) have begun to develop. In 2014, Matti Strese et al. collected visual information and tactile information (acceleration, impact acceleration, friction, etc.) of 69 materials, defined several features about the tactile information and visual information of the materials, and used these features to train the naive Bayes classifier, achieving a classification accuracy of 74%. In 2020, Matti Strese et al. further expanded the database of tactile materials and constructed a tactile material database containing 184 materials, including visual information and tactile information of tactile materials, defined up to 15 features about the visual information and tactile information of tactile materials, and used the random forest classification method to achieve a classification accuracy of 90%. However, these classification methods all need to collect visual information and tactile information of tactile materials at the same time in actual classification scenarios. The tactile information collection process of tactile materials is cumbersome and requires expensive tactile information collection equipment, which increases the cost of classification.

[0005] As a deep reinforcement learning algorithm, DDQN is very popular in the field of robot decision-making and games. Compared with existing classification methods, its adversarial system composed of evaluation network and target network can make the training of network parameters faster and prevent overfitting more prominently. At the same time, DDQN's representation ability is much stronger than existing classification methods. It maps the input features to a higher-dimensional space, and the classification effect is more significant. However, there is no research on using DDQN for classification problems.

[0006] In general methods, the features used for classification are always confused with noise during processing, which affects the performance of the classifier. VGG19 was proposed by the Visual Geometry Group of the University of Oxford in 2015. Its proposal proves that increasing the depth of the network affects the performance of the network to a certain extent. VGG19 consists of 16 convolutional layers and 3 fully connected layers, and performs very well in classifying images and extracting image features. However, the last three fully connected layers of VGG19 contain a large number of parameters, which seriously affects the performance of the method. Summary of the invention

[0007] The present invention provides a tactile material classification method based on DDQN and generative adversarial network to solve the problem of requiring complex tactile information collection equipment when classifying tactile materials in real scenes.

[0008] The technical solution adopted by the present invention comprises the following steps:

[0009] (1) Use images of tactile materials and acceleration spectrograms to train the Generative Adversarial Network (GAN);

[0010] (2) Send the acceleration spectrogram and tactile material images generated by the trained generative adversarial network GAN into the VGG19 network to extract features;

[0011] (3) The material features extracted by VGG19 are regarded as the states of DDQN, and their labels are regarded as the actions of DDQN;

[0012] (4) Set the reward function of DDQN;

[0013] (5) The evaluation network of DDQN estimates the Q value and selects the action with the largest Q value to pass to the target network;

[0014] (6) The target network of DDQN calculates the temporal difference error of the estimated Q value of the evaluation network;

[0015] (7) Perform stochastic gradient descent optimization on the time difference error of the estimated Q value of the evaluation network until the global optimal solution is found. The training is completed and the trained network is saved.

[0016] (8) When a new image is input into the trained network, the generative adversarial network generates an acceleration spectrum corresponding to the material based on the new image, and then inputs the new image and the generated acceleration spectrum into VGG19 to extract the material features. The evaluation network Q′ of DDQN estimates the Q value corresponding to each action based on the material features, and classifies the image according to the action corresponding to the maximum Q value.

[0017] The specific method of training the generative adversarial network GAN in step (1) of the present invention is:

[0018] The category label of tactile material can be expressed as Where M represents M kinds of materials, and the image set of tactile materials is represented as in Indicates The image set of materials, the acceleration spectrum atlas of tactile materials is expressed as in Indicates The acceleration spectrum atlas of the material is collected at the same time. That is, the acceleration spectrum of the material is collected at the same time as the image information of the material. The collection process of the material is repeated multiple times, and the image and acceleration spectrum of the material are collected one by one each time.

[0019] Use the image set of tactile materials and the acceleration spectrum atlas to train the generative adversarial network GAN. The generative adversarial network GAN includes a generator network Generator and a discriminator network Discriminator, referred to as G and D respectively. The generator network G inputs a noise output to generate a new image. The discriminator network D is used to judge whether the new image generated by the generator network G is true, that is, whether it meets the requirements. When training the generative adversarial network, the generator network G is first fixed, the optimal discriminator network D is solved, and then the optimal discriminator network D is used to solve the generator network G;

[0020]

[0021] V(G,D) is the loss function of the generative adversarial network GAN, p data is the data distribution of the acceleration spectrum y, p G is the data distribution obeyed by the data generated by the generator network G, D(y) is the probability that the acceleration spectrum graph generated by the generator G is the real acceleration spectrum graph, and the generator G obtained after training should conform to the following formula:

[0022]

[0023] D(y) should be as close to 1 as possible, and the generator G should have the smallest error possible, so that the error between the generated acceleration spectrum and the real acceleration spectrum is minimized;

[0024] After a large amount of data training, when the generative adversarial network converges, the training ends, and the picture of the tactile material is input into the trained generative adversarial network GAN to obtain the generated acceleration spectrum map, which is used to extract features from the picture of the tactile material and the acceleration spectrum map generated by the generative adversarial network GAN.

[0025] The specific approach of VGG19 feature extraction in step (2) of the present invention is:

[0026] VGG19 is used as the feature extraction network. VGG19 consists of 16 convolutional layers and 3 fully connected layers. A global average pooling layer is used to replace the three fully connected layers. The global average pooling layer outputs the average value of all pixels in each feature map obtained by convolution. These average values ​​are then regarded as feature vectors. The image set and the acceleration spectrum map generated by the generative adversarial network GAN are respectively input into VGG19 to obtain the features of the material image and the acceleration spectrum map generated by the generative adversarial network GAN. The image features and acceleration spectrum map features obtained by a single sampling of a material are then concatenated to obtain the corresponding material features. The material feature set is represented as in Indicates These extracted material features are used to construct the states and actions of DDQN in the next step.

[0027] The specific approach of constructing the DDQN state and action in step (3) of the present invention is:

[0028] Status: Before each iteration, randomly shuffle the material features extracted by VGG19 from the image set and acceleration spectrum atlas Then treat them as states {S1,…,S h}, h is the sum of the number of material features of all materials, and this method is used to convert image features into state sequences;

[0029] Action: The goal of classification is to find the correct label for each image, which is defined as making the correct action. As the action space, DDQN is in state S t Then the state S t Transform to a new state S t+1 , t represents the tth step performed by DDQN.

[0030] The specific method of setting the reward function in step (4) of the present invention is:

[0031] The reward function of DDQN is used to encourage the DDQN classifier to make correct decisions. t That is, when a material feature is input into DDQN, DDQN will make an action a t That is, a label is selected, and then a reward R is given according to whether the action is correct or not, that is, whether the correct label of the material feature is selected. If the correct action is made, a positive reward is given, otherwise, a negative reward is given. The label is l i Take the material characteristics as an example:

[0032]

[0033] The reward function is used by DDQN to calculate action values.

[0034] The specific way of evaluating the action of selecting the maximum Q value in step (5) of the present invention is:

[0035] In DDQN, a neural network-based evaluation network Q′ is established to estimate the action value Q;

[0036] The Q value reflects the estimate of the future reward that DDQN will obtain if it takes action a in state s.

[0037]

[0038] Among them, w (t) represents the parameters of the evaluation network Q′ at step t, G t represents the reward for taking action a in state s plus the estimated future reward, represents the expectation of multiple estimated rewards, G t It represents the reward obtained by taking action a in state s plus the future reward, and uses the discount factor γ to reduce the impact of future rewards on the current action selection;

[0039] G t =R t +γR t+1 +γ 2 R t+2 +γ 3 R t+3 +…+γ k R t+k

[0040] R t Represents the state s at step t t Make an action t Then the reward is obtained according to whether the action is right or wrong, R t+1 represents the reward of step t+1, and k represents the number of steps of DDQN in one iteration;

[0041] In addition to the evaluation network Q′, DDQN has another network target network Its construction is the same as the evaluation network Q′, and its parameters are copied from the evaluation network Q′ at regular intervals. t+1 , the evaluation network Q′ estimates the Q values ​​corresponding to all actions, and then selects the action corresponding to the maximum Q value:

[0042]

[0043] Among them, Q(s t+1 ,a;w (t) ) represents the Q value estimated by the evaluation network Q′, and the evaluation network Q′ selects the action a corresponding to the maximum Q value * After that, action a * The temporal difference error passed to the target network for computing the estimated Q value.

[0044] The specific method of calculating the time difference error of the target network in step (6) of the present invention is:

[0045] Target Network The action a corresponding to the maximum Q value selected by the evaluation network Q′ * Used to calculate the time difference target TD targety t ,y t The calculation formula is as follows:

[0046]

[0047] r represents the state of DDQN in the previous state s t The reward obtained after taking an action according to the unbalanced reward function, w (t-tmodc) Indicates that the parameters of the evaluation network Q' are copied to the target network every C steps To calculate the time difference error, the calculated time difference target is used to calculate the time difference error δ of the estimated Q value of the evaluation network Q′ t ;

[0048] δ t =Q(s t ,a;w (t) )-y t

[0049] The time difference error δ obtained at step t is t Will be used for parameter update.

[0050] The specific approach of reducing the estimated Q value error optimization parameter in step (7) of the present invention is:

[0051] The target network The time difference error is obtained by subtracting the calculated time difference target from the Q value evaluated by the evaluation network Q′. The stochastic gradient descent method is used to reduce this error. First, the error function is defined as:

[0052]

[0053] Then use the stochastic gradient descent method to reduce the error and optimize the parameters. The steps are:

[0054]

[0055] where w (t+1) Represents the parameters of the t+1th step. As the training progresses, the gradient descent is continuously used to optimize the parameters until the global optimal solution is found, that is, the value of the error function L(w) is minimized. The training ends and the trained network is saved.

[0056] The advantages of the present invention are:

[0057] 1. Use the tactile information generated by the generative adversarial network GAN to classify tactile materials, which improves the data's ability to resist noise. The generative adversarial network GAN can convert two correlated data distributions into each other, and the visual information and tactile information of the tactile material are correlated. Use the visual information pictures and tactile information acceleration spectrum graphs of the tactile material to train the generative adversarial network, and you can get a model that generates acceleration spectrum graphs when the input picture is input. In the actual classification scenario, you only need to collect pictures of the tactile material, and you can get the tactile information acceleration spectrum graph of the tactile material through the generative adversarial network. Use the pictures and the generated acceleration spectrum graphs for classification to achieve the classification effect of visual and tactile fusion.

[0058] 2. This paper applies DDQN to classification problems for the first time, regards the classification problem as a decision-making process, regards the classification of data and finding the correct label as making the correct action in a certain state, and perfectly applies the advantages of DDQN to the classification problem, achieving very good results. The parameter update mechanism of DDQN avoids the occurrence of overfitting and ensures the robustness of the trained network; introduces the convolutional neural network VGG19 to process images, replaces the fully connected layer with the global average pooling layer, reduces parameters and improves operating performance, avoids noise interference, and ensures the robustness of the extracted features.

[0059] 3. In actual classification scenarios, the present invention only needs to collect visual information pictures of tactile materials to achieve the classification accuracy of visual and tactile fusion, and does not need to collect tactile information of tactile materials, thus avoiding the complicated tactile information collection process and saving classification costs. Compared with the traditional classification CNN method, it only takes a very small time cost, improves the classification accuracy, and avoids the purchase of expensive tactile information collection equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is the structure in which VGG19 in the present invention extracts features of the acceleration spectrogram generated by the image and the generative adversarial network GAN and feeds them into the DDQN classification;

[0061] Figure 2 is a data flow diagram of the present invention;

[0062] Figure 3 is a VGG19 structure diagram used in the present invention to extract material features;

[0063] Figure 4 It is an acceleration spectrum diagram generated by the generative adversarial network of the present invention. DETAILED DESCRIPTION

[0064] The following steps are involved:

[0065] (1) Use images of tactile materials and acceleration spectrograms to train the Generative Adversarial Network (GAN):

[0066] The category label of tactile material can be expressed as Where M represents M types of materials. The image set of tactile materials is represented as in Indicates The acceleration spectrum atlas of tactile materials is represented as in Indicates Acceleration spectrum atlas of a material. It should be noted that the image set and acceleration spectrum atlas used for training are collected at the same time, that is, when collecting the image information of a material, the acceleration spectrum of the material is also collected. The collection process for a material is repeated multiple times, and each time the material image and acceleration spectrum are collected one by one;

[0067] Use the image set of tactile materials and the acceleration spectrum atlas to train the Generative Adversarial Network (GAN). The Generative Adversarial Network (GAN) includes the Generator Network (G) and the Discriminator Network (D). The Generator Network G inputs a noise output to generate a new image. The Discriminator Network D is used to determine whether the new image generated by the Generator Network G is true, that is, whether it meets the requirements. When training the Generative Adversarial Network, first fix the Generator Network G, solve the optimal Discriminator Network D, and then use the optimal Discriminator Network D to solve the Generator Network G.

[0068]

[0069] V(G,D) is the loss function of the generative adversarial network GAN, p data is the data distribution of the acceleration spectrum y, pG is the data distribution obeyed by the data generated by the generator network G, D(y) is the probability that the acceleration spectrum graph generated by the generator G is the real acceleration spectrum graph, and the generator G obtained after training should conform to the following formula:

[0070]

[0071] D(y) should be as close to 1 as possible, and the generator G should have the smallest error possible, so that the error between the generated acceleration spectrum and the real acceleration spectrum is minimized;

[0072] After a large amount of data training, when the generative adversarial network converges, the training ends, and the picture of the tactile material is input into the trained generative adversarial network GAN to obtain the generated acceleration spectrum. The next step is to extract features using the picture of the tactile material and the acceleration spectrum generated by the generative adversarial network GAN.

[0073] (2) Send the acceleration spectrum and tactile material images generated by the trained generative adversarial network GAN into the VGG19 network to extract features:

[0074] VGG19 is used as the feature extraction network. VGG19 consists of 16 convolutional layers and 3 fully connected layers. Compared with networks using large convolutional kernels (such as AlexNet), its deeper convolutional layers and smaller convolutional kernels ensure that the extracted features are more robust; however, the fully connected layer increases the computational complexity and easily leads to overfitting; therefore, a global average pooling layer is used to replace the three fully connected layers. The global average pooling layer outputs the average value of all pixels in each feature map obtained by convolution, and then these average values ​​are regarded as feature vectors; the picture set and the acceleration spectrum map generated by the generative adversarial network GAN are respectively input into VGG19 to obtain the features of the material picture and the acceleration spectrum map generated by the generative adversarial network GAN, and then the picture features and acceleration spectrum map features obtained by a single sampling of a material are spliced ​​to obtain the corresponding material features, and the material feature set is represented as in Indicates The material feature set of the material will be used to construct the state and action of DDQN in the next step;

[0075] (3) Treat the material features extracted by VGG19 as the states of DDQN and their labels as the actions of DDQN:

[0076] Status: Before each iteration, we randomly shuffle the material features extracted by VGG19 from the image set and acceleration spectrum atlas Then treat them as states {S1,…,S h}, h is the sum of the number of material features of all materials. We use this method to convert image features into state sequences;

[0077] Action: The goal of classification is to find the correct label for each image, which we define here as making the correct action. As the action space, DDQN is in state S t Then the state S t Transform to a new state S t+1 , t represents the tth step of DDQN (unless otherwise specified in the present invention, t refers to the tth step), and the next step is to set the reward function of DDQN to motivate DDQN to make the correct decision;

[0078] (4) Set the reward function of DDQN:

[0079] In this invention, the reward function of DDQN is used to motivate the DDQN classifier to make correct decisions. t That is, when a material feature is input into DDQN, DDQN will make an action a t That is, a label is selected, and then a reward R is given according to whether the action is correct or not, that is, whether the correct label of the material feature is selected. If the correct action is made, a positive reward is given, otherwise, a negative reward is given. The label is Take the material characteristics as an example:

[0080]

[0081] The reward function is used by DDQN to calculate the action value in the next step;

[0082] (5) The evaluation network of DDQN estimates the Q value and selects the action with the largest Q value to pass to the target network:

[0083] In DDQN, a neural network-based evaluation network Q′ is established to estimate the action value (Q value);

[0084] The Q value reflects the estimate of the future reward obtained by DDQN taking action a in state s;

[0085]

[0086] Among them, w (t) represents the parameters of the evaluation network Q′ at step t, G t It represents the reward obtained by taking action a in state s plus the future reward. It means taking the expectation of multiple estimated rewards and using the discount factor γ to reduce the impact of future rewards on current action selection;

[0087] G t =R t +γR t+1 +γ 2 R t+2 +γ 3 R t+3 +…+γ k R t+k

[0088] R t Represents the state s at step t t Make an action t Then the reward obtained based on the right or wrong of the action and the unbalanced reward function, R t+1 represents the reward of step t+1, and k represents the number of steps of DDQN in one iteration;

[0089] In particular, in addition to the evaluation network Q′, DDQN has another network target network Its construction is the same as the evaluation network Q′, and its parameters are copied from the evaluation network Q′ at regular intervals. t+1 , the evaluation network Q′ estimates the Q values ​​corresponding to all actions, and then selects the action corresponding to the maximum Q value;

[0090]

[0091] Among them, Q(s t+1 ,a;w (t) ) represents the Q value estimated by the evaluation network Q′. The evaluation network Q′ selects the action a corresponding to the maximum Q value * After that, action a * The time difference error passed to the target network for calculating the estimated Q value;

[0092] (6) The target network of DDQN calculates the temporal difference error of the estimated Q value of the evaluation network:

[0093] Target Network The action a corresponding to the maximum Q value selected by the evaluation network Q′ * Used to calculate the time difference target (TD target) y t ,y t The calculation formula is as follows:

[0094]

[0095] r represents the state of DDQN in the previous state s t The reward obtained after taking an action according to the unbalanced reward function, w (t-tmodc) Indicates that the parameters of the evaluation network Q' are copied to the target network every C steps To calculate the time difference error, the calculated time difference target is used to calculate the time difference error δ of the estimated Q value of the evaluation network Q′ t ;

[0096] δ t =Q(s t ,a;w (t) )-y t

[0097] The time difference error δ obtained at step t is t Will be used for parameter update;

[0098] (7) The target network The time difference error is obtained by subtracting the calculated time difference target from the Q value evaluated by the evaluation network Q′. The stochastic gradient descent method is used to reduce this error. The error function is defined as:

[0099]

[0100] Then use the stochastic gradient descent method to reduce the error and optimize the parameters. The steps are:

[0101]

[0102] where w (t+1) Represents the parameters of step t+1. As the training progresses, the parameters are continuously optimized using gradient descent until the global optimal solution is found, that is, the value of the error function L(w) is minimized. The trained network is saved after the training ends.

[0103] (8) When a new image is input into the trained network, VGG19 extracts features from the image, and the evaluation network Q′ of DDQN estimates the Q value corresponding to each action based on the image, and classifies the image according to the action corresponding to the maximum Q value.

[0104] The applicability of the present invention to the existing tactile material database is analyzed below through simulation experiment data. The software used in the simulation experiment is PyCharm software and Tensorflow framework.

[0105] Simulation experiment:

[0106] The LTM108 database proposed by the Technical University of Munich in 2017 was selected to evaluate the tactile material classification method based on DDQN and generative adversarial networks. This study used materials such as marble, compressed wood, rubber, glass, grid, carpet, foam, and leather. The training set of each material has 10 pictures and 10 acceleration spectrograms. In the programming implementation, VGG19 is used as the generator structure of the generative adversarial network, and four convolution layers with 64, 128, 256, and 512 convolution kernels are used as the discriminator of the generative adversarial network. When extracting material features, a global average pooling layer is used to replace the three fully connected layers of VGG19, and the other layers of VGG19 remain unchanged. The parameters used by VGG19 are the parameters pre-trained in the public image dataset ImageNet, and VGG19 is used to extract material features. These features are then used along with the labels of the materials to which they belong. Define the state and action of DDQN, and then define the reward function. The batch size used for training each time in the experiment is 256, and the number of steps k of each iteration of DDQN is set to 200. The experiment uses the python framework tensorflow to build the evaluation network and target network of DDQN. The structure of the two networks is exactly the same. They are both composed of 5 fully connected layers. The number of neurons in these 5 fully connected layers is 1000, 700, 500, 200, and 9 respectively. The activation function of the fully connected layer is ReLU. Input the state and action of DDQN, define the reward function, and use the Adam stochastic gradient descent method to train the network until the error function reaches the minimum. After the training, save the trained network, and then use the trained network to classify the test set. The classification accuracy of the obtained test set is shown in Table 1.

[0107] Table 1

[0108]

[0109] Table 1 shows the classification results of DDQN on classified pictures, acceleration spectrograms generated by generative adversarial networks, and pictures and acceleration spectrograms generated by generative adversarial networks proposed in the method. It can be clearly seen that after adding the acceleration spectrogram, the classification accuracy is significantly improved. At the same time, this method does not need to collect tactile information in actual deployment, but only needs to collect visual information pictures to achieve the classification effect of visual-tactile fusion. Figure 4 Shows the acceleration spectrogram generated by the Generative Adversarial Network.

[0110] In the actual classification scenario of tactile materials, the method proposed by the present invention can achieve the classification effect of visual-tactile fusion without collecting tactile information of tactile materials. The use of generative adversarial network GAN avoids the complicated tactile information collection process, saves the expensive cost of purchasing tactile information collection equipment, and saves classification costs. DDQN has a strong representation ability and anti-overfitting ability, and its classification accuracy is significantly better than that of existing classifiers.

Claims

1. A tactile material classification method based on DDQN and generative adversarial network, characterized in that: The following steps are involved: (1) Use images of tactile materials and acceleration spectrograms to train the Generative Adversarial Network (GAN); (2) Send the acceleration spectrogram and tactile material images generated by the trained generative adversarial network GAN into the VGG19 network to extract features; (3) The material features extracted by VGG19 are regarded as the states of DDQN, and their labels are regarded as the actions of DDQN; (4) Setting the reward function of DDQN. The specific approach is: The reward function of DDQN is used to encourage the DDQN classifier to make correct decisions. t That is, when a material feature is input into DDQN, DDQN will make an action a t That is, a label is selected, and then a reward R is given according to whether the action is correct or not, that is, whether the correct label of the material feature is selected. If the correct action is made, a positive reward is given, otherwise, a negative reward is given. The label is Take the material characteristics as an example: The reward function is used by DDQN to calculate the action value; (5) The evaluation network of DDQN estimates the Q value and selects the action with the largest Q value to pass to the target network; (6) The target network of DDQN calculates the temporal difference error of the estimated Q value of the evaluation network, and the specific path is: Target Network The action a corresponding to the maximum Q value selected by the evaluation network Q′ * Used to calculate the time difference target TDtargety t ,y t The calculation formula is as follows: r represents the state of DDQN in the previous state s t The reward obtained after taking an action according to the unbalanced reward function, w (t-tmodc) Indicates that the parameters of the evaluation network Q' are copied to the target network every C steps To calculate the time difference error, the calculated time difference target is used to calculate the time difference error δ of the estimated Q value of the evaluation network Q′ t ; δ t =Q(s t ,a;w (t) )-y t The time difference error δ obtained at step t is t Will be used for parameter update; (7) Perform stochastic gradient descent optimization on the time difference error of the estimated Q value of the evaluation network until the global optimal solution is found. The training is completed and the trained network is saved. The specific way to reduce the estimated Q value error optimization parameters is: The target network The time difference error is obtained by subtracting the calculated time difference target from the Q value evaluated by the evaluation network Q′. The stochastic gradient descent method is used to reduce this error. First, the error function is defined as: Then use the stochastic gradient descent method to reduce the error and optimize the parameters. The steps are: where w (t+1) represents the parameters of the t+1th step. As the training progresses, the gradient descent is continuously used to optimize the parameters until the global optimal solution is found, that is, the value of the error function L(w) is minimized. The training ends and the trained network is saved. (8) When a new image is input into the trained network, the generative adversarial network generates an acceleration spectrum corresponding to the material based on the new image, and then inputs the new image and the generated acceleration spectrum into VGG19 to extract the material features. The evaluation network Q′ of DDQN estimates the Q value corresponding to each action based on the material features, and classifies the image according to the action corresponding to the maximum Q value.

2. The tactile material classification method based on DDQN and generative adversarial network according to claim 1, characterized in that: The specific method of training the generative adversarial network GAN in step (1) is: The category label of tactile material can be expressed as Where M represents M kinds of materials, and the image set of tactile materials is represented as in Indicates The image set of materials, the acceleration spectrum atlas of tactile materials is expressed as in Indicates The acceleration spectrum atlas of the material is collected at the same time. That is, the acceleration spectrum of the material is collected at the same time as the image information of the material. The collection process of the material is repeated multiple times, and the image and acceleration spectrum of the material are collected one by one each time. Use the image set of tactile materials and the acceleration spectrum atlas to train the generative adversarial network GAN. The generative adversarial network GAN includes a generator network Generator and a discriminator network Discriminator, referred to as G and D respectively. The generator network G inputs a noise output to generate a new image. The discriminator network D is used to judge whether the new image generated by the generator network G is true, that is, whether it meets the requirements. When training the generative adversarial network, the generator network G is first fixed, the optimal discriminator network D is solved, and then the optimal discriminator network D is used to solve the generator network G; V(G,D) is the loss function of the generative adversarial network GAN, p data is the data distribution of the acceleration spectrum y, p G is the data distribution obeyed by the data generated by the generator network G, D(y) is the probability that the acceleration spectrum graph generated by the generator G is the real acceleration spectrum graph, and the generator G obtained after training should conform to the following formula: D(y) should be as close to 1 as possible, and the generator G should have the smallest error possible, so that the error between the generated acceleration spectrum and the real acceleration spectrum is minimized; After a large amount of data training, when the generative adversarial network converges, the training ends, and the picture of the tactile material is input into the trained generative adversarial network GAN to obtain the generated acceleration spectrum map, which is used to extract features from the picture of the tactile material and the acceleration spectrum map generated by the generative adversarial network GAN.

3. The tactile material classification method based on DDQN and generative adversarial network according to claim 1, characterized in that: The specific approach of VGG19 feature extraction in step (2) is: VGG19 is used as the feature extraction network. VGG19 consists of 16 convolutional layers and 3 fully connected layers. A global average pooling layer is used to replace the three fully connected layers. The global average pooling layer outputs the average value of all pixels in each feature map obtained by convolution. These average values ​​are then regarded as feature vectors. The image set and the acceleration spectrum map generated by the generative adversarial network GAN are respectively input into VGG19 to obtain the features of the material image and the acceleration spectrum map generated by the generative adversarial network GAN. The image features and acceleration spectrum map features obtained by a single sampling of a material are then concatenated to obtain the corresponding material features. The material feature set is represented as in Indicates These extracted material features are used to construct the states and actions of DDQN in the next step.

4. The tactile material classification method based on DDQN and generative adversarial network according to claim 1, characterized in that: The specific way of constructing the DDQN state and action in step (3) is: Status: Before each iteration, randomly shuffle the material features extracted by VGG19 from the image set and acceleration spectrum atlas Then treat them as states {S1,…,S h }, h is the sum of the number of material features of all materials, and this method is used to convert image features into state sequences; Action: The goal of classification is to find the correct label for each image, which is defined as making the correct action. As the action space, DDQN is in state S t Then the state S t Transform to a new state S t+1 , t represents the tth step performed by DDQN.

5. The tactile material classification method based on DDQN and generative adversarial network according to claim 1, characterized in that: The specific way of evaluating the action of selecting the maximum Q value in step (5) is: In DDQN, a neural network-based evaluation network Q′ is established to estimate the action value Q; The Q value reflects the estimate of the future reward that DDQN will obtain if it takes action a in state s. Among them, w (t) represents the parameters of the evaluation network Q′ at step t, G t represents the reward for taking action a in state s plus the estimated future reward, represents the expectation of multiple estimated rewards, G t It represents the reward obtained by taking action a in state s plus the future reward, and uses the discount factor γ to reduce the impact of future rewards on the current action selection; G t =R t +γR t+1 +g 2 R t+2 +g 3 R t+3 +…+c k R t+k R t Represents the state s at step t t Make an action t Then the reward is obtained according to whether the action is right or wrong, R t+1 represents the reward of step t+1, and k represents the total number of steps of DDQN in one iteration; In addition to the evaluation network Q′, DDQN has another network target network Its construction is the same as the evaluation network Q′, and its parameters are copied from the evaluation network Q′ at regular intervals. t+1 , the evaluation network Q′ estimates the Q values ​​corresponding to all actions, and then selects the action corresponding to the maximum Q value: Among them, Q(s t+1 ,a;w (t) ) represents the Q value estimated by the evaluation network Q′, and the evaluation network Q′ selects the action a corresponding to the maximum Q value * After that, action a * The temporal difference error passed to the target network for computing the estimated Q value.

Images