A single real sample based haptic image dataset expansion method and device

Abstract

This invention belongs to the technical field of network training and discloses a method and device for expanding a tactile image dataset based on a single real sample. The steps are as follows: (1) Constructing a simulation scene corresponding to the real environment, and then constructing a simulation tactile dataset; (2) Obtaining a single real tactile image sample and using the single real tactile image sample to train a multi-scale generative adversarial network model; (3) After completing the training of the generator at each scale in the multi-scale generative adversarial network model, fixing the weights of the generator and adding adversarial perturbations to the discriminator; (4) Inputting the simulation tactile dataset into the trained multi-scale adversarial generative neural network to obtain a preliminarily expanded dataset, and performing various image transformations on the preliminarily expanded dataset to achieve further expansion. This invention can construct a large-scale tactile dataset based on a single real tactile sample collected by an arbitrary high-resolution tactile sensor, reducing the collection cost of a large-scale tactile dataset.

Description

Technical Field

[0001] This invention belongs to the technical field of network training, and more specifically, relates to a method and device for expanding a tactile image dataset based on a single real sample. Background Technology

[0002] Tactile image data is tactile information collected by high-resolution vision-based tactile sensors, which can reflect the distribution of forces and surface texture on the contact surface in an image format. However, since these high-resolution tactile sensors mostly acquire tactile images by capturing the deformation of the contact gel surface with a camera, and the gel surface is very fragile, this poses a significant challenge to constructing large-scale tactile image datasets. In recent years, with the development of deep learning technology, many object classification models, vision-tactile reconstruction models, and robot grasping models based on tactile image data rely on large-scale datasets. Therefore, augmenting tactile image data based on a single real tactile sample can help improve the performance of deep learning tasks based on tactile image data.

[0003] Chinese Patent 201811270865.4 proposes a method for expanding the training dataset for feasible regions of mobile robots. This invention uses a binocular camera to acquire a set of original images containing terrain features. The images are then standardized to facilitate subsequent dataset expansion and transformation. Based on this, image samples obtained under different weather and shooting conditions are expanded through image synthesis, the addition of rain and snow imprints, and simulated infrared imaging, transforming the data to obtain image samples that might only be acquired under special circumstances. This invention effectively expands the dataset coverage, provides more training samples for subsequent machine learning, significantly shortens the dataset construction cycle, reduces the cost of dataset construction, and helps improve the training effect of mobile robots, increasing the robot's recognition rate of feasible regions under various special conditions.

[0004] Chinese Patent 201911056394.1 discloses a method, training method, and related apparatus for expanding training images, relating to the field of pedestrian recognition in machine learning. The method includes: acquiring multiple images of a pedestrian to be converted; the multiple images to be converted include at least two pedestrian color data; acquiring a color data generation model based on the multiple images to be converted; the color data generation model is a model trained on at least two pedestrian color data using a generative adversarial network; inputting the multiple images to be converted into the color data generation model to acquire multiple target color images; and using the images to be converted and the multiple target color images as a color training set. Using the data expansion method provided in this application, more specific color data can be acquired based on real color data, thereby meeting the training data requirements for color recognition model training and improving the accuracy of color recognition.

[0005] In summary, no data augmentation method has been proposed for tactile image data, and data augmentation methods for single samples are also urgently needed. Therefore, there is an urgent need in this field to propose a tactile image dataset augmentation method based on a single real sample in order to achieve high-performance deep learning tasks based on a single real tactile sample. Summary of the Invention

[0006] To address the above-mentioned deficiencies or improvement needs of existing technologies, this invention provides a method and device for expanding tactile image datasets based on a single real sample. It can construct a large-scale tactile dataset based on a single real tactile sample collected by any high-resolution tactile sensor, which can greatly reduce the collection cost of large-scale tactile datasets. At the same time, due to the introduction of adversarial training methods, the generated tactile images are of high quality.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for augmenting a tactile image dataset based on a single real sample is provided, the method comprising the following steps:

[0008] (1) Construct simulation scenarios that correspond to the real environment, and then construct a simulation tactile dataset;

[0009] (2) Obtain a single real tactile image sample and use the single real tactile image sample to train the multi-scale generative adversarial network model; during the training process, the first scale in the multi-scale generative adversarial network model is directly input with a noise map of the corresponding size; for each subsequent scale, a real tactile image sample scaled to the corresponding scale and a noise map of the same scale are input, and then the weight of the discriminator at that scale is fixed, and adversarial perturbation is added to the generator.

[0010] (3) After training the generator at each scale in the multi-scale generative adversarial network model, fix the weights of the generator and add adversarial perturbations to the discriminator.

[0011] (4) Input the simulated tactile dataset into the trained multi-scale adversarial generative neural network to obtain the initially expanded dataset, and then further expand the initially expanded dataset.

[0012] Furthermore, the generator and discriminator structures of the multi-scale generative adversarial network are identical, consisting of five fully connected layers. The input scale of the fully connected layer in each scale generative adversarial network model is the same as that of the previous scale. .

[0013] Furthermore, the network loss during generator training is:

[0014]

[0015] in It is a generator for generative adversarial networks. It is the first generative adversarial network. Discriminators at various scales, To generate adversarial networks An adversarial loss at each scale is used to compute the original image. Fake samples generated by the generator The differences between them; It is the reconstruction loss, used to ensure that each generator can generate the same result as the previous one. Similar generated images.

[0016] Furthermore, in the discriminator WGAN-GP loss is selected. Using L2 loss; adding adversarial perturbations to the discriminator includes the following sub-steps:

[0017] First, based on the generated sample images Computational Generator Networks Training loss And calculate the gradient of the training loss. ;

[0018] Then, the generated sample image is input at the corresponding scale. Add perturbation The disturbance satisfies The specific calculation method is as follows:

[0019]

[0020] in .

[0021] Then, T perturbations are added to the input image at the corresponding scale. The perturbation is calculated as follows:

[0022]

[0023] in: , .

[0024] Furthermore, affine transformations, rotational transformations, and elastic transformations are used to further expand the initially expanded dataset.

[0025] Furthermore, the steps for further expanding the initially expanded dataset using elastic transformation are as follows:

[0026] First, based on each pixel in the tactile image Construct two random values, called and Its value ranges from -1 to 1, representing the values ​​in... and The distance moved in a direction;

[0027] Then, construct a Gaussian kernel of size n*n. As the kernel function for convolution operations, its mean is 0 and its standard deviation is... ;

[0028] Then, based on and The offset image is then convolved to obtain the final deformed image.

[0029] Furthermore, The value is achieved by minimizing the following combined objective function:

[0030]

[0031] in This is an image structure similarity function used to calculate the image similarity between a single real tactile image and its corresponding simulated tactile image; This is the root mean square error measurement function, used to calculate the image difference between a single real tactile image and its corresponding simulated tactile image.

[0032] Furthermore, adding adversarial perturbations to the generator involves the following sub-steps:

[0033] First, based on real sample images Computational Generator Networks Training loss And calculate the gradient of the training loss. ;

[0034] Then, input real sample images at the corresponding scale. Add perturbation The disturbance satisfies The specific calculation method is as follows:

[0035]

[0036] in .

[0037] Then, T perturbations are added to the input image at the corresponding scale. The perturbation is calculated as follows:

[0038]

[0039] in: , .

[0040] The present invention also provides a tactile image dataset augmentation system based on a single real sample. The system includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the tactile image dataset augmentation method based on a single real sample as described above.

[0041] The present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the tactile image dataset augmentation method based on a single real sample as described above.

[0042] In summary, compared with the prior art, the tactile image dataset augmentation method and device based on a single real sample provided by the present invention have the following beneficial effects:

[0043] 1. This invention inputs a simulated tactile dataset into a trained multi-scale adversarial generative neural network to obtain a preliminarily expanded dataset. This allows for the construction of a large-scale tactile dataset based on a single real tactile sample collected by an arbitrary high-resolution tactile sensor, which can greatly reduce the collection cost of large-scale tactile datasets. At the same time, due to the introduction of adversarial training, the generated tactile images are of high quality.

[0044] 2. This invention first constructs a simulation scene corresponding to the real environment, uses a simulation engine to collect tactile data in the simulation environment in advance, and then uses a trained generative adversarial network to perform data style transfer. The final tactile data has a corresponding geometric and physical relationship with the installation position of the tactile sensor and the position of the contact object in the real scene, which is very suitable for the field of robot learning.

[0045] 3. The tactile image data augmentation method proposed in this invention uses affine transformation, rotation transformation, and elastic transformation for tactile image data to further augment the dataset, resulting in a large-scale tactile dataset with richer features. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a method for expanding a tactile image dataset based on a single real sample, provided by the present invention.

[0047] Figure 2 yes Figure 1 The flowchart of the transformation from a simulated tactile dataset to a realistic tactile image dataset is shown in the method for augmenting tactile image datasets based on a single real sample. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0049] This invention provides a method for augmenting a tactile image dataset based on a single real sample. The method mainly includes the following steps:

[0050] Step 1: Construct a simulation scene that corresponds to the real environment, and then construct a simulation tactile dataset.

[0051] Specifically, a simulation scene corresponding to the real environment is constructed, and a simulated tactile dataset is built based on a tactile data simulation engine. In this embodiment, the tactile data simulation engine used is a simulation engine such as Tacto / Taxim that quickly generates high-resolution tactile image data in multiple scenes.

[0052] Step 2: Obtain a single real tactile image sample and use it to train the multi-scale generative adversarial network model. During training, a noise map of the corresponding size is directly input into the first scale of the multi-scale generative adversarial network model. For each subsequent scale, a real tactile image sample scaled to the corresponding scale and a noise map of the same scale are input. Then, the weights of the discriminator at that scale are fixed, and adversarial perturbations are added to the generator to enhance the diversity of the images generated by the model.

[0053] The random seeds for the noise maps at different scales input to the generative adversarial network (GAN) model are fixed, which effectively ensures the network's convergence. The generator and discriminator structures of the multi-scale GAN are identical, consisting of five fully connected layers. The input scale of each fully connected layer in the GAN model is the same as the previous scale. .

[0054] The network loss during generator training is:

[0055]

[0056] in It is a generator for generative adversarial networks. It is the first generative adversarial network. Discriminators at various scales, To generate adversarial networks An adversarial loss at each scale is used to compute the original image. Fake samples generated by the generator The differences between them; It is the reconstruction loss, used to ensure that each generator can generate the same result as the previous one. Similar generated images.

[0057] WGAN-GP loss is selected. L2 loss is used.

[0058] Step 3: After training the generator at each scale in the multi-scale generative adversarial network model, fix the weights of the generator and add adversarial perturbations to the discriminator to enhance the discriminator's ability to distinguish whether a sample is real or not, thereby improving the quality of the images generated by the model.

[0059] In the discriminator WGAN-GP loss is selected. Using L2 loss; adding adversarial perturbations to the discriminator includes the following sub-steps:

[0060] First, based on the generated sample images Computational Generator Networks Training loss And calculate the gradient of the training loss. ;

[0061] Then, the generated sample image is input at the corresponding scale. Add perturbation The disturbance satisfies The specific calculation method is as follows:

[0062]

[0063] in .

[0064] Then, T perturbations are added to the input image at the corresponding scale. The perturbation is calculated as follows:

[0065]

[0066] in: , .

[0067] Step four involves inputting the simulated tactile dataset into the trained multi-scale adversarial generative neural network to obtain a preliminarily expanded dataset, which is then further expanded.

[0068] Specifically, after obtaining a multi-scale generative adversarial network capable of fully extracting features from individual training samples, the noise maps at the generator inputs at different scales are replaced with images from the scaled simulated tactile dataset to obtain a pre-expanded dataset. Subsequently, the pre-expanded dataset is further expanded using affine transformation, rotation transformation, and elastic transformation for tactile image data.

[0069] Elastic transformation is particularly suitable for enhancing tactile data, as it can fully simulate the deformation of a sensor when it comes into contact with a moving object in tactile simulation. Its specific implementation includes the following sub-steps:

[0070] First, based on each pixel in the tactile image Construct two random values, called and Its value ranges from -1 to 1, representing the values ​​in... and The distance moved in a direction;

[0071] Then, construct a Gaussian kernel of size n*n. As the kernel function for convolution operations, its mean is 0 and its standard deviation is... ;

[0072] Then, based on and The offset image is then convolved to obtain the final deformed image.

[0073] The choice of augmentation dataset has a significant impact: small Values ​​that result in distortion are similar to randomly shifting each pixel in an image, while large values... The value will produce results that are significantly different from the original image. Therefore, optimization Values ​​are crucial to ensuring the effectiveness of data augmentation, which can be achieved by minimizing the following combined objective function:

[0074]

[0075] in This is an image structure similarity function used to calculate the image similarity between a single real tactile image and its corresponding simulated tactile image; This is the root mean square error measurement function, used to calculate the image difference between a single real tactile image and its corresponding simulated tactile image.

[0076] The present invention will be further described in detail below with reference to specific embodiments.

[0077] Please see Figure 1 and Figure 2 The tactile image dataset augmentation method based on a single real sample provided in this embodiment mainly includes the following steps:

[0078] S1, construct a simulation scene corresponding to the real environment, and build a simulation tactile dataset based on the tactile data simulation engine.

[0079] In this project, Gazabo was used to construct a simulation scene corresponding to a real-world environment, specifically a traditional robot grasping scenario. The tactile data simulation engine used was Taxim.

[0080] S2, acquires a single real tactile image sample for training a multi-scale generative adversarial network model.

[0081] The Gelsight Mini sensor was used to acquire single real-world tactile image samples with a resolution of 512*512. The random seed for the noise maps at different scales input to the generative adversarial network (GAN) model was 2023. The multi-scale GAN consists of six scales where the generator and discriminator have identical structures, each composed of five fully connected layers. The input scale of the fully connected layers in each scale GAN model is the same as the previous scale. The input resolution of the first scale generative adversarial network model is 64*64.

[0082] S3, during network training, in the multi-scale generative adversarial network model, the first scale is directly input with a noise map of the corresponding size. For each subsequent scale, the input is a real tactile image sample scaled to the corresponding scale and a noise map of the same scale. Then, the weights of the discriminator at that scale are fixed, and adversarial perturbations are added to the generator to enhance the diversity of the images generated by the model.

[0083] The network loss for training each scale generator is:

[0084]

[0085] in It is a generator for generative adversarial networks. It is the first generative adversarial network. Discriminators at various scales, To generate adversarial networks An adversarial loss at each scale is used to compute the original image. Fake samples generated by the generator The differences between them It is the reconstruction loss, used to ensure that each generator can generate the same result as the previous one. Similar generated images.

[0086] Among them, the network loss during training of each scale generator WGAN-GP loss is selected. L2 loss is used.

[0087] The method of adding adversarial perturbations to the generator includes the following sub-steps:

[0088] First, based on real sample images Computational Generator Networks Training loss And calculate the gradient of the training loss. ;

[0089] Then, input real sample images at the corresponding scale. Add perturbation The disturbance satisfies The specific calculation method is as follows:

[0090]

[0091] in .

[0092] Then, T perturbations are added to the input image at the corresponding scale. The perturbation is calculated as follows:

[0093]

[0094] in: , .

[0095] S4. After training the generators at each scale in the multi-scale generative adversarial network model, fix the generator weights and add adversarial perturbations to the discriminator to enhance the discriminator's ability to distinguish whether a sample is real, thereby improving the quality of the images generated by the model.

[0096] The network loss during discriminator training is:

[0097]

[0098] Among them, in the discriminator WGAN-GP loss is selected. L2 loss is used.

[0099] Adding adversarial perturbations to the discriminator includes the following sub-steps:

[0100] First, based on the generated sample images Computational Generator Networks Training loss And calculate the gradient of the training loss. ;

[0101] Then, the sampled images input at the corresponding scale Add perturbation The disturbance satisfies The specific calculation method is as follows:

[0102]

[0103] in .

[0104] Then, T perturbations are added to the input image at the corresponding scale. The perturbation is calculated as follows:

[0105]

[0106] in: , .

[0107] S5 yields a multi-scale generative adversarial network that fully extracts features from individual training samples. Images from the scaled simulated tactile dataset are used to replace the noise maps at different scales in the original input network, resulting in a preliminarily expanded dataset.

[0108] S6 further expands the initially expanded dataset using methods such as affine transformation, rotation transformation, and elastic transformation for tactile image data.

[0109] The implementation of elastic transformation includes the following sub-steps:

[0110] First, based on each pixel in the tactile image Construct two random values, called and Its value ranges from -1 to 1, representing the values ​​in... and The distance moved in a direction;

[0111] Then, construct a Gaussian kernel of size n*n (n=5). As the kernel function for convolution operations, its mean is 0 and its standard deviation is... ;

[0112] Then, based on and The offset image is then convolved to obtain the final deformed image.

[0113] The choice of augmentation dataset has a significant impact: small Values ​​that result in distortion are similar to randomly shifting each pixel in an image, while large values... The value will produce results that are significantly different from the original image. Therefore, optimization Ensuring the effectiveness of data augmentation is crucial, and this can be achieved by minimizing the following combined objective function:

[0114]

[0115] in This is an image structure similarity function used to calculate the image similarity between a single real tactile image and its corresponding simulated tactile image. This is the root mean square error measurement function, used to calculate the image difference between a single real tactile image and its corresponding simulated tactile image.

[0116] The present invention also provides a tactile image dataset augmentation system based on a single real sample. The system includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the tactile image dataset augmentation method based on a single real sample as described above.

[0117] The present invention also provides a computer-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the tactile image dataset augmentation method based on a single real sample as described above.

[0118] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for augmenting a tactile image dataset based on a single real sample, characterized in that, The method includes the following steps: (1) Construct a simulation scene corresponding to the real environment, and build a simulation tactile dataset in this simulation scene; (2) Obtain a single real tactile image sample and use the single real tactile image sample to train the multi-scale generative adversarial network model; during the training process, the first scale in the multi-scale generative adversarial network model is directly input with a noise map of the corresponding size; for each subsequent scale, a real tactile image sample scaled to the corresponding scale and a noise map of the same scale are input, and then the weight of the discriminator at that scale is fixed, and adversarial perturbation is added to the generator. (3) After training the generator at each scale in the multi-scale generative adversarial network model, fix the weights of the generator and add adversarial perturbations to the discriminator. (4) Input the simulated tactile dataset into the trained multi-scale adversarial generative neural network to obtain the initially expanded dataset, and then further expand the initially expanded dataset. The steps for further expanding the initially augmented dataset using elastic transformation are as follows: First, based on each pixel in the tactile image Construct two random values, called and Its value ranges from -1 to 1, representing the values ​​in... and The distance moved in a direction; Then, construct a Gaussian kernel of size n*n. As the kernel function for convolution operations, its mean is 0 and its standard deviation is... ; Then, based on and The offset image is then convolved to obtain the final deformed image.

2. The method for augmenting a tactile image dataset based on a single real sample as described in claim 1, characterized in that: The generator and discriminator of a multi-scale generative adversarial network (GAN) have the same structure, consisting of five fully connected layers. The input scale of the fully connected layer in each scale GAN model is the same as that of the previous scale. .

3. The method for augmenting a tactile image dataset based on a single real sample as described in claim 2, characterized in that: The network loss during generator training is: in It is a generator for generative adversarial networks. It is the first generative adversarial network. Discriminators at various scales, To generate adversarial networks An adversarial loss at each scale is used to compute the original image. Fake samples generated by the generator The differences between them; It is the reconstruction loss, used to ensure that each generator can generate the same result as the previous one. Similar generated images.

4. The method for augmenting a tactile image dataset based on a single real sample as described in claim 1, characterized in that: In the discriminator WGAN-GP loss is selected. Using L2 loss; adding adversarial perturbations to the discriminator includes the following sub-steps: First, based on the generated sample images Computational Generator Networks Training loss And calculate the gradient of the training loss. ; Then, the generated sample image is input at the corresponding scale. Add perturbation The disturbance satisfies The specific calculation method is as follows: in ; Then, T perturbations are added to the input image at the corresponding scale. The perturbation is calculated as follows: in: , .

5. The method for augmenting a tactile image dataset based on a single real sample as described in any one of claims 1-4, characterized in that: Affine transformations or rotation transformations can be used to further expand the initially expanded dataset.

6. The method for augmenting a tactile image dataset based on a single real sample as described in claim 1, characterized in that: The value is achieved by minimizing the following combined objective function: in This is an image structure similarity function used to calculate the image similarity between a single real tactile image and its corresponding simulated tactile image; This is the root mean square error measurement function, used to calculate the image difference between a single real tactile image and its corresponding simulated tactile image.

7. The method for augmenting a tactile image dataset based on a single real sample as described in any one of claims 1-4, characterized in that: Adding adversarial perturbations to the generator involves the following sub-steps: First, based on real sample images Computational Generator Networks Training loss And calculate the gradient of the training loss. ; Then, input real sample images at the corresponding scale. Add perturbation The disturbance satisfies The specific calculation method is as follows: in ; Then, T perturbations are added to the input image at the corresponding scale. The perturbation is calculated as follows: in: , .

8. A tactile image dataset augmentation system based on a single real sample, characterized in that: The system includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it performs the tactile image dataset augmentation method based on a single real sample as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the tactile image dataset augmentation method based on a single real sample as described in any one of claims 1-7.

Images