A cross-modal visual-tactile data simulation generation model and model training device

Abstract

The present invention relates to a cross-modal visual-tactile data simulation generation model and a training device, wherein the cross-modal visual-tactile data simulation generation model comprises a fusion layer and a first generator; wherein the fusion layer is used to fuse force condition data with visual image data after dimension upgrading to obtain fused image data; the first generator is used for first attention-generation adversarial processing to obtain visual-tactile images; the present invention introduces an attention mechanism into the data generation process of the generative adversarial network, so that the model can avoid the extracted image features from deviating from the main features, and this synergistic effect makes the generated data more realistic in terms of texture contours, etc., solves some problems of the generated data of the traditional generation method, such as lack of realism and insufficient details, improves the quality of the generated data set, and is ultimately expected to be widely used in the fields of robot tactile sensor simulation, robot tactile rendering, dexterous hand tactile rendering, virtual reality tactile rendering, etc.

Description

Technical Field

[0001] The present invention relates to the field of data simulation, and in particular to the field of a cross-modal visual-tactile data simulation generation model and a model training device. Background Art

[0002] Tactile perception has been studied and proven to play a key role in the interaction between humans and the environment. Tactile sensing is widely used in research and application in various fields. How to better collect and apply tactile information has become a concern of many studies. Among them, visual tactile sensors have the advantages of easy manufacturing, high resolution, and no electrical interference problems. Compared with traditional single-mode tactile sensors, they can provide richer tactile information. Therefore, visual tactile data has been widely used in robotics research, dexterous hand tactile perception, three-dimensional image reconstruction, industrial production and other fields, playing an increasingly important role. However, there is a certain time cost in the collection of visual tactile data, and long-term repeated data collection work will cause a certain degree of sensor damage. At the same time, the characteristics of the sensor may change during long-term use, and calibration and maintenance are often required, otherwise the collected data may cause deviations in subsequent research results. At the same time, in robot design, robot tactile rendering, dexterous hand tactile rendering, virtual reality tactile and other fields, there is a great demand for tactile data simulation generation. Therefore, in order to save time and resources, it is necessary to build a generation model that can be separated from physical interaction and generate new visual tactile data.

[0003] To solve the above problems, the common method is to use virtual simulation reinforcement learning to collect visual and tactile data in a simulated environment and perform machine simulation training, and finally use it in reality, so as to make up for the blind spots in reality where data is difficult to capture and the machine is easy to wear. However, most of the current successes based on reinforcement learning are limited to abstract and simplified simulation environments, and the system cannot maintain high performance in more complex real-world environments. In recent years, deep learning networks such as generative adversarial networks and diffusion models have made great progress in data generation fields such as image generation. The visual and tactile data currently generated based on common generative models have certain limitations, which can easily interfere with the generation of tactile data that actually only focuses on texture, shape and gradient, and it is difficult to focus on capturing the tactile information of the target. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a cross-modal visual-tactile data simulation generation model and a model training device to solve the above-mentioned problems.

[0005] The present invention is implemented through the following technical solutions. On the one hand, the present invention provides a cross-modal visual-tactile data simulation generation model, which includes:

[0006] A fusion layer is used to fuse the input force condition data with the input visual image data after dimension upgrading to obtain fused image data; and

[0007] The first generator is used to perform a first attention-generation adversarial process on the fused image data to obtain a generated visual-tactile image.

[0008] Furthermore, the first generator comprises:

[0009] The convolution layer is used to perform convolution operation on the fused image data to obtain image feature parameters;

[0010] The residual layer is used to perform residual mapping calculation on the image feature parameters to obtain intermediate variables;

[0011] The attention module is used to calculate the attention weight matrix based on the intermediate variables, enhance the intermediate variables according to the attention weight matrix, and obtain the image feature representation;

[0012] The deconvolution layer deconvolves the image feature representation to generate a visual-tactile image.

[0013] Furthermore, the first generator is trained by the following method:

[0014] S10: preprocessing the training set, and then sequentially inputting the cross-modal visual-tactile data simulation generation model; wherein the training set includes an original visual image data set, an original visual-tactile image data set, and an original force condition data set; each original visual image corresponds to an original visual-tactile image and an original force condition;

[0015] The cross-modal visual-tactile data simulation generation model processes the input visual image data and force condition data to generate a visual-tactile image;

[0016] S20: subjecting the generated visual-tactile image output by the cross-modal visual-tactile data simulation generation model to a second attention-generation adversarial process to obtain a generated visual image;

[0017] S30: performing discrimination processing on the original visual-tactile image and the generated visual-tactile image to obtain a first discrimination output, and performing discrimination processing on the original visual image and the generated visual image to obtain a second discrimination output;

[0018] S40: Calculating a first adversarial loss according to the first discriminant output and the discriminant output of the generated visual-tactile image; calculating a second adversarial loss according to the second discriminant output and the discriminant output of the generated visual image; calculating a cycle consistency loss according to the original visual-tactile data and the fused data;

[0019] S50: The total loss is obtained by weighted summing of the first adversarial loss, the first adversarial loss and the cycle consistency loss. If the total loss is less than the convergence threshold, or the number of iterations exceeds the iteration threshold, the training is completed, otherwise, step S60 is executed;

[0020] S60: Calculate the gradient of the adversarial loss and the gradient of the cycle consistency loss respectively, and update the network parameters of the model according to the gradient back propagation.

[0021] Further, the first attention-generation adversarial processing, and / or the second attention-generation adversarial processing includes:

[0022] Perform convolution operation on the input image to obtain image feature parameters;

[0023] Perform residual mapping calculation on the image feature parameters to obtain intermediate variables;

[0024] Calculate the attention weight matrix according to the intermediate variables, enhance the intermediate variables according to the attention weight matrix, and obtain the image feature representation;

[0025] Deconvolution is performed on the image feature representation to generate a visual-tactile image.

[0026] Furthermore, the discrimination process includes:

[0027] Perform convolution operation on the input image to obtain image feature parameters;

[0028] Calculate the attention weight matrix according to the image feature parameters, enhance the input data according to the attention weight matrix, and obtain the image feature representation;

[0029] The image feature representation is convolutionally mapped to obtain the discriminative output.

[0030] Furthermore, the calculation formula of the attention weight matrix is:

[0031]

[0032] Where Q = XW Q ,K=XW K ,V=XW V , respectively represent the query, key, and value matrices in the attention mechanism, which are all obtained by linear transformation, W Q ,W K ,W V is the preset weight matrix, X is the parameter or variable entering the attention processing, and its data dimension size is d k .

[0033] Furthermore, the first adversarial loss calculation formula is:

[0034] Loss GAN1 =E[logD T (y)]+E[log(1-D T (G(x)))]

[0035] Where D T represents the first discrimination process, which discriminates the authenticity of the visual-tactile image, G represents the first attention-generation adversarial process; x represents the matrix obtained by fusion of the real visual image input in the data set and the force data after dimensionality upgrading, and y represents the real visual-tactile data input in the data set. G(x) represents the generated visual-tactile image;

[0036] The second adversarial loss calculation formula is:

[0037] Loss GAN2 =E[logD V (x)]+E[log(1-D V (F(y)))]

[0038] Where D V represents the second discrimination process, which discriminates the authenticity of the visual image, F represents the second attention-generation adversarial process; x represents the matrix obtained by fusion of the real visual image input in the data set and the force data after dimensionality upgrading, y represents the real visual and tactile data input in the data set. F(y) represents the generated visual image.

[0039] Furthermore, the calculation formula of the cycle consistency loss is as follows:

[0040] Loss cycle =E[‖G(F(y))-y‖]+E[‖F(G(x))-x‖]

[0041] Where G represents the first attention-generative adversarial processing, F represents the second attention-generative adversarial processing, E represents the expected value, ‖·‖ represents the norm (such as L1 norm, L2 norm), x represents the matrix obtained by fusion of the real visual image input in the dataset and the force data after dimensionality upgrading, and y represents the real tactile data input in the dataset.

[0042] Furthermore, the data preprocessing specifically includes:

[0043] S11: resampling the original visual image data and the original visual-tactile data;

[0044] S12: randomly selecting the resampled visual image data for data enhancement, and performing the same data enhancement operation on the visual-tactile data corresponding to the visual data;

[0045] S13: Batch normalize all data in the training set to obtain a preprocessed training set.

[0046] On the other hand, the present invention provides a training device for a cross-modal visual-tactile data simulation generation model, comprising:

[0047] A preprocessing unit is used to perform data preprocessing on the training set, and then sequentially input the cross-modal visual-tactile data simulation generation model; wherein the training set includes an original visual image data set, an original visual-tactile image data set, and an original force condition data set; each original visual image corresponds to an original visual-tactile image and an original force condition;

[0048] The second generator unit is used for subjecting the generated visual-tactile image output by the cross-modal visual-tactile data simulation generation model to a second attention-generation adversarial process to obtain a generated visual image;

[0049] Discriminator unit: used for discriminating the original visual-tactile image and the generated visual-tactile image to obtain a first discriminant output, and discriminating the original visual image and the generated visual image to obtain a second discriminant output;

[0050] A loss calculation unit: used to calculate a first adversarial loss according to the first discriminant output and the discriminant output for generating a visual-tactile image; calculate a second adversarial loss according to the second discriminant output and the discriminant output for generating a visual image; and calculate a cycle consistency loss according to the original visual-tactile data and the fused data;

[0051] Training end discriminant unit: the weighted sum of the first adversarial loss, the first adversarial loss and the cycle consistency loss is used to obtain the total loss. If the total loss is less than the convergence threshold, or the number of iterations exceeds the iteration threshold, the training is completed, otherwise the model is updated;

[0052] Model update unit: used to calculate the gradient of adversarial loss and the gradient of cycle consistency loss respectively, and update the network parameters of the model according to the gradient back propagation

[0053] The present invention is based on a generative adversarial network to train a cross-modal visual-tactile data simulation generation model, introduces an attention mechanism into the data generation process and discrimination process of the generative adversarial network, and participates in the training after fusing the visual image and force condition data in the training set, so that the generative adversarial network can focus on learning key information and avoid deviation from the main features during the learning process. This synergistic effect makes the generated data more realistic in terms of texture, shape and contour, and can solve some problems in traditional generation methods such as lack of realism and insufficient details of generated data, improve the quality of generated data sets, and provide a more reliable data basis for research and application in related visual-tactile fields. In addition, the present invention also performs batch normalization on features during the preprocessing process in the training set to maintain the stability of the training set data, accelerate the convergence speed of the deep network, and improve the training speed of the model.

[0054] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a structural diagram of a collection platform constructed in one embodiment of the present invention;

[0056] Figure 2 A schematic diagram of a cross-modal visual-tactile data simulation generation model provided by the present invention

[0057] Figure 3 The present invention provides an exemplary method for training Figure 2 A structural diagram of a training device for generating a model as shown;

[0058] Figure 4 For execution Figure 3 A flow chart of a training method for a cross-modal visual-tactile data simulation generation model of the training device shown;

[0059] Figure 5 This is an example diagram of tactile rendering for a robot's dexterous hand using visual tactile data generation in one embodiment of the present invention;

[0060] Figure 6 A radar chart showing the root mean square error between generated data and visual data and generated data and real data in one embodiment of the present invention;

[0061] Figure 7 A radar chart showing the structural similarity between generated data-visual data and generated data-real data in one embodiment of the present invention;

[0062] Figure 8 It is a radar chart of peak signal-to-noise ratio of generated data-visual data and generated data-real data in one embodiment of the present invention.

[0063] Fig. 9An example of generating visual-tactile data for several objects in one embodiment of the present invention is shown in FIG.

[0064] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. DETAILED DESCRIPTION

[0065] At present, the visual-tactile data generation method based on the generative model mainly focuses on the modeling of the mapping relationship from one data domain to another data domain, ignoring the subtle differences between visual perception and tactile perception. In this regard, the present invention provides a method for generating a cross-modal tactile data simulation model, which realizes the conversion between visual images and visual-tactile images based on a generative network, and introduces an attention mechanism in the training process of the generator and the discriminator, and finally completes the generation of the cross-modal tactile data simulation model, and the model can generate more accurate visual-tactile data based on visual image data.

[0066] Model training is inseparable from effective training set data. In order to collect training set data that meets the requirements, it is first necessary to design and build a collection platform for visual data and visual-tactile data according to the requirements. The present invention provides a specific collection platform structure as an exemplary illustration. The specific structure is as follows Figure 1 As shown, the acquisition platform includes a lifting and displacement platform (wherein the lifting and displacement platform is equipped with a six-axis force sensor to detect the contact force condition), a collection surface contact part, a visual tactile sensor, a stage, a control console and a computer. The lifting platform and the stage are fixed on the optical platform at a suitable distance. The visual tactile sensor is fixed on a stage with adjustable front, back, left and right displacements, and the horizontal position and horizontal angle of the sensor can be adjusted by adjusting the stage. The contact part and the displacement plate of the displacement platform are combined together through an assembly, and the distance between the contact part of the contact part and the visual tactile contact surface of the visual tactile sensor can be adjusted by controlling the displacement of the lifting platform through a computer-controlled control console.

[0067] Then, contact parts are designed as workpieces for data collection. In the present invention, simple graphics commonly seen in daily necessities are printed as contact body parts, and visual data and visual-tactile data of these graphic parts are collected for training and testing. The contact body parts are divided into contact parts and background parts by color. The contact part of the contact body parts in the training set is black, and the background part is white. The contact parts used in the test set are basically the same as those in the training set, but in order to distinguish them from the training data, the contact parts of the contact parts are printed using pink PLA material.

[0068] Before acquisition, adjust the focal length of the built-in camera of the tactile sensor according to the interaction between the tactile part and the sensor, so that when the tactile part is in the position where clear tactile data is formed, after removing the contact body and the support plate of the tactile sensor, the center of the tactile part is at the focus of the built-in camera of the sensor. Since the thickness of the tactile parts is the same, the camera focal length only needs to be adjusted once before acquisition.

[0069] First, collect the visual and tactile data of the training set. In order to more conveniently observe the relative position of the contact part and the sensor, remove the sensor contact plate part during the initial adjustment, turn on the control switch of the control console and the camera inside the sensor, and adjust the lifting plate to the highest position. Clean the contact parts to be collected with alcohol cotton pads and fix the contact parts on the lifting platform. Further, start data collection. Control the lifting platform to descend. When the contact parts completely cover the screen in the background part, adjust the horizontal position of the stage so that the contact parts appear completely in the shooting picture of the sensor camera. After the shooting picture of the contact parts is fully adjusted, install the sensor contact panel, and the lifting platform can be properly controlled to rise to leave enough space. After the sensor contact panel is installed, the horizontal position and rotation angle of the stage are adjusted by regular steps to make the collection reach the specified position and the specified posture angle, and the lifting platform is controlled to descend until the object contacts the contact surface to form a clear and complete visual and tactile data, and the visual and tactile data of the object at this time is captured, and the force data of the object currently paired with the six-axis force sensor is recorded at the same time. At this time, the paired force data and the visual and tactile image data of the object are obtained. Then, the lifting platform is raised, and the above steps are continued to complete the collection of visual and tactile data of the contact parts and the object force condition data set.

[0070] The preparation work for visual image acquisition is similar to that for visual-tactile data acquisition. After completing the preparation work, the horizontal position and rotation angle of the stage are adjusted in regular steps so that the acquisition reaches the specified position and specified posture angle consistent with the visual-tactile data acquisition process mentioned above, and the lifting platform is lowered until the object contacts the contact surface to form visual-tactile data with clear and complete contours. The position of the lifting platform at this time is adjusted to zero point to ensure that the collected visual image has corresponding visual-tactile data in real life. Then control the lifting platform to rise, remove the sensor contact plate part, and then reset the lifting platform to zero point, and take a picture of the visual image of the contact part at this time. At this time, the visual image data of the object paired with the visual-tactile data collected at the specified position can be obtained, and the paired acquisition of the three types of data can be realized. Then lift the lifting platform and continue to follow the above steps to complete the acquisition of the visual image of the contact part.

[0071] In addition, the model to be trained also needs to be built in advance, please refer to Figure 2 The model includes a fusion layer and a first generator, and the model is trained based on a model training device, see Figure 3The model training device includes a preprocessing unit 10, a second generator unit 20, a discriminator unit 30, a loss calculation unit 40, a model updating unit 50 and a training end judgment unit 60.

[0072] Based on the training set data collected in the above steps and the initialized model, start model training. Figure 3 , Figure 3 Flow chart of the generation method of the cross-modal tactile data simulation model provided by the present invention The generation method of the cross-modal tactile data simulation model of the present invention can output visual tactile data by inputting contact conditions such as the force conditions during contact and the visual image of the corresponding contact object, which solves the problems of time cost and sensor damage in visual tactile data collection, and the large demand for tactile data in the fields of robot tactile sensor simulation, robot tactile rendering, dexterous hand tactile rendering, virtual reality tactile rendering, etc., and improves the quality of visual tactile data generation through the combination of attention mechanism and generative adversarial network. Specifically, the training method of the cross-modal tactile data simulation generation model specifically includes:

[0073] S10: preprocessing the training set, and then sequentially inputting the cross-modal visual-tactile data simulation generation model; wherein the training set includes an original visual image data set, an original visual-tactile image data set, and an original force condition data set; each original visual image corresponds to an original visual-tactile image and an original force condition. Step S10 is performed by the preprocessing unit 10.

[0074] The specific operations of training set preprocessing are as follows:

[0075] S11: resampling the original visual image data and the original visual-tactile data;

[0076] S12: Randomly select the resampled visual image data for data enhancement, and perform the same data enhancement operation on the visual-tactile data corresponding to the visual data.

[0077] In order to expand the diversity of visual image data and simulate different sampling situations in reality, the present embodiment randomly selects visual images in the training data for data enhancement, and selects images in the training set with a probability of 50% for random flipping, scaling, cropping, and hue transformation operations; in addition, in order to ensure the consistency of visual-tactile data corresponding to the same object under different imaging conditions, the visual-tactile image data corresponding to the visual image data for data enhancement shall be subjected to the same random flipping, scaling, and cropping data enhancement operations without hue transformation.

[0078] S13: Batch normalize all data in the training set to obtain a preprocessed training set.

[0079] All data in the training set are batch normalized to scale the features to the interval (-1, 1) to maintain the numerical stability of the input distribution and accelerate the convergence of the deep network.

[0080] The cross-modal tactile data simulation model obtains visual image data and force condition data in the preprocessed training set. The fusion layer upgrades the force condition data and fuses it with the visual image data to obtain the fused image data, and the first generator processes the fused data through the first attention-generative adversarial process to obtain a generated visual-tactile image.

[0081] S20: subjecting the generated visual-tactile image to a second attention-generation adversarial process to obtain a generated visual image. Step S21 is performed by the second generator unit 20.

[0082] Specifically, the first attention-generation adversarial processing, and / or the second attention-generation adversarial processing includes: performing a convolution operation on the input image to obtain image feature parameters; performing residual mapping calculation on the image feature parameters to obtain intermediate variables; calculating the attention weight matrix based on the intermediate variables, enhancing the intermediate variables based on the attention weight matrix to obtain image feature representation; and deconvolving the image feature representation to obtain generated visual-tactile image data.

[0083] The generative model in the network architecture design of the present invention can select mainstream image generation models such as common generative adversarial networks, variational autoencoders or diffusion models. Therefore, the fusion method of the attention mechanism structure may also be different according to the different generative networks and the actual training effect. In this embodiment, a common cyclic generative adversarial network is selected; in the network training architecture, a convolution-residual network-deconvolution encoding and decoding structure is used as a generator. The convolution layer includes three identical convolution modules, each of which includes a convolution layer, an instance normalization layer and a Leaky ReLU activation function. After three convolution modules, the output training batch number, number of channels, channel height, and channel width change from [1,3,256,256] to [1,256,32,32]. The residual network part uses 9 layers of residual blocks, each layer of residual blocks includes two sets of reflection filling, convolution, instance normalization and ReLU activation function, and then adds to the previous layer of network. After the residual network, the characteristic parameters of the input image remain unchanged. The attention mechanism calculates the attention matrix through the convolution transformation of query, key, and value, and then obtains the attention weight matrix through the softmax function. Finally, the attention weight is applied to the value to obtain the enhanced feature representation without changing the dimension of the feature. Therefore, after the attention mechanism, the parameter dimension of the feature is still [1,256,32,32]. The deconvolution layer includes three identical deconvolution modules, each of which includes a deconvolution layer, an instance normalization layer, and a ReLU activation function. After three deconvolution modules, the parameter feature dimension of the output image changes back to [1,3,256,256].

[0084] S30 : performing discrimination processing on the original visual-tactile image and the generated visual-tactile image to obtain a first discrimination output, and performing discrimination processing on the original visual image and the generated visual image to obtain a second discrimination output. Step S30 is performed by the discriminator unit 30 .

[0085] Among them, the specific steps of the discriminant processing are: perform a convolution operation on the input image to obtain image feature parameters; calculate the attention weight matrix based on the image feature parameters, enhance the input data according to the attention weight matrix to obtain the image feature representation; perform convolution mapping on the image feature representation to obtain the discriminant output.

[0086] Specifically, the calculation formula of the attention weight matrix is:

[0087]

[0088] Where Q = XW Q ,K=XW K ,V=XW V, respectively represent the query, key, and value matrices in the attention mechanism, which are all obtained by linear transformation, W Q ,W K ,W V is the preset weight matrix, X is the parameter or variable entering the attention processing, and its data dimension size is d k .

[0089] PatchGAN with a receptive field of 70×70 and a convolution-attention mechanism-convolution structure is used as the discriminator unit. The first convolution layer includes a convolution layer and three convolution modules. Each convolution module includes a convolution layer, an instance normalization layer, and a Leaky ReLU activation function. More advanced features are gradually extracted through three convolution modules. The attention mechanism part includes the convolution transformation of query, key, and value. The softmax function obtains the attention weight matrix, and finally the attention weight is applied to the value to obtain the enhanced feature representation. The second convolution layer maps the feature map to a single-channel prediction map, and obtains a 30x30 matrix as the evaluation matrix. Finally, the mean of the output matrix is ​​taken as the discriminant output.

[0090] S40: Calculate a first adversarial loss based on the first discriminant output and the discriminant output for generating a visual-tactile image; calculate a second adversarial loss based on the second discriminant output and the discriminant output for generating a visual image; calculate a cycle consistency loss based on the original visual-tactile data and the fused data; step S40 is performed by the loss calculation unit 40.

[0091] The first adversarial loss calculation formula is:

[0092] Loss GAN1 =E[logD T (y)]+E[log(1-D T (G(x)))]

[0093] Where D T represents the first discrimination process, which discriminates the authenticity of the visual-tactile image, G represents the first attention-generation adversarial process; x represents the matrix obtained by fusion of the real visual image input in the data set and the force data after dimensionality upgrading, and y represents the real visual-tactile data input in the data set. G(x) represents the generated visual-tactile image;

[0094] The second adversarial loss calculation formula is:

[0095] Loss GAN2 =E[logD V (x)]+E[log(1-D V (F(y)))]

[0096] Where D V represents the second discrimination process, which discriminates the authenticity of the visual image, F represents the second attention-generation adversarial process; x represents the matrix obtained by fusion of the real visual image input in the data set and the force data after dimensionality upgrading, y represents the real visual and tactile data input in the data set. F(y) represents the generated visual image.

[0097] The calculation formula of cycle consistency loss is as follows:

[0098] Loss cycle =E[‖G(F(y))-y‖]+E[‖F(G(x))-x‖]

[0099] Where G represents the first attention-generative adversarial processing, F represents the second attention-generative adversarial processing, E represents the expected value, ‖·‖ represents the norm (such as L1 norm, L2 norm), x represents the matrix obtained by fusion of the real visual image input in the dataset and the force data after dimensionality upgrading, and y represents the real tactile data input in the dataset.

[0100] The adversarial loss is calculated to evaluate the difference between the generated tactile images and the real tactile images, and the quality of the generated tactile images is continuously optimized through adversarial training. The cycle consistency loss is calculated to ensure that the visual images converted by the two generators are as consistent as possible with the original visual images to maintain the cycle consistency of the data.

[0101] S50: The total loss is obtained by weighted summing of the first adversarial loss, the first adversarial loss and the cycle consistency loss. If the total loss is less than the convergence threshold, or the number of iterations exceeds the iteration threshold, the training is completed, otherwise step S60 is executed. Step S50 is executed by the training end determination unit 50.

[0102] The network parameters of the generated model are saved regularly. If the loss is less than the convergence threshold before the preset training rounds, the training is completed directly. After the preset training rounds, the training is completed. At this time, the network parameters with the best performance on the validation set are selected as the network for the final test and evaluation.

[0103] S60: Calculate the gradient of the adversarial loss and the gradient of the cycle consistency loss respectively, and update the network parameters of the model according to the gradient back propagation; step S60 is performed by the model updating unit 60.

[0104] The network parameters of the generative model are updated through back-propagation according to the gradient to improve the quality of the generated image.

[0105] The present invention is based on a generative adversarial network to train a cross-modal visual-tactile data simulation generation model, and introduces an attention mechanism in the data generation process and discrimination process of the generative adversarial network, and participates in the training after the visual image and force condition data in the training set are fused, so that the generative adversarial network can focus on learning key information and avoid deviating from the main features during the learning process. This synergistic effect makes the generated data more realistic in terms of texture, shape and contour, and can solve some problems in traditional generation methods such as lack of realism and insufficient details of generated data, improve the quality of generated data sets, and further solve the problem of large demand for tactile data in the fields of robot tactile sensor simulation, robot tactile rendering, dexterous hand tactile rendering, virtual reality tactile rendering, etc., and provide a more reliable data basis for research and application in related visual-tactile fields. In addition, the present invention also performs batch normalization on features during the preprocessing process in the training set to maintain the stability of the training set data, accelerate the convergence speed of the deep network, and improve the training speed of the model.

[0106] The model obtained after the above training can accept any visual image data and force data as input, and convert the visual image into a visual-tactile image through the trained generator model. The generator model converts the input visual image and force information into high-quality tactile features through a series of convolutional layers, attention mechanisms, residual networks, and forward propagation processes of deconvolutional layers, and finally outputs post-processed visual-tactile images, which can be used in scenarios such as visual-tactile sensor simulation and robot tactile sensing simulation. For further information, see Figure 5 The generated visual-tactile images can eventually be used in the fields of robot tactile rendering, virtual reality tactile rendering, etc. By inputting the visual image of a preset object and its corresponding contact force data, the corresponding visual-tactile data can be obtained, which can be mapped into actuation signals of a robot or virtual reality tactile device through relevant algorithms, thus achieving high-fidelity tactile rendering.

[0107] In addition, the model can be evaluated in the following way: a test set and a validation set are obtained again. The preparation work for the collection of visual-visual-tactile data pairs in the validation set and the test set is the same as the image collection of the training set. After the preparation work is completed, the lifting platform is lowered until the object contacts the contact surface to form visual-tactile data with a clear and complete outline and located in the center of the field of view. The position and angle of the contact part are controlled as constants to eliminate possible interference with the test results. The visual-tactile data at this time are photographed and the six-axis force data are recorded, and the position of the lifting platform at this time is adjusted to zero. The lifting platform is controlled to rise, the sensor contact plate part is removed, and the lifting platform is reset to zero. The visual image of the contact part at this time is photographed. The two images collected so far are a pair of matching data. After collecting several groups of images, the contact part is replaced. According to the above steps, the visual-visual-tactile data pair of the object is collected. In the test set, the visual image and visual-tactile data of the pink contact part need to be paired one by one. Among them, the force data and visual image data are used for model generation testing, and the corresponding visual-tactile data are used for result evaluation. To evaluate the effects of the shape and size of the contacting parts, paired images were collected in the control test set with other variables being the same.

[0108] The visual data and force data of the test set are input into the trained model to obtain the generated visual-tactile data, that is, the visual-force-visual-tactile data pairs collected for each contact part, and the obtained image groups are used to calculate the evaluation indicators of generated data-visual data and generated data-real data and take the average value to obtain the average value of the two groups of evaluation indicators corresponding to each part.

[0109] The closeness of the generated image to the real image is evaluated by calculating the root mean square error (RMSE), peak signal to noise ratio (PSNR), and structural similarity (SSIM) between the generated image and the real visual and tactile image. The lower the RMSE value, the closer the generated image is to the real image in the target domain, and the higher the model accuracy. The higher the PSNR value, the smaller the image distortion and the better the quality of the generated image; the higher the SSIM value, the higher the structural similarity between the generated image and the real image and the better the visual effect. Therefore, the lower the RMSE value, the higher the SSIM and PSNR values, the more realistic the generated image is and the better the image quality. The formulas for the three evaluation indicators are as follows.

[0110] The root mean square error is used to measure the difference between the predicted value and the actual value of the estimated object. Its calculation formula is as follows:

[0111]

[0112] in and g(x, y) are the predicted value and the actual value respectively, which are the expressions of the generated image and the real image in the present invention.

[0113] The structural similarity index method believes that if the perception of structural information changes, it means that the image has degraded. Its specific expression formula is:

[0114] SSIM(x,y)=[l(x,y)] α [c(x,y)] β ·[s(x,y)] γ

[0115] Where l is the brightness (used to compare brightness between two images), c is the contrast (used to distinguish the range between the lightest and darkest areas in the two images), and s is the structure coefficient (used to compare local brightness patterns between two images to find similarities and differences between the images). α, β, and γ are positive constants.

[0116] Peak signal-to-noise ratio is the most commonly used quality assessment technique for measuring the reconstruction quality of lossy image compression codecs. It calculates the ratio between the maximum possible signal powers in decibels. Its calculation formula is:

[0117] PSNR = 10log 10 (peakval 2 ) / RMSE 2

[0118] Where peakval is the maximum possible pixel value in the image data.

[0119] Through the above three parameters, the radar chart of the average evaluation index of the generated data-visual data and the generated data-real data of all contact parts is drawn to evaluate the effectiveness of the generated model. The results are as follows Figure 6-8 As shown, in this embodiment, the average evaluation index of the generated data-real data of each contact part is better than the average evaluation index of the generated data-visual data, which proves that the visual-tactile data generation method has a considerable beneficial effect. Fig. 9 An example of generating tactile data for a common contact object according to this embodiment is shown.

[0120] The generation model in the network architecture design of the present invention can select common mainstream image generation models such as generative adversarial networks, variational autoencoders or diffusion models. The fusion method of the attention mechanism structure of this solution can also be arbitrarily selected according to the difference of the generation network and the actual training effect. The network input of this solution is not only limited to the visual image and force information in the embodiment, but can be one or more of visual images, angle information, force information, position information, etc.

[0121] The present invention is not only applicable to the generation of visual-tactile image data, but also can be used to generate a variety of tactile data, including capacitive tactile data, piezoelectric tactile data, piezoresistive tactile data, triboelectric tactile data, electromagnetic tactile data and optical fiber tactile data. The wide adaptability of this invention makes it of great significance in various tactile data applications, and can meet the data generation requirements of different types of tactile sensors. The data acquisition platform of the present invention can be composed of any motion platform, image acquisition system, tactile sensor (including but not limited to visual-tactile), mechanical sensor, etc.

[0122] The present invention may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0123] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A cross-modal visual-tactile data simulation generation model, characterized in that: include: The fusion layer is used to upgrade the input force condition data and fuse it with the input visual image data to obtain fused image data; and A first generator, used for performing a first attention-generation adversarial process on the fused image data to obtain a generated visual-tactile image; It includes: a convolution layer, which is used to perform a convolution operation on the fused image data to obtain image feature parameters; The residual layer is used to perform residual mapping calculation on the image feature parameters to obtain intermediate variables; The attention module is used to calculate the attention weight matrix based on the intermediate variables, enhance the intermediate variables according to the attention weight matrix, and obtain the image feature representation; Deconvolution layer, deconvolution of image feature representation to generate visual and tactile images; The first generator is trained by the following method: S10: preprocessing the training set, and then sequentially inputting the cross-modal visual-tactile data simulation generation model; wherein the training set includes an original visual image data set, an original visual-tactile image data set, and an original force condition data set; each original visual image corresponds to an original visual-tactile image and an original force condition; The cross-modal visual-tactile data simulation generation model processes the input visual image data and force condition data to generate a visual-tactile image; S20: subjecting the generated visual-tactile image output by the cross-modal visual-tactile data simulation generation model to a second attention-generation adversarial process to obtain a generated visual image; S30: performing discrimination processing on the original visual-tactile image and the generated visual-tactile image to obtain a first discrimination output, and performing discrimination processing on the original visual image and the generated visual image to obtain a second discrimination output; S40: Calculating a first adversarial loss according to the first discriminant output and the discriminant output of the generated visual-tactile image; calculating a second adversarial loss according to the second discriminant output and the discriminant output of the generated visual image; calculating a cycle consistency loss according to the original visual-tactile data and the fused data; S50: The total loss is obtained by weighted summing of the first adversarial loss, the first adversarial loss and the cycle consistency loss. If the total loss is less than the convergence threshold, or the number of iterations exceeds the iteration threshold, the training is completed, otherwise, step S60 is executed; S60: Calculate the gradient of the adversarial loss and the gradient of the cycle consistency loss respectively, and update the network parameters of the model according to the gradient back propagation.

2. A cross-modal visual-tactile data simulation generation model according to claim 1, characterized in that: The second attention-generation adversarial processing in step S20 includes: Perform convolution operation on the input image to obtain image feature parameters; Perform residual mapping calculation on the image feature parameters to obtain intermediate variables; Calculate the attention weight matrix according to the intermediate variables, enhance the intermediate variables according to the attention weight matrix, and obtain the image feature representation; Deconvolution is performed on the image feature representation to generate a visual-tactile image.

3. A cross-modal visual-tactile data simulation generation model according to claim 2, characterized in that: The first discrimination process or the second discrimination process of step S30 includes: Perform convolution operation on the input image to obtain image feature parameters; Calculate the attention weight matrix according to the image feature parameters, enhance the input data according to the attention weight matrix, and obtain the image feature representation; The image feature representation is convolutionally mapped to obtain the discriminative output.

4. A cross-modal visual-tactile data simulation generation model according to claim 3, characterized in that: The calculation formula of the attention weight matrix is: Where Q = XW Q ,K=XW K ,V=XW V , respectively represent the query, key, and value matrices in the attention mechanism, which are all obtained by linear transformation, W Q ,W K ,W V is the preset weight matrix, X is the parameter or variable entering the attention processing, and its data dimension size is d r .

5. A cross-modal visual-tactile data simulation generation model according to claim 4, characterized in that: The first adversarial loss calculation formula is: Loss GAN1 =E[logD T (y)]+E[log(1-D T (G(x)))] Where D T represents the first discrimination process, which discriminates the authenticity of the visual-tactile image, E represents the expected value; G represents the first attention-generation adversarial process; x represents the matrix obtained by fusion of the real visual image input in the data set and the force data after dimensionality upgrading, y represents the real visual-tactile data input in the data set; G(x) represents the generated visual-tactile image; The second adversarial loss calculation formula is: Loss GAN2 =E[logD V (x)]+E[log(1-D V (F(y)))] Where D V represents the second discrimination processing, which discriminates the authenticity of the visual image, E represents the expected value; F represents the second attention-generative adversarial processing; x represents the matrix obtained by fusion of the real visual image input in the data set and the force data after dimensionality upgrading, y represents the real visual and tactile data input in the data set; F(y) represents the generated visual image.

6. A cross-modal visual-tactile data simulation generation model according to claim 5, characterized in that: The calculation formula of the cycle consistency loss is as follows: Loss cycle =E[‖G(F(y))-y‖]+E[‖F(G(x))-x‖] Where G represents the first attention-generative adversarial processing, F represents the second attention-generative adversarial processing, E represents the expected value, ‖·‖ represents the norm, x represents the matrix obtained by fusion of the real visual image input in the dataset and the force data after dimensionality upgrading, and y represents the real tactile data input in the dataset.

7. A cross-modal visual-tactile data simulation generation model according to claim 6, characterized in that: The preprocessing of the training set specifically includes: S11: resampling the original visual image data and the original visual-tactile data to obtain resampled visual image data and resampled visual-tactile data; S12: randomly selecting the resampled visual image data for data enhancement, and performing the same data enhancement operation on the resampled visual-tactile data corresponding to the resampled visual image data; S13: Batch normalize all data in the training set to obtain a preprocessed training set.

8. A training device for the cross-modal visual-tactile data simulation generation model according to claim 1, characterized in that: include: A preprocessing unit is used to perform data preprocessing on the training set, and then sequentially input the cross-modal visual-tactile data simulation generation model; wherein the training set includes an original visual image data set, an original visual-tactile image data set, and an original force condition data set; each original visual image corresponds to an original visual-tactile image and an original force condition; The second generator unit is used for subjecting the generated visual-tactile image output by the cross-modal visual-tactile data simulation generation model to a second attention-generation adversarial process to obtain a generated visual image; Discriminator unit: used for discriminating the original visual-tactile image and the generated visual-tactile image to obtain a first discriminant output, and discriminating the original visual image and the generated visual image to obtain a second discriminant output; A loss calculation unit: used to calculate a first adversarial loss according to the first discriminant output and the discriminant output for generating a visual-tactile image; calculate a second adversarial loss according to the second discriminant output and the discriminant output for generating a visual image; and calculate a cycle consistency loss according to the original visual-tactile data and the fused data; Training end discriminant unit: the weighted sum of the first adversarial loss, the first adversarial loss and the cycle consistency loss is used to obtain the total loss. If the total loss is less than the convergence threshold, or the number of iterations exceeds the iteration threshold, the training is completed, otherwise the model is updated; Model update unit: used to calculate the gradient of adversarial loss and the gradient of cycle consistency loss respectively, and update the network parameters of the model according to the gradient back propagation.

Images