Enhanced simulation image generation method based on deep learning

Abstract

The invention relates to an enhanced simulation image generation method based on deep learning, belongs to the field of computer vision and graphics, and solves the problems that in an image translation method in the prior art, image conversion errors inevitably occur due to the fact that paired data sets need to be adopted for training, and semantic tags are difficult to obtain during supervised training. According to the method, an unsupervised image translation method based on the generative adversarial network is adopted, a non-paired data set is used for training, the image generation process is constrained through comparative learning and HOG features of the image, and the target of simulation image reality is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and graphics, and relates to a method for generating enhanced simulated images based on deep learning. The method adopts an adversarial generative network using contrastive learning and image HOG features, and applies the network to an image translation task from a simulated image to a real image. Background Art

[0002] How to enhance the realism of simulated images has long been a key research area in computer graphics. With the advancement of technologies such as physically based light transport simulation, scientific representation of material appearance, and photogrammetric modeling, computer rendering technology has made significant progress. However, even the most advanced rendering technology cannot fully reach a level comparable to real images. Significant differences between simulated and real images still exist.

[0003] In recent years, some new technologies based on deep learning and generative adversarial networks (GANs) have emerged in the field of computer vision and graphics, bypassing the physical modeling process of geometric layout, material appearance, and light transmission, and realizing domain conversion from simulated images to real images. Generative adversarial networks consist of two parts: the generator is responsible for generating images, while the discriminator is used to ensure the consistency of the generated images with the target images. Traditional generative adversarial networks usually rely on a large amount of paired training data, but in practical applications, it is often difficult to obtain paired data. In addition, in the process of converting simulated images to real images, conversion errors may occur (such as mistakenly converting the sky into vegetation). To solve this problem, some methods introduce semantic labels of simulated images for supervision, which initially reduces the occurrence of conversion errors. However, semantic annotation of all simulated images is time-consuming and laborious, so the effect is not ideal. Summary of the invention

[0004] In view of the above problems, the present invention discloses an enhanced simulation image generation method based on deep learning; the problem that the image translation method in the prior art needs to adopt paired data sets for training, inevitably leads to image conversion errors and difficulty in obtaining semantic labels during supervised training is solved. The method of the present invention adopts an unsupervised image translation method based on a generative adversarial network, uses an unpaired data set for training, and constrains the image generation process through contrastive learning and the HOG features of the image, thereby achieving the goal of realistic simulation images.

[0005] The present invention provides a method for generating an enhanced simulation image based on deep learning, and the specific steps are as follows:

[0006] Step 1: obtaining a simulated image and a real image of the object, and an unpaired data set based on the simulated image and the real image;

[0007] Step 2: Build an adversarial network architecture including a generator and a discriminator;

[0008] Step 3: Construct a PatchNCE loss function based on contrastive learning;

[0009] Step 4: Use the encoder in step 2 to extract multi-level features of the input image; use the PatchNCE loss function in step 3 on the extracted multi-level features of the input image to obtain the PatchNCE loss;

[0010] Step 5: Use the unpaired dataset obtained in step 1, PatchNCE loss, GAN loss adversarial loss and HOG loss to train the adversarial network architecture;

[0011] Step 6: Based on the adversarial network architecture obtained through training in step 5, input the simulated image to obtain the realistic generated image.

[0012] Optionally, an input image layer, an initial convolutional layer with reflection padding, 2 downsampling layers, and 9 residual blocks constitute the encoder.

[0013] Optionally, the initial convolutional layer of the injection padding includes a 2d convolutional layer, a normalization layer, and a ReLU activation function.

[0014] Optionally, the convolution kernel size of the 2D convolution layer of the initial convolution layer with reflection padding is 7*7; the stride is 1; and the padding operation is 0.

[0015] Optionally, each downsampling layer of the encoder includes a convolution layer, a normalization layer, a ReLU activation function, and a double downsampling layer.

[0016] Optionally, the convolution kernel size of the downsampling layer is 3*3, the stride is 1, and the padding operation is 1.

[0017] Optionally, two upsampling layers and an output layer including a reflection padding layer constitute the decoder of the generator.

[0018] Optionally, the expression of the PatchNCE loss function is:

[0019]

[0020] Where l(.) represents the cross entropy loss; v represents the query vector of the query patch of the input image, v + Represents the positive sample vector of the input image, v - Represents the set of negative sample vectors of the input image; τ is the temperature coefficient, which is used to control the softening degree of the Softmax function in the similarity calculation; Represents the nth negative sample vector, n = 1, 2, ..., N, N represents the total number of negative samples of small patches of the input image from other positions in the input image.

[0021] Optionally, the PatchNCE loss function of step 3 is used on the multi-level features of the extracted input image to obtain the expression of PatchNCE loss:

[0022]

[0023] Where G and H are the generator and multi-layer perceptron MLP respectively; l(.) represents the PatchNCE loss function in step 3; and They represent the output and input features of the sth patch in the input image extracted by the lth extraction layer, respectively. represents the negative sample features of the other patches in the S patches extracted by the lth extraction layer, excluding the sth patch; X represents the set of input images in the input domain; x represents the xth input image; E x～X Represents the input mathematical expectation obtained by averaging the loss of each input image in the input image set X of the input domain.

[0024] Optionally, based on the set of generated images and the set of real images, an evaluation index of the generated images is obtained.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] 1. The present invention adopts unpaired data sets for training, thus avoiding the reliance on a large amount of paired data and improving the practicality and applicability of the method.

[0027] 2. The present invention utilizes contrastive learning and uses the InfoNCE loss function to maximize the mutual information between the input image and the generated output image. By mapping the corresponding regions of the input image and the output image to similar regions in the feature space and distinguishing the irrelevant parts, the input image and the output image are kept consistent in semantic structure, which avoids the limitation of paired data and improves the quality of the generated image.

[0028] 3. The contrastive learning framework of the present invention extracts multi-level features in the encoder and applies contrastive learning on these feature layers, ensuring that the input and output images maintain content consistency at different scales, effectively improving the quality of the generated images.

[0029] 4. The present invention uses the HOG features of the input image and the output image to constrain the generator of the generative adversarial network. Since the HOG feature is robust to style changes and can represent the spatial information of the image to a certain extent, using HOG loss to constrain the generative adversarial network helps the generator to keep the spatial structure of the input image and the output image consistent, reducing the possibility of translation errors.

[0030] 5. The method of the present invention is an unsupervised image translation method using unpaired data, which minimizes the occurrence of conversion errors to better achieve the goal of realizing simulated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the overall structure of the generative adversarial network constructed for the present invention.

[0032] Figure 2 A schematic diagram of the structure of the generator in the generative adversarial network constructed for the present invention.

[0033] Figure 3 A schematic diagram of the structure of the discriminator in the generative adversarial network constructed for the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only for illustration and are not intended to limit the present application.

[0035] One embodiment of the present invention, see Figure 1-3 , discloses a method for generating enhanced simulation images based on deep learning, the specific steps are as follows:

[0036] Step 1: obtain a simulated image and a real image of the object, and an unpaired data set based on the simulated image and the real image.

[0037] Among them, the unpaired dataset is used for simulation image realization.

[0038] Preferably, Autodesk Maya is used to perform texture mapping on the object being photographed; and Unity3D is used for rendering.

[0039] Specifically, texture mapping of the object includes selecting a suitable material for the object, and manually drawing and adjusting the texture for each surface of the object to obtain a texture mapping image;

[0040] Furthermore, through UV unfolding, we make sure that the texture can accurately cover the surface of the object; and use Maya's mapping tools to adjust every detail of the texture until the desired effect is achieved.

[0041] Specifically, texture includes color, texture scale, glossiness, roughness, transparency, etc.

[0042] Furthermore, the texture map image is rendered. Specifically, during rendering, lighting is configured, including a main light source and an auxiliary light source, and the reflection parameters and shadow effects of the texture map image are adjusted to obtain a simulated image.

[0043] Furthermore, in order to further enhance the rendering effect, the simulated image is subjected to a post-processing stack, specifically including tone mapping, anti-aliasing and depth of field effects.

[0044] Preferably, Unity is used to post-process the simulation images.

[0045] Preferably, when acquiring the real image of the object, real images of the same object and scene from multiple sources and shooting conditions are acquired to improve accuracy.

[0046] Preferably, when obtaining an unpaired data set, the simulated images and real images are sorted separately to filter out images of unqualified quality, so that when a machine learning method such as a generative adversarial network (GAN) is used for style transfer in the future, the mapping relationship between the simulated images and the real images can be learned through the algorithm.

[0047] Step 2: Build an adversarial network architecture including a generator and a discriminator;

[0048] Preferably, see Figure 1 , the generator includes an encoder and a decoder; the encoder is used to extract the features of the input image to obtain a feature map, and the decoder is used to convert the feature map into an output image of the target domain.

[0049] Specifically, see Figure 2 , the encoder of the generator includes a ResNet structure; the input image layer of the ResNet structure, the initial convolution layer of reflection padding, 2 downsampling layers and 9 residual blocks constitute the encoder of the generator.

[0050] When used, the input image first passes through an initial convolution layer with reflection padding; the convolution kernel size of the initial convolution layer with reflection padding is 7*7, the step size is 1, the padding operation is 0, the 2D convolution layer, the normalization layer and a ReLU activation function;

[0051] This is followed by two downsampling layers, each of which contains a 2D convolution layer with a convolution kernel size of 3*3, a stride of 1, a padding operation of 1, a normalization layer, a ReLU activation function, and a double downsampling layer;

[0052] The features processed by the two downsampling layers then pass through 9 residual blocks. Each residual block contains two 2D convolutional layers with a convolution kernel size of 3*3, a step size of 1, and a padding of 1, as well as the corresponding normalization layer and ReLU activation function. Each residual block retains the jump connection of the input features.

[0053] Furthermore, the two upsampling layers of the ResNet structure and the output layer including the reflection padding layer constitute the decoder of the generator.

[0054] When used, the features output by the encoder of the generator enter the two upsampling layers of the decoder. Each upsampling layer contains a double upsampling layer, a 2d convolution layer with a convolution kernel size of 3*3, a step size of 1, a padding operation of 1, a normalization layer and a ReLU activation function.

[0055] Finally, there is the output layer including the reflection padding layer. The convolution kernel size of the output layer including the reflection padding layer is 7*7, the step size is 1, the padding operation is 0, and the 2D convolution layer and the Tanh activation function.

[0056] Specifically, see Figure 3 The discriminator includes a discriminator input layer, a first intermediate layer, a second intermediate layer and a discriminator output layer; wherein the discriminator input layer includes a 2D convolution layer with a convolution kernel size of 4*4, a step size of 1, and a padding of 1, a LeakyReLU activation function and a two-fold downsampling layer; each intermediate layer includes a 2D convolution layer with a convolution kernel size of 4*4, a step size of 1, and a padding of 1, a normalization layer, a LeakyReLU activation function and a two-fold downsampling layer; the output layer includes a 2D convolution layer with a convolution kernel size of 4*4, a step size of 1, and a padding of 1, a normalization layer, a LeakyReLU activation function and a 2D convolution layer with a convolution kernel size of 4*4, a step size of 1, and a padding of 1, a normalization layer, a LeakyReLU activation function and a 2D convolution layer with a convolution kernel size of 4*4, a step size of 1, and a padding of 1.

[0057] Step 3: Construct a PatchNCE loss function based on contrastive learning;

[0058] Specifically, the PatchNCE loss function is used to find the input image patch that best matches the corresponding position in the input image of each output image patch; the optimal match between the output image patch and the input image patch in the feature space is obtained through contrastive learning.

[0059] Furthermore, the search is a back-propagation search.

[0060] Furthermore, the PatchNCE loss function constructs an (N+1) classification problem, where N represents the total number of negative samples of the input image patch from other positions in the input image, and 1 represents a positive sample of the output image patch at the corresponding position of the input image.

[0061] It can be understood that positive samples are samples that are similar or correlated with the target sample in the feature space, the same image after transformation or cropping, or images that belong to the same class as the target image. The goal is to enable the model to map the feature vectors of positive samples and target samples closer during training. Negative samples are samples that have no similarity or correlation with the target sample in the feature space. They usually come from different categories or different instances. During training, the role of negative samples is to help the model learn to distinguish features between different categories or instances.

[0062] For example, if the target sample is a picture of a cat, then a cropped version of the picture or other pictures of cats can be used as positive samples. If the target sample is a picture of a cat, then any picture of a dog or other images not related to cats can be used as negative samples.

[0063] The PatchNCE loss function first extracts the features of the input image and the output image through the encoder, and then maps the features of the input image and the output image to a shared embedding space through a two-layer multi-layer perceptron MLP; finally, the similarity between the query patch and all candidate patches (including positive and negative samples) is calculated through the Softmax function, and the cross-entropy loss function is used to train the generative adversarial network architecture to achieve the distinction between positive and negative samples in the candidate patches, so as to constrain the generator so that the image generated by the generator is consistent with the input image in terms of image content.

[0064] The query patch is the contrast region extracted from the output image and is the object to be contrasted by the model in the feature space. The candidate patches come from the input image and include positive samples and negative samples. The positive samples correspond to the patches of the input image at the same position of the query patch in the output image; the negative samples are patches of other regions randomly selected from the input image, which are unrelated to the query patch.

[0065] The present invention calculates the similarity between the query patch and all candidate patches (including positive and negative samples) by using the Softmax function to quantify the similarity between the query patch and the candidate patches. A high similarity means that the query patch and a candidate patch have similar features in the feature space, while a low similarity means that they are not similar.

[0066] For the matching of positive samples: the query patch and its corresponding positive sample patch (from the same position of the input image) should have a high similarity because they represent different perspectives of the same information. In this way, the model learns to cluster positive samples together as much as possible in the feature space.

[0067] For the distinction of negative samples: Negative sample patches are selected from other areas of the output image, which are irrelevant to the query patch in terms of content. By ensuring that the similarity between the query patch and these negative samples is low, the model enhances the learning of positive samples while suppressing the interference of irrelevant content.

[0068] Furthermore, the expression of PatchNCE loss function is:

[0069]

[0070] Where l(.) represents the cross entropy loss; v represents the query vector of the query patch of the input image, v + Represents the positive sample vector of the input image, v - Represents the set of negative sample vectors of the input image; τ is the temperature coefficient, which is used to control the softening degree of the Softmax function in the similarity calculation; Represents the nth negative sample vector, n = 1, 2, ..., N, N represents the total number of negative samples of small patches of the input image from other positions in the input image.

[0071] It can be understood that the PatchNCE loss function represents the probability of selecting a positive sample instead of a negative sample.

[0072] Step 4: Use the encoder of the generator in step 2 to extract multi-level features of the input image; use the PatchNCE loss function in step 3 to obtain the PatchNCE loss based on the extracted multi-level features of the input image.

[0073] The present invention uses non-corresponding areas in the input image as negative samples by selecting internal negative samples, thereby enhancing the effect of contrastive learning; based on the output PatchNCE loss, the present invention can obtain an output image with consistent content with the input image at all scales.

[0074] Specifically, the five extraction layers of the generator encoder in step 2 are used to extract features from the input image. The five extraction layers include an input image layer, a first downsampling layer and a second downsampling layer, a first residual block and a fifth residual block. The corresponding receptive field sizes of the five layers are 1×1, 9×9, 15×15, 35×35 and 99×99.

[0075] Furthermore, for each layer of extracted features, 256 random positions are sampled and a 2-layer multi-layer perceptron MLP is applied to obtain the 256-dimensional final features of the input image with respect to the query vector, positive sample vector and negative sample vector of the query patch. The PatchNCE loss is obtained based on the 256-dimensional final features and the PatchNCE loss function in step 3, and the expression is:

[0076]

[0077] Where G and H are the generator and multi-layer perceptron MLP respectively; l(.) represents the PatchNCE loss function in step 3; and They represent the output and input features of the sth patch in the input image extracted by the lth extraction layer, respectively. represents the negative sample features of the other patches in the S patches extracted by the lth extraction layer, excluding the sth patch; X represents the set of input images in the input domain; x represents the xth input image; E x～X Represents the input mathematical expectation obtained by averaging the loss of each input image in the input image set X of the input domain.

[0078] It is understandable that the final 256-dimensional feature Respectively with v,v in the PatchNCE loss function in step 3 + ,v - correspond.

[0079] Step 5: Use the unpaired dataset obtained in step 1, PatchNCE loss, GAN loss adversarial loss and HOG loss to train the adversarial network architecture;

[0080] The present invention uses GAN loss adversarial loss to ensure that the output image is visually consistent with the target domain image, and uses PatchNCE loss to maximize the mutual information between the input and output regions, thereby achieving content consistency. In addition, the present invention also introduces HOG loss in the process of training the generator of the adversarial network architecture. This loss calculates the L2 loss between the HOG features of the input image of the generator and the generated image. By constraining the consistency of the HOG features of the input image and the output image of the generator, the spatial structure and semantic content of the generated image are basically consistent with the input image, thereby reducing the situation of image translation errors. Through training, the generator gradually learns the optimal mapping, so that the input image can be accurately converted into the target domain image.

[0081] Specifically, the GAN loss adversarial loss is used to train the adversarial network architecture to ensure that the output image is visually consistent with the target domain image. The expression of the GAN loss adversarial loss is:

[0082] L GAN (G,D,X,Y)=E y～Y logD(y)+E x～X log(1-D(G(x)))

[0083] Among them, G represents the generator, D represents the discriminator, X represents the set of input images of the input domain, Y represents the set of output images of the output domain, x and y represent the xth input image of the input domain and the yth output image of the output domain respectively, E y～Y Represents the output mathematical expectation obtained by averaging the loss of each output image in the output image set Y of the output domain.

[0084] Furthermore, PatchNCE loss is used to maximize the mutual information between input and output regions, thus achieving content consistency.

[0085] In addition, the present invention also introduces HOG loss in the process of training the generator. This loss calculates the L2 loss between the HOG features of the input image of the generator and the generated image. By constraining the consistency of the HOG features of the input image and the output image of the generator, the spatial structure and semantic content of the generated image are basically consistent with the input image, thereby reducing the situation of image translation errors. The expression of HOG loss is:

[0086] HOG loss =|kH o -H θ || 2

[0087] Among them, H o Represents the HOG feature of the input image, H g Represents the HOG features of the output image.

[0088] For example, the number of training iterations is set to 200, the batch size is 16, the initial learning rate is 0.0002, the cosine annealing method is used to reduce the learning rate during the training process, and the Adam optimizer is used for training to obtain the optimal network parameter weights. Through training, the generator gradually learns the optimal mapping, so that the input image can be accurately converted to the target domain image.

[0089] Step 6: Based on the adversarial network architecture obtained through training in step 5, a simulated image is input to obtain a realistic generated image; based on the set of generated images and the set of real images, an evaluation index of the generated image is obtained.

[0090] The present invention compares the set of generated images with the set of real images and calculates the evaluation index, so as to quantitatively measure the effect of the simulation image realization.

[0091] Specifically, Inception Score (IS), Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) are used as quantitative indicators to evaluate the realism effect of generated images.

[0092] The expression of the evaluation index Inception Score (IS) is:

[0093]

[0094] Among them, KL represents KL divergence, P represents the total number of generated images, and p(o|x i ) represents the i-th generated image x i The conditional label distribution of , p(o) is the marginal label distribution; o represents the category of the image.

[0095] The expression of the evaluation index Fréchet Inception Distance (FID) is:

[0096]

[0097] Among them, μ r With μ g represents the mean of the real image features and the generated image features, ∑ r and∑ g Represents the covariance of the real image features and the generated image features.

[0098] The expression of the evaluation index Kernel Inception Distance (KID) is:

[0099]

[0100] in, represents the feature map of the corresponding image, M represents the total number of actual images; m k represents the kth actual image; m j represents the jth actual image; p k Describe the i-th generated image; x j represents the jth generated image.

[0101] The above applications are only some implementation methods of the present application. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A method for generating enhanced simulation images based on deep learning, characterized in that: The specific steps are as follows: Step 1: obtaining a simulated image and a real image of the object, and an unpaired data set based on the simulated image and the real image; Step 2: Build an adversarial network architecture including a generator and a discriminator; Step 3: Construct a PatchNCE loss function based on contrastive learning; Step 4: Use the encoder in step 2 to extract multi-level features of the input image; use the PatchNCE loss function in step 3 on the extracted multi-level features of the input image to obtain the PatchNCE loss; Step 5: Use the unpaired dataset obtained in step 1, PatchNCE loss, GAN loss adversarial loss and HOG loss to train the adversarial network architecture; Step 6: Based on the adversarial network architecture obtained through training in step 5, input the simulated image to obtain the realistic generated image.

2. The enhanced simulation image generation method according to claim 1, characterized in that: The encoder consists of an input image layer, an initial convolutional layer with reflection padding, 2 downsampling layers, and 9 residual blocks.

3. The enhanced simulation image generation method according to claim 2, characterized in that: The initial convolutional layer with injection padding includes a 2d convolutional layer, a normalization layer, and a ReLU activation function.

4. The enhanced simulation image generation method according to claim 3, characterized in that: The convolution kernel size of the 2D convolution layer of the initial convolution layer with reflection padding is 7*7; the step size is 1; and the padding operation is 0.

5. The enhanced simulation image generation method according to claim 2, characterized in that: Each downsampling layer of the encoder consists of a convolutional layer, a normalization layer, a ReLU activation function, and a double downsampling layer.

6. The enhanced simulation image generation method according to claim 5, characterized in that: The convolution kernel size of the downsampling layer is 3*3, the stride is 1, and the padding operation is 1.

7. The enhanced simulation image generation method according to claim 5, characterized in that: Two upsampling layers and an output layer including a reflective padding layer constitute the decoder of the generator.

8. The enhanced simulation image generation method according to any one of claims 1 to 7, characterized in that: The expression of PatchNCE loss function is: Where l(.) represents the cross entropy loss; v represents the query vector of the query patch of the input image, v + Represents the positive sample vector of the input image, v - Represents the set of negative sample vectors of the input image; τ is the temperature coefficient, which is used to control the softening degree of the Softmax function in the similarity calculation; represents the nth negative sample vector, n = 1, 2, ..., N, N represents the total number of negative samples of small patches of the input image from other positions in the input image.

9. The enhanced simulation image generation method according to claim 8, characterized in that: The expression of PatchNCE loss is obtained by using the PatchNCE loss function in step 3 on the multi-level features extracted from the input image: Where G and H are the generator and multi-layer perceptron MLP respectively; l(.) represents the PatchNCE loss function in step 3; and They represent the output and input features of the sth patch in the input image extracted by the lth extraction layer, respectively. represents the negative sample features of the other patches in the S patches extracted by the lth extraction layer, excluding the sth patch; x represents the set of input images in the input domain; x represents the xth input image; E x～X Represents the input mathematical expectation obtained by averaging the loss of each input image in the input image set X of the input domain.

10. The enhanced simulation image generation method according to claim 1, characterized in that: Based on the set of generated images and the set of real images, an evaluation index of the generated images is obtained.

Images