A 3D Image Generation Method and Device Based on Generative Adversarial Network

Abstract

The present invention provides a 3D image generation method and device based on a generative adversarial network. The method includes: inputting multi-view visual 2D images captured from different perspectives and a random variable into a trained generative adversarial network to generate a 2D image; inputting the generated 2D image into a 3DINN network to obtain the 3D structure corresponding to the 2D image; splicing the 3D structure and the 2D image and inputting the result into a trained 3D convolutional neural network to output predicted values for the three channels of R, G, and B. The present invention uses a generative adversarial network to generate 3D images based on multi-view visual 2D images captured from different perspectives of the same scene, which can effectively reduce noise and solve problems in the prior art such as the generated 3D scenes being relatively single and incomplete when generating 3D images based on monocular visual 2D images.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial neural networks, and particularly relates to a 3D image generation method and device based on a generative adversarial network. Background Art

[0002] A deep neural network is a framework for deep learning and is widely used in image segmentation, classification, and detection. It is a neural network with at least one hidden layer. A deep neural network can model complex non-linear systems, and a deep neural network with multiple hidden layers can provide a higher abstraction layer for the model, thus improving the model's expressive ability. A generative adversarial network (GAN, Generative Adversarial Networks) is an important method in the field of image generation. A GAN consists of a generative network and a discriminative network. The generative network generates images, and the discriminative network judges the authenticity of the images.

[0003] At present, most of the methods for generating images are based on GAN. The general steps of such methods are as follows: input a latent N-dimensional vector into the generative network to output the generated image; input an image into the discriminative network to judge the authenticity of the image. However, the quality and authenticity of the generated image may not be high due to the large difference between the distribution of the latent N-dimensional vector and the distribution of the real image. Multi-view vision refers to obtaining images of the same scene from multiple angles. The 2D images generated in this way have a complete scene and low noise. Most of the methods for generating 3D scenes from 2D images are based on adversarial generative networks for monocular images. The general steps of such methods are as follows: collect a 2D image, construct a 3D generative network and a discriminative network, use the generative network to map a latent vector to a 3D scene, and the discriminative network outputs the confidence that the scene is a real scene. The 3D scenes generated by this method are relatively single and incomplete, and the scenes lose many of their own features. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a 3D image generation method and device based on a generative adversarial network.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions.

[0006] In a first aspect, the present invention provides a 3D image generation method based on a generative adversarial network, including the following steps:

[0007] Input multi-view vision 2D images taken from different perspectives and a random variable into a trained generative adversarial network to generate a 2D image;

[0008] Input the generated 2D image into a 3DINN network to obtain the 3D structure corresponding to the 2D image;

[0009] Input the spliced 3D structure and 2D image into a trained 3D convolutional neural network to output the predicted values of the R, G, and B channels.

[0010] Furthermore, the method further includes: splicing multi-view visual 2D images and stretching them into a one-dimensional vector, then splicing it with an N-dimensional random vector to obtain an N + 1-dimensional vector, and inputting the N + 1-dimensional vector into a generative adversarial network.

[0011] Furthermore, the 3D convolutional neural network includes 5 basic networks, a fully connected layer, and an output layer containing 3 neurons. The basic network is a lightweight ResNet18.

[0012] Furthermore, the method further includes: caching the generated 2D images. When generating 2D images next time, first check whether there is an image similar to the input multi-view visual 2D image in the cache. If so, obtain the similar image from the cache, splice it with the input multi-view visual 2D image, and then input it into the generative adversarial network.

[0013] Furthermore, the method for checking whether two images are similar includes: representing the two images to be checked as one-dimensional vectors respectively, and obtaining the similarity of the two images by calculating the cosine distance between the two vectors; if the similarity is greater than a set threshold, the images to be checked are similar; otherwise, they are not similar. The formula for calculating the cosine distance between two vectors X and Y is as follows:

[0014]

[0015] In the formula, cos(X,Y) is the cosine distance between X and Y, X=(x1,x2,…,x n ), Y=(y1,y2,…,y n ).

[0016] In a second aspect, the present invention provides a 3D image generation device based on a generative adversarial network, including:

[0017] A 2D image generation module, configured to input multi-view visual 2D images taken from different perspectives and a random variable into a trained generative adversarial network to generate a 2D image;

[0018] A 3D structure generation module, configured to input the generated 2D image into a 3DINN network to obtain the 3D structure corresponding to the 2D image;

[0019] A 3D image generation module, configured to input the spliced 3D structure and 2D image into a trained 3D convolutional neural network to output the predicted values of the R, G, and B channels.

[0020] Further, the device further includes a vector splicing module, which is used to splice and stretch the multi-view visual 2D images into a one-dimensional vector, and then splice it with an N-dimensional random vector to obtain an N+1-dimensional vector, and input the N+1-dimensional vector into the generative adversarial network.

[0021] Further, the 3D convolutional neural network includes 5 basic networks, a fully connected layer, and an output layer containing 3 neurons. The basic network is a lightweight ResNet18.

[0022] Further, the device further includes a 2D image caching module, which is used to cache the generated 2D images. When generating 2D images next time, first check whether there is an image similar to the input multi-view visual 2D image in the cached 2D images. If so, obtain the similar image from the cache, splice it with the input multi-view visual 2D image, and then input it into the generative adversarial network.

[0023] Furthermore, the method for checking whether two images are similar includes: representing the two images to be checked as a one-dimensional vector respectively, and obtaining the similarity of the two images by calculating the cosine distance between the two vectors; if the similarity is greater than the set threshold, the images to be checked are similar; otherwise, they are not similar; the formula for calculating the cosine distance between two vectors X and Y is as follows:

[0024]

[0025] where cos(X,Y) is the cosine distance between X and Y, X=(x1,x2,…,x n ), Y=(y1,y2,…,y n ).

[0026] Compared with the prior art, the present invention has the following beneficial effects.

[0027] In the present invention, by inputting multi-view visual 2D images taken from different perspectives and a random variable into a trained generative adversarial network to generate a 2D image, inputting the generated 2D image into a 3D INN network to obtain the 3D structure corresponding to the 2D image, and splicing the 3D structure with the 2D image and then inputting it into a trained 3D convolutional neural network to output the predicted values of the R, G, and B channels, the automatic generation of 3D images is realized. The present invention uses a generative adversarial network to generate 3D images based on multi-view visual 2D images taken from different perspectives of the same scene, which can effectively reduce noise and solve the problems of the existing technology such as the generated 3D scenes being relatively single and incomplete when generating 3D images based on monocular vision (2D images taken from the same angle). BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1Flowchart of a 3D image generation method based on a generative adversarial network according to an embodiment of the present invention.

[0029] Figure 2 Schematic structural diagram of a generative adversarial network.

[0030] Figure 3 Schematic diagram of a 3D image generated from a multi-view visual 2D image.

[0031] Figure 4 Block diagram of a 3D image generation device based on a generative adversarial network according to an embodiment of the present invention. Detailed implementation manners

[0032] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described below with reference to the accompanying drawings and specific implementation manners. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts shall fall within the scope of protection of the present invention.

[0033] Figure 1 Flowchart of a 3D image generation method based on a generative adversarial network according to an embodiment of the present invention, including the following steps:

[0034] Step 101: Input multi-view visual 2D images taken from different perspectives and a random variable into a trained generative adversarial network to generate a 2D image;

[0035] Step 102: Input the generated 2D image into a 3DINN network to obtain the 3D structure corresponding to the 2D image;

[0036] Step 103: After splicing the 3D structure and the 2D image, input them into a trained 3D convolutional neural network, and output the predicted values of the R, G, and B channels.

[0037] In this embodiment, step 101 is mainly used to generate a 2D image by using a generative adversarial network. The principle of the generative adversarial network GAN (Generative Adversarial Networks) comes from the zero-sum game idea in game theory and has very important significance in the field of deep learning. GAN consists of a generative model and a discriminative model. The generative model mainly learns the feature distribution of sample data; the discriminative model can be regarded as a classifier that judges the input data and gives feedback on whether it is real data or a generated sample. As Figure 2As shown, the GAN consists of a generator network and a discriminator network, which are trained in an adversarial manner. The generator network aims to learn the data distribution of real sample data, while the discriminator network is used to distinguish whether the input is real or fake generated data. They are trained alternately until the two parts of the network reach an excellent balance state. The two are a process of mutual optimization. The generator network is optimized through the discriminative feedback of the discriminator network on real data or generated data, so that the generation ability of the generator network continues to improve. In this process, the data generated by the generator network gets closer and closer to the real samples. To distinguish well between the real data and the generated data input to the discriminator network, the performance of the discriminator network also needs to be improved significantly. During the adversarial learning process of the two, the generator network and the discriminator network gradually reach equilibrium. Through this continuously optimized process, the generator network can generate generated data with the same feature distribution as the sample data, and the discriminator network can no longer distinguish them well until the discrimination probability P = 0.5 to complete the training.

[0038] In the prior art, the input to the GAN is generally a monocular vision 2D image, that is, a video image of a scene taken from one perspective. When the scene is relatively complex (such as there are many objects in the scene), there will be a phenomenon that some objects are blocked in the monocular vision 2D image. In severe cases, some objects cannot appear in the image at all. Therefore, the 3D image generated based on the monocular vision 2D image will be incomplete due to occlusion. For this reason, in this embodiment, the monocular vision 2D image input to the GAN is replaced with a multiocular vision 2D image taken from different perspectives (stitched together as the real image input to the GAN).

[0039] In this embodiment, step 102 is mainly used to obtain the 3D structure based on the generated 2D image. In this embodiment, the existing 3DINN (3D Interpreter Network) network is used to generate the three-dimensional structure. The 3DINN network is an end-to-end three-dimensional interpreter network that outputs the key point information and three-dimensional structure of the given 2D image. Generating the three-dimensional structure from the 2D image is achieved by formulating the three-dimensional key point positions as a weighted sum of a set of basic shapes, and then modeling through rotation mapping and translation mapping. For example, for a chair, the basic shapes refer to the simplification of the chair skeleton representation: one is the average shape of the chair, the second is to control how the back bends, and the third is the legs of the chair. The weight of each basic shape determines the intensity of the deformation, and these weights are represented as the internal parameters of the object. To recover the 2D image to three-dimensional structure information, during training, in addition to learning these internal parameters and the parameters of rotation mapping and translation mapping, for supervised training, the mapping transformation parameters from the three-dimensional structure to the 2D key points also need to be learned. In this embodiment, the end-to-end 3DINN network is used to generate the 3D structure, which can reduce the complexity of generating the 3D scene.

[0040] In this embodiment, step 103 is mainly used to generate a 3D image of the scene. In this embodiment, the 3D structure obtained in step 102 and the 2D image obtained in step 101 are spliced and then input into a trained 3D convolutional neural network to output the predicted values of the R, G, and B channels, thereby obtaining the 3D image of the scene. When training the 3D convolutional neural network, the model is optimized by minimizing the root mean square error between the current actual output value and the label value in the training set. This embodiment utilizes the color and texture features of the objects in the 2D image, as well as the spatial structure information and position information of the objects in the 3D structure, to generate a real 3D scene.

[0041] In this embodiment, the monocular vision 2D image input into the GAN is replaced with a multiocular vision 2D image taken from different perspectives, which solves the problems existing in the prior art, such as the generated 3D scene being relatively single and incomplete. As Figure 3 shown, the three 2D images in the upper row are taken from three different angles, and each image is incomplete. For example, the mouse and the water cup do not appear in the leftmost image; the image in the lower row is the finally generated 3D image. Obviously, all the objects in the scene are included in the 3D image.

[0042] As an optional embodiment, the method further includes: splicing the multiocular vision 2D images and stretching them into a one-dimensional vector, and then splicing it with an N-dimensional random vector to obtain an N + 1-dimensional vector, and inputting the N + 1-dimensional vector into the generative adversarial network.

[0043] This embodiment provides a technical solution for one-dimensionalizing a 2D image. In this embodiment, before inputting the real image into the generative adversarial network, the multiocular vision 2D images (at least two) taken from different angles are first spliced together; then the 2D image is stretched into a one-dimensional vector and spliced with an N-dimensional random vector to obtain an N + 1-dimensional vector; finally, the obtained N + 1-dimensional vector is input into the generative adversarial network.

[0044] As an optional embodiment, the 3D convolutional neural network includes five basic networks, a fully connected layer, and an output layer including three neurons, and the basic network is the lightweight ResNet18.

[0045] This embodiment provides a technical solution for a 3D convolutional neural network. The 3D convolutional neural network of this embodiment includes 5 basic networks, a fully connected layer, and an output layer containing 3 neurons. The basic network is used to extract features of the image, and its high-level features are extracted through 5 basic networks. The basic network is connected to a fully connected layer and an output auxiliary node. During the training process, the output feature values can better approximate the true feature values. To simplify the model, reduce the computational load, and thus improve the running speed, the 5 basic networks in this embodiment all adopt the lightweight ResNet18. Network lightweighting is generally achieved through the principle of model compression to solve the problem of large memory occupation by network parameters, thereby improving the network calculation speed. ResNet18 is a relatively common lightweight neural network model at present. It uses a basic unit with 2 convolutional operations of 3×3. After the input image passes through a convolutional layer with a stride of 2, it then passes through a BN layer and a ReLU layer, and finally outputs through a max pooling layer. Then it passes through 4 convolutional blocks of ResNet18, where each convolutional block has 4 convolutional layers, for a total of 16 convolutional layers. Finally, it passes through an average pooling layer, a fully connected layer, and softmax to output the final result.

[0046] As an optional embodiment, the method further includes: caching the generated 2D image. When generating a 2D image next time, first check whether there is an image similar to the input multi-view visual 2D image in the cached 2D images. If so, obtain the similar image from the cache, splice it with the input multi-view visual 2D image, and then input it into the generative adversarial network.

[0047] This embodiment provides a technical solution for reducing the computational load and improving the running speed. In this embodiment, the generated 2D images are cached, and the cached images are used to reduce the computational load. Specifically, first cache the generated 2D images. Before generating a new 2D image, read the 2D images in the cache. If there is an image similar to the multi-view visual 2D image in the cache, splice the similar image in the cache with the multi-view visual 2D image, and then input it into the generative adversarial network; if there is no similar image, put the multi-view visual 2D image into the cache.

[0048] As an optional embodiment, the method for checking whether two images are similar includes: representing the two images to be checked as a one-dimensional vector respectively, and obtaining the similarity of the two images by calculating the cosine distance between the two vectors; if the similarity is greater than the set threshold, the images to be checked are similar; otherwise, they are not similar; the formula for calculating the cosine distance of two vectors X and Y is as follows:

[0049]

[0050] In the formula, cos(X,Y) is the cosine distance between X and Y, X = (x1, x2,..., xn ),Y = (y1, y2, …, y n )。

[0051] This embodiment provides a technical solution for checking whether two images are similar. In this embodiment, the two images are first represented as a one-dimensional vector respectively; then the cosine distance between the two one-dimensional vectors (the calculation formula is as above) is calculated, that is, the cosine distance is used as the similarity between the two vectors; then it is compared with a set threshold. If the similarity is greater than the set threshold, it is considered that the two images are similar; otherwise, they are not similar. The threshold can be set according to experience. For example, the threshold can be set to 0.9.

[0052] Figure 4 It is a schematic diagram of the composition of a 3D image generation device based on a generative adversarial network according to an embodiment of the present invention. The device includes:

[0053] A 2D image generation module 11, configured to input multi-view vision 2D images taken from different perspectives and a random variable into a trained generative adversarial network to generate a 2D image;

[0054] A 3D structure generation module 12, configured to input the generated 2D image into a 3DINN network to obtain the 3D structure corresponding to the 2D image;

[0055] A 3D image generation module 13, configured to splice the 3D structure and the 2D image and then input them into a trained 3D convolutional neural network to output the predicted values of the R, G, and B channels.

[0056] The device in this embodiment can be used to execute Figure 1 the technical solution of the method embodiment shown. The implementation principle and technical effect are similar and will not be elaborated here. The same is true for the subsequent embodiments and will not be further described.

[0057] As an optional embodiment, the device further includes a vector splicing module, configured to splice multi-view vision 2D images and then stretch them into a one-dimensional vector, and then splice them with an N-dimensional random vector to obtain an N + 1-dimensional vector, and input the N + 1-dimensional vector into the generative adversarial network.

[0058] As an optional embodiment, the 3D convolutional neural network includes 5 basic networks, a fully connected layer, and an output layer including 3 neurons. The basic network is a lightweight ResNet18.

[0059] Further, the device further includes a 2D image cache module for caching the generated 2D images. When generating 2D images next time, first check whether there is an image similar to the input multi-view visual 2D image in the cached 2D images. If so, obtain the similar image from the cache, splice it with the input multi-view visual 2D image, and then input the result into the generative adversarial network.

[0060] As an optional embodiment, the method for checking whether two images are similar includes: representing the two images to be checked as a one-dimensional vector respectively, and obtaining the similarity between the two images by calculating the cosine distance between the two vectors; if the similarity is greater than a set threshold, the images to be checked are similar; otherwise, they are not similar; the formula for calculating the cosine distance between two vectors X and Y is as follows:

[0061]

[0062] In the formula, cos(X,Y) is the cosine distance between X and Y, X = (x1, x2,..., x n ), Y = (y1, y2,..., y n ).

[0063] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A 3D image generation method based on a generative adversarial network, characterized in that, Including the following steps: Input multi-view visual 2D images captured from different perspectives and a random variable into a trained generative adversarial network to generate a 2D image; Input the generated 2D image into a 3DINN network to obtain the 3D structure corresponding to the 2D image; Input the spliced 3D structure and 2D image into a trained 3D convolutional neural network to output predicted values for the three channels of R, G, and B; The method further includes: caching the generated 2D image. When generating a 2D image next time, first check whether there is an image similar to the input multi-view visual 2D image in the cached 2D images. If so, obtain the similar image from the cache, splice it with the input multi-view visual 2D image, and then input it into the generative adversarial network.

2. The 3D image generation method based on a generative adversarial network according to claim 1, wherein The method further includes: splicing the multi-view visual 2D images and stretching them into a one-dimensional vector, then splicing it with an N-dimensional random vector to obtain an N + 1-dimensional vector, and inputting the N + 1-dimensional vector into the generative adversarial network.

3. The 3D image generation method based on a generative adversarial network according to claim 1, characterized in that The 3D convolutional neural network includes 5 basic networks, a fully connected layer, and an output layer containing 3 neurons. The basic network is a lightweight ResNet18.

4. The 3D image generation method based on a generative adversarial network according to claim 1, characterized in that The method for checking whether two images are similar includes: representing the two images to be checked as one-dimensional vectors respectively, obtaining the similarity of the two images by calculating the cosine distance between the two vectors; if the similarity is greater than the set threshold, the images to be checked are similar; otherwise, they are not similar. The formula for calculating the cosine distance between two vectors X and Y is as follows: Where, cos(X,Y) is the cosine distance between X and Y, X = (x1, x2, …, x n ), Y = (y1, y2, …, y n ).

5. A 3D image generation device based on a generative adversarial network, characterized in that, Including: A 2D image generation module for inputting multi-view visual 2D images captured from different perspectives and a random variable into a trained generative adversarial network to generate a 2D image; A 3D structure generation module for inputting the generated 2D image into a 3DINN network to obtain the 3D structure corresponding to the 2D image; A 3D image generation module for inputting the spliced 3D structure and 2D image into a trained 3D convolutional neural network to output predicted values for the three channels of R, G, and B; A 2D image caching module for caching the generated 2D image. When generating a 2D image next time, first check whether there is an image similar to the input multi-view visual 2D image in the cached 2D images. If so, obtain the similar image from the cache, splice it with the input multi-view visual 2D image, and then input it into the generative adversarial network.

6. The 3D image generation device based on a generative adversarial network according to claim 5, characterized in that The device further includes a vector splicing module for splicing the multi-view visual 2D images and stretching them into a one-dimensional vector, then splicing it with an N-dimensional random vector to obtain an N + 1-dimensional vector, and inputting the N + 1-dimensional vector into the generative adversarial network.

7. The 3D image generation device based on a generative adversarial network according to claim 5, wherein The 3D convolutional neural network includes 5 basic networks, a fully connected layer, and an output layer containing 3 neurons. The basic network is a lightweight ResNet18.

8. The 3D image generation device based on a generative adversarial network according to claim 5, characterized in that, The method for checking whether two images are similar includes: representing the two images to be checked as one-dimensional vectors respectively, and obtaining the similarity between the two images by calculating the cosine distance between the two vectors; if the similarity is greater than a set threshold, the images to be checked are similar; otherwise, they are not similar; the formula for calculating the cosine distance between two vectors X and Y is as follows: where cos(X, Y) is the cosine distance between X and Y, X = (x1, x2, …, x n ), Y = (y1, y2, …, y n ).

Images