A method for generating 3D graphics relighting based on generative adversarial networks

Abstract

This invention discloses a method for generating relit 3D graphics based on generative adversarial networks (GANs), belonging to the fields of deep learning and computer vision. The method includes the following steps: obtaining training and testing datasets with illumination values ​​based on illumination parameters; constructing a scene transformation GAN and a shadow estimation GAN; iteratively training the scene transformation GAN and shadow estimation GAN using the training dataset, and inputting the testing dataset into the trained scene transformation GAN and shadow estimation GAN to obtain unlit scene features and shadow distribution estimation features; constructing a re-rendering network; and obtaining a relit image based on the unlit scene features, shadow distribution estimation features, and the re-rendering network. This invention enables the rapid generation of high-quality rendered images of new scenes from 3D scenes after changing the light source and viewpoint.

Description

Technical Field

[0001] This invention belongs to the field of strain measurement technology, and particularly relates to a method for generating relighting of three-dimensional graphics based on generative adversarial networks. Background Technology

[0002] Generative Adversarial Networks (GANs) are deep learning models that construct two models, a generator and a discriminator, and allow them to learn from each other through a game, thus producing high-quality generated images.

[0003] In 3D scenes, the appearance of 3D models often requires the generation of lighting rendering effects from different light sources and perspectives. Traditional relighting methods are image-based relighting, which requires collecting images of the scene under different lighting conditions and synthesizing the results under new lighting from these images. This requires customized equipment and consumes a lot of financial, material, and human resources. With the development of deep learning technology, based on deep generative models and optical physics knowledge in the field of computer graphics, the field of neural rendering technology has been formed. Neural rendering is of great significance for the generation of relighting scenes. Through neural rendering technology, the system can learn the lighting direction and relighting function of the scene from a small number of images and generate relighting images. However, the image quality obtained by existing solutions is still not good. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a method for generating relit 3D graphics based on generative adversarial networks (GANs). By employing GANs and neural rendering techniques, this method solves the problem of insufficient quality in relit images generated from a limited number of images.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0006] This invention provides a method for generating relighting of 3D graphics based on generative adversarial networks, comprising the following steps:

[0007] S1. Obtain training and test datasets with illumination values ​​based on illumination parameters;

[0008] S2. Construct a scene transformation generative adversarial network based on the test dataset, scene transformation generator, and scene discriminator;

[0009] S3. Based on the test dataset, scene transformation generator, and shadow discriminator, construct a shadow estimation generative adversarial network;

[0010] S4. Use the training dataset to iteratively train the scene transformation generative adversarial network and the shadow estimation generative adversarial network respectively, and input the test dataset into the trained scene transformation generative adversarial network and the shadow estimation generative adversarial network respectively to obtain the features of the unlit scene and the shadow distribution estimation features.

[0011] S5. Construct a re-rendering network;

[0012] S6. Based on the features of the unlit scene, the shadow distribution estimation features, and the re-rendering network, a relit image is obtained.

[0013] The beneficial effects of this invention are as follows: This invention provides a 3D graphics relighting generation method based on generative adversarial networks (GANs). It establishes a training dataset and a test dataset composed of images with illumination values, and separately establishes a scene transformation GAN and a shadow estimation GAN. After training the scene transformation GAN and shadow estimation GAN using the training dataset, the test dataset with illumination values ​​is used as input to the trained adversarial networks to obtain unlit scene features and shadow distribution estimation features. A re-rendering network is then constructed, enabling the generation of relit images based on unlit scene features, shadow distribution estimation features, and the re-rendering network. This solution can quickly generate high-quality rendered images of new scenes after changing the light source and viewpoint of a 3D scene.

[0014] Further, step S1 includes the following steps:

[0015] S11. Generate illumination parameters including photovoltaic intensity, illumination distance, light source direction and light source elevation angle by random method. The variation range of photovoltaic intensity and illumination distance is greater than 1 and less than 20. The variation range of light source azimuth angle θ is greater than 0° and less than 360°. The variation range of light source elevation angle γ is greater than -90° and less than 90°.

[0016] S12. Add the lighting parameters to the rendering channel of the 3D scene to obtain a rendering dataset with lighting labels.

[0017] S13. Preprocess the rendering dataset to obtain a dataset with lighting values;

[0018] S14. Divide the dataset into a training dataset and a test dataset with illumination values.

[0019] The beneficial effects of adopting the above-mentioned further scheme are: constructing training and testing datasets with illumination values, providing a foundation for training scene transformation generative adversarial networks and shadow estimation generative adversarial networks, and obtaining features of unlit scenes and shadow distribution estimation features through the trained scene transformation generative adversarial networks and shadow estimation generative adversarial networks.

[0020] Furthermore, the scene transformation generator includes a first downsampling module, and a first residual module and a first upsampling module, both connected to the first downsampling module; the first residual module is connected to the first upsampling module; the shadow estimation generator includes a second downsampling module, a second residual module, and a second upsampling module connected in sequence.

[0021] The first downsampling module includes a first input convolutional layer, a first downsampling convolutional layer, a second downsampling convolutional layer, a third downsampling convolutional layer, and a fourth downsampling convolutional layer connected sequentially through a first output terminal; the input terminal of the first input convolutional layer serves as the input terminal of the first downsampling module, and inputs each image in the test dataset; the output terminal of the fourth downsampling convolutional layer serves as the output terminal of the first downsampling module and is connected to the input terminal of the first residual module;

[0022] The first residual module includes nine residual sub-modules connected in sequence. The input terminal of the first residual sub-module serves as the input terminal of the first residual module and is connected to the output terminal of the first downsampling module. The output terminal of the last residual sub-module serves as the output terminal of the first residual module and is connected to the input terminal of the first upsampling module.

[0023] The first upsampling module includes a first upsampling transposed convolutional layer, a second upsampling transposed convolutional layer, a third upsampling transposed convolutional layer, a fourth upsampling transposed convolutional layer, and a first output convolutional layer, which are connected sequentially through a first input terminal. The input terminal of the first upsampling transposed convolutional layer serves as the input terminal of the first upsampling module and is connected to the output terminal of the first residual module. The second output terminals of the first input convolutional layer, the first downsampling convolutional layer, the second downsampling convolutional layer, and the third downsampling convolutional layer are connected one-to-one with the second input terminals of the first output convolutional layer, the fourth upsampling transposed convolutional layer, the third upsampling transposed convolutional layer, and the second upsampling transposed convolutional layer. The output terminal of the first output convolutional layer serves as the output terminal of the first upsampling module and outputs the features of the scene without illumination.

[0024] The scene discriminator adopts a Markov discriminator structure, which consists of four stacked strided convolutional layers. The input of the first strided convolutional layer serves as the input of the scene discriminator and is connected to the output of the first upsampling module. The output of the fourth strided convolutional layer serves as the output of the scene discriminator, forming a scene transformation generation network.

[0025] The beneficial effects of adopting the above-mentioned further scheme are: providing a specific structure for scene transformation generative adversarial networks, and providing a foundation for obtaining the features of unlit scenes through scene transformation generative adversarial networks.

[0026] Furthermore, the shadow estimation generator includes a second downsampling module, a second residual module, and a second upsampling module connected in sequence;

[0027] The second downsampling module includes a second input convolutional layer, a fifth downsampling convolutional layer, a sixth downsampling convolutional layer, a seventh downsampling convolutional layer, and an eighth downsampling convolutional layer connected in sequence; the input end of the second input convolutional layer serves as the input end of the second downsampling module, and inputs each image in the test dataset; the output end of the eighth downsampling convolutional layer serves as the output end of the second downsampling module, and is connected to the input end of the second residual module;

[0028] The second residual module includes nine residual sub-modules connected in sequence. The input terminal of the first residual sub-module serves as the input terminal of the second residual module and is connected to the output terminal of the second downsampling module. The output terminal of the last residual sub-module serves as the output terminal of the second residual module and is connected to the input terminal of the second upsampling module.

[0029] The second upsampling module includes a fifth upsampling transposed convolutional layer, a sixth upsampling transposed convolutional layer, a seventh upsampling transposed convolutional layer, an eighth upsampling transposed convolutional layer, and a second output convolutional layer connected in sequence; the input of the fifth upsampling transposed convolutional layer serves as the input of the second upsampling module and is connected to the output of the second residual module; the output of the second output convolutional layer serves as the output of the first upsampling module, outputting shadow distribution estimation features;

[0030] The shadow discriminator adopts a Markov discriminator structure, consisting of four stacked strided convolutional layers. The input of the first strided convolutional layer serves as the input of the shadow discriminator and is connected to the output of the second upsampling module. The output of the fourth strided convolutional layer serves as the output of the shadow discriminator, forming a shadow estimation and generation network.

[0031] The beneficial effects of adopting the above-mentioned further scheme are: it provides a specific structure for the shadow estimation generative adversarial network, and provides a basis for obtaining shadow distribution estimation features through the shadow estimation generative adversarial network.

[0032] Furthermore, the computational expressions for the scene transformation generator and the shadow estimation generator are as follows:

[0033]

[0034]

[0035] Among them, G A G represents the result of the scene transformation. B The image represents the shadow estimation result, λ represents the expectation coefficient, X represents the input image, and Y represents the image under the target light source. Y represents the expected value of an image that conforms to a normal distribution under the input image and the target light source. || represents the vector magnitude. sf Let G(X) represent the target image of the shadow, G(X) represent the generator output after inputting the image, and arg represent taking a subset of the domain. Denotes the lower bound of generator G. This represents the upper bound of the discriminator D. This represents the loss function of a generative adversarial network based on a generator G and a discriminator D. Represents the shadow discriminator D shad The upper realm, This is based on a generator G and a shadow discriminator D. shad Generative adversarial network loss function.

[0036] The beneficial effects of adopting the above-mentioned further scheme are: providing specific computational methods for scene transformation generators and shadow estimation generators, and providing a foundation for constructing scene transformation generative adversarial networks and shadow estimation generative adversarial networks.

[0037] Furthermore, the scene discriminator is used to distinguish between the output no-light scene features of the scene transformation generator and the shadowless images in the test dataset; the shadow discriminator is used to distinguish between the output shadow distribution estimation features of the shadow estimation generator and the shadowless images in the test dataset; both the scene discriminator and the shadow discriminator calculate the loss only at the patch level, judge the true or false for each N×N patch, apply discriminator convolution on the entire image, and use the average result as the final output;

[0038] The loss functions of the scene discriminator and the shadow discriminator are calculated using the following expressions:

[0039]

[0040] in, The result of the discriminator's loss function is represented by X, which represents the input image, and Y represents the input image. sf Represents a baseline image without shadows. D(X,Y) represents the mathematical expectation of a specified distribution that conforms to both the input image and the shadowless reference image. sf This represents the discriminator output based on the input image and a shadowless reference image. G(X) represents the expected value of the input image according to the specified distribution. G(X) represents the generator output based on the input image. D(X,G(X)) represents the discriminator output based on the input image and the generator output.

[0041] The beneficial effects of adopting the above-mentioned further scheme are: providing a specific calculation method for the loss function of the scene discriminator and the shadow discriminator, and providing a foundation for constructing scene transformation generative adversarial networks and shadow estimation generative adversarial networks.

[0042] Furthermore, the re-rendering network includes a multi-scale perception module, a channel recalibration module, and a rendering module connected in sequence;

[0043] The multi-scale perception module includes several perception size filters; each of the perception size filters obtains global information features and local information features through features of the unlit scene and shadow distribution estimation features;

[0044] The channel recalibration module uses a compression and excitation network SENet; the compression and excitation network SENet is used to set global information feature weights and local information feature weights respectively;

[0045] The rendering module uses a convolutional layer with a filter size of 7×7, a padding of 3, and a stride of 1. The activation function of the convolutional layer is Tanh, which is used to recalibrate the feature map through the mapping process between the feature space and the image space to obtain the relit image.

[0046] The beneficial effects of adopting the above-mentioned further scheme are: it provides a specific network structure for the re-rendering network, which provides a basis for using the re-rendering network to process the features of unlit scenes and shadow distribution estimation features to obtain relit images.

[0047] Furthermore, the calculation expression for the loss function of the re-rendering network is as follows:

[0048]

[0049] Where Y′ represents the recalibrated feature map, λ represents the re-illuminated image, λ′ represents the re-illumination balance parameter, and feat(·) represents the feature map extracted from the image using the VGG19 network.

[0050] The beneficial effects of adopting the above-mentioned further scheme are: it provides a specific calculation method for the loss function of the re-rendering network, and provides a basis for obtaining re-illuminated images by rendering recalibrated feature maps through the renderer.

[0051] Further, step S6 includes the following steps:

[0052] S61. Input the features of the unlit scene and the shadow distribution estimation features into the multi-scale perception module to obtain global information features and local information features that include multi-scale information.

[0053] S62. Input the global information features and local information features into the channel recalibration module, and perform feature equalization, pooling and full connection on the global information features and local information features in sequence, as well as perform element-wise multiplication with the global information features and local information features to obtain the recalibration feature map;

[0054] S63. Input the recalibrated feature map into the rendering module, and obtain the relit image through the mapping between the feature space and the image space.

[0055] The beneficial effects of adopting the above-mentioned further scheme are as follows: it provides a specific method for obtaining a relit image based on features of a no-light scene, shadow distribution estimation features, and a re-rendering network. Through multi-scale feature perception fusion and weight adjustment, multi-scale fusion features with key values ​​are obtained, namely, recalibrated feature maps. The obtained features are input into the renderer, and the target image is obtained after rendering. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the steps of a three-dimensional graphics relighting generation method based on generative adversarial networks in an embodiment of the present invention.

[0057] Figure 2 This is a schematic diagram of the scene transformation generator in an embodiment of the present invention.

[0058] Figure 3 This is a schematic diagram of the shadow estimation generator in an embodiment of the present invention. Detailed Implementation

[0059] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0060] SENet (Squeeze-and-Excitation Networks): A compression and excitation network that explicitly models the interdependencies between feature channels and achieves the fusion of feature channels through feature recalibration.

[0061] like Figure 1 As shown, in one embodiment of the present invention, the present invention provides a method for generating relighting of three-dimensional graphics based on generative adversarial networks, comprising the following steps:

[0062] S1. Obtain training and test datasets with illumination values ​​based on illumination parameters;

[0063] Step S1 includes the following steps:

[0064] S11. Generate illumination parameters including photovoltaic intensity, illumination distance, light source direction and light source elevation angle by random method. The variation range of photovoltaic intensity and illumination distance is greater than 1 and less than 20. The variation range of light source azimuth angle θ is greater than 0° and less than 360°. The variation range of light source elevation angle γ is greater than -90° and less than 90°.

[0065] S12. Add the lighting parameters to the rendering channel of the 3D scene to obtain a rendering dataset with lighting labels.

[0066] S13. Preprocess the rendering dataset to obtain a dataset with lighting values;

[0067] S14. Divide the dataset into a training dataset and a test dataset with illumination values;

[0068] S2. Construct a scene transformation generative adversarial network based on the test dataset, scene transformation generator, and scene discriminator;

[0069] like Figure 2 As shown, the scene conversion generator includes a first downsampling module, and a first residual module and a first upsampling module, both connected to the first downsampling module; the first residual module is connected to the first upsampling module.

[0070] The first downsampling module includes a first input convolutional layer, a first downsampling convolutional layer, a second downsampling convolutional layer, a third downsampling convolutional layer, and a fourth downsampling convolutional layer connected sequentially through a first output terminal; the input terminal of the first input convolutional layer serves as the input terminal of the first downsampling module, and inputs each image in the test dataset; the output terminal of the fourth downsampling convolutional layer serves as the output terminal of the first downsampling module and is connected to the input terminal of the first residual module;

[0071] The first residual module includes nine residual sub-modules connected in sequence. The input terminal of the first residual sub-module serves as the input terminal of the first residual module and is connected to the output terminal of the first downsampling module. The output terminal of the last residual sub-module serves as the output terminal of the first residual module and is connected to the input terminal of the first upsampling module.

[0072] The first upsampling module includes a first upsampling transposed convolutional layer, a second upsampling transposed convolutional layer, a third upsampling transposed convolutional layer, a fourth upsampling transposed convolutional layer, and a first output convolutional layer, which are connected sequentially through a first input terminal. The input terminal of the first upsampling transposed convolutional layer serves as the input terminal of the first upsampling module and is connected to the output terminal of the first residual module. The second output terminals of the first input convolutional layer, the first downsampling convolutional layer, the second downsampling convolutional layer, and the third downsampling convolutional layer are connected one-to-one with the second input terminals of the first output convolutional layer, the fourth upsampling transposed convolutional layer, the third upsampling transposed convolutional layer, and the second upsampling transposed convolutional layer. The output terminal of the first output convolutional layer serves as the output terminal of the first upsampling module and outputs the features of the scene without illumination.

[0073] The scene discriminator adopts a Markov discriminator structure, which consists of four stacked strided convolutional layers. The input of the first strided convolutional layer serves as the input of the scene discriminator and is connected to the output of the first upsampling module. The output of the fourth strided convolutional layer serves as the output of the scene discriminator, forming a scene transformation generation network.

[0074] S3. Based on the test dataset, scene transformation generator, and shadow discriminator, construct a shadow estimation generative adversarial network;

[0075] like Figure 3 As shown, the shadow estimation generator includes a second downsampling module, a second residual module, and a second upsampling module connected in sequence.

[0076] The second downsampling module includes a second input convolutional layer, a fifth downsampling convolutional layer, a sixth downsampling convolutional layer, a seventh downsampling convolutional layer, and an eighth downsampling convolutional layer connected in sequence; the input end of the second input convolutional layer serves as the input end of the second downsampling module, and inputs each image in the test dataset; the output end of the eighth downsampling convolutional layer serves as the output end of the second downsampling module, and is connected to the input end of the second residual module;

[0077] The second residual module includes nine residual sub-modules connected in sequence. The input terminal of the first residual sub-module serves as the input terminal of the second residual module and is connected to the output terminal of the second downsampling module. The output terminal of the last residual sub-module serves as the output terminal of the second residual module and is connected to the input terminal of the second upsampling module.

[0078] The second upsampling module includes a fifth upsampling transposed convolutional layer, a sixth upsampling transposed convolutional layer, a seventh upsampling transposed convolutional layer, an eighth upsampling transposed convolutional layer, and a second output convolutional layer connected in sequence; the input of the fifth upsampling transposed convolutional layer serves as the input of the second upsampling module and is connected to the output of the second residual module; the output of the second output convolutional layer serves as the output of the first upsampling module, outputting shadow distribution estimation features;

[0079] The shadow discriminator adopts a Markov discriminator structure, consisting of four stacked strided convolutional layers. The input of the first strided convolutional layer serves as the input of the shadow discriminator and is connected to the output of the second upsampling module. The output of the fourth strided convolutional layer serves as the output of the shadow discriminator, forming a shadow estimation and generation network.

[0080] The computational expressions for the scene transformation generator and the shadow estimation generator are as follows:

[0081]

[0082]

[0083] Among them, G A G represents the result of the scene transformation. B The image represents the shadow estimation result, λ represents the expectation coefficient, X represents the input image, and Y represents the image under the target light source. Y represents the expected value of an image that conforms to a normal distribution under the input image and the target light source. || represents the vector magnitude. sf Let G(X) represent the target image of the shadow, G(X) represent the generator output after inputting the image, and arg represent taking a subset of the domain. Denotes the lower bound of generator G. This represents the upper bound of the discriminator D. This represents the loss function of a generative adversarial network based on a generator G and a discriminator D. Represents the shadow discriminator D shad The upper realm, This is based on a generator G and a shadow discriminator D. shad The loss function of the generative adversarial network;

[0084] The scene discriminator is used to distinguish between the output of the scene transformation generator (no lighting scene features) and the shadowless image in the test dataset; the shadow discriminator is used to distinguish between the output of the shadow estimation generator (shadow distribution estimation features) and the shadowless image in the test dataset; both the scene discriminator and the shadow discriminator calculate the loss only at the patch level, judge the truth value of each N×N patch, apply discriminator convolution on the entire image, and use the average result as the final output;

[0085] The loss functions of the scene discriminator and the shadow discriminator are calculated using the following expressions:

[0086]

[0087] in, The result of the discriminator's loss function is represented by X, which represents the input image, and Y represents the input image. sf Represents a baseline image without shadows. D(X,Y) represents the mathematical expectation of a specified distribution that conforms to both the input image and the shadowless reference image. sf This represents the discriminator output based on the input image and a shadowless reference image. G(X) represents the expected value of the input image according to the specified distribution, G(X) represents the generator output based on the input image, and D(X,G(X)) represents the discriminator output based on the input image and the generator output.

[0088] S4. Use the training dataset to iteratively train the scene transformation generative adversarial network and the shadow estimation generative adversarial network respectively, and input the test dataset into the trained scene transformation generative adversarial network and the shadow estimation generative adversarial network respectively to obtain the features of the unlit scene and the shadow distribution estimation features.

[0089] S5. Construct a re-rendering network;

[0090] The re-rendering network includes a multi-scale perception module, a channel recalibration module, and a rendering module connected in sequence.

[0091] The multi-scale perception module includes several perception size filters; each of the perception size filters obtains global information features and local information features through features of the unlit scene and shadow distribution estimation features;

[0092] The channel recalibration module uses a compression and excitation network SENet; the compression and excitation network SENet is used to set global information feature weights and local information feature weights respectively;

[0093] The rendering module uses a convolutional layer with a filter size of 7×7, a padding of 3, and a stride of 1. The activation function of the convolutional layer is Tanh, which is used to recalibrate the feature map through the mapping process between the feature space and the image space to obtain the relit image.

[0094] The loss function of the re-rendered network is calculated as follows:

[0095]

[0096] Where Y′ represents the recalibrated feature map, λ represents the re-illuminated image, λ′ represents the re-illuminated balance parameter, and feat(·) represents the feature map extracted from the image using the VGG19 network.

[0097] S6. Based on the features of the unlit scene, the shadow distribution estimation features, and the re-rendering network, the relit image is obtained;

[0098] Step S6 includes the following steps:

[0099] S61. Input the features of the unlit scene and the shadow distribution estimation features into the multi-scale perception module to obtain global information features and local information features that include multi-scale information.

[0100] S62. Input the global information features and local information features into the channel recalibration module, and perform feature equalization, pooling and full connection on the global information features and local information features in sequence, as well as perform element-wise multiplication with the global information features and local information features to obtain the recalibration feature map;

[0101] S63. Input the recalibrated feature map into the rendering module, and obtain the relit image through the mapping between the feature space and the image space.

[0102] The beneficial effects of this invention are as follows: This invention provides a 3D graphics relighting generation method based on generative adversarial networks (GANs). It establishes a training dataset and a test dataset composed of images with illumination values, and separately establishes a scene transformation GAN and a shadow estimation GAN. After training the scene transformation GAN and shadow estimation GAN using the training dataset, the test dataset with illumination values ​​is used as input to the trained adversarial networks to obtain unlit scene features and shadow distribution estimation features. A re-rendering network is then constructed, enabling the generation of relit images based on unlit scene features, shadow distribution estimation features, and the re-rendering network. This solution can quickly generate high-quality rendered images of new scenes after changing the light source and viewpoint of a 3D scene.

Claims

1. A method for generating a three-dimensional image based on a generative adversarial network, characterized in that, The method comprises the following steps: S1, obtaining a training dataset and a test dataset with illumination values based on illumination parameters; S2, constructing a scene conversion generator based on the test dataset, a scene discriminator, and a shadow discriminator; S3, constructing a shadow estimation generative adversarial network based on the test dataset, the scene conversion generator, and the shadow discriminator; S4, iteratively training the scene conversion generator and the shadow estimation generative adversarial network respectively using the training dataset, and inputting the test dataset into the trained scene conversion generator and the shadow estimation generative adversarial network respectively to obtain an illumination-free scene feature and a shadow distribution estimation feature; S5, constructing a re-rendering network; S6, obtaining a re-illumination image based on the illumination-free scene feature, the shadow distribution estimation feature, and the re-rendering network; The calculation expressions of the scene conversion generator and the shadow estimation generator are as follows respectively: wherein, denotes a scene transition generation result, denotes a shadow estimation generation result, denotes a desired coefficient, X denotes an input image, Y denotes an image under a target light source, denotes a mathematical expectation conforming to normal distribution of the input image and the image under the target light source, denotes a vector length, denotes a shadow target image, denotes a generator output result after the input image, denotes a definition domain subset, denotes a generator G lower bound, denotes a discriminator D upper bound, denotes a generative adversarial network loss function based on the generator G and the discriminator D , denotes a shadow discriminator upper bound, denotes a generative adversarial network loss function based on the generator G and the shadow discriminator ; The scene discriminator is configured to discriminate the output of the scene transition generator from the shadow-free image in the test dataset; the shadow discriminator is configured to discriminate the output of the shadow estimation generator from the shadow-free image in the test dataset; the scene discriminator and the shadow discriminator both only calculate loss at the patch level, judge true or false for each patch of a predetermined size, apply discriminator convolution on the whole image, and take the average result as the final output. N x N size of the patch to judge true or false, apply discriminator convolution on the whole image, and take the average result as the final output. The re-rendering network comprises a multi-scale perception module, a channel recalibration module, and a rendering module connected in sequence; The multi-scale perception module comprises a plurality of perception size filters; each perception size filter obtains global information features and local information features through the illumination-free scene feature and the shadow distribution estimation feature; The channel recalibration module adopts a compression and excitation network SENet; the compression and excitation network SENet sets global information feature weights and local feature information feature weights for the global information features and the local information features respectively; The rendering module adopts a convolution layer with a filter size of 7×7, a padding amount of 3, and a stride of 1, wherein the activation function of the convolution layer is Tanh, which is used to process the recalibration feature map through the mapping between the feature space and the image space to obtain the re-illumination image.

2. The three-dimensional image relighting generation method based on a generative adversarial network according to claim 1, characterized in that, The step S1 comprises the following steps: S11, generating the illumination parameters including photovoltaic intensity, illumination distance, light source azimuth and light source elevation angle by random method, wherein the variation range of photovoltaic intensity and illumination distance is greater than 1 and less than 20, the variation range of light source azimuth is greater than 0° and less than 360°, and the variation range of light source elevation angle is greater than -90° and less than 90°; S12, adding the illumination parameters to the rendering channel in the three-dimensional scene to obtain a rendering dataset with illumination labels; S13, pre-processing the rendering dataset to obtain a dataset with illumination values; S14, dividing the dataset to obtain a training dataset and a test dataset with illumination values.

3. The three-dimensional image relighting generation method based on a generative adversarial network according to claim 2, characterized in that, The scene conversion generator comprises a first downsampling module, a first residual module, and a first upsampling module connected with the first downsampling module; the first residual module is connected with the first upsampling module; The first downsampling module comprises a first input convolution layer, a first downsampling convolution layer, a second downsampling convolution layer, a third downsampling convolution layer, and a fourth downsampling convolution layer connected in sequence through a first output end; the input end of the first input convolution layer serves as the input end of the first downsampling module and inputs each image in the test dataset; the output end of the fourth downsampling convolution layer serves as the output end of the first downsampling module and is connected with the input end of the first residual module; The first residual module comprises nine residual sub-modules connected in sequence, wherein the input end of the first residual sub-module serves as the input end of the first residual module and is connected with the output end of the first downsampling module, and the output end of the last residual sub-module serves as the output end of the first residual module and is connected with the input end of the first upsampling module; The first residual module comprises nine residual sub-modules connected in sequence, wherein the input end of the first residual sub-module serves as the input end of the first residual module and is connected with the output end of the first downsampling module, and the output end of the last residual sub-module serves as the output end of the first residual module and is connected with the input end of the first upsampling module; The first up-sampling module comprises a first up-sampling transpose convolution layer, a second up-sampling transpose convolution layer, a third up-sampling transpose convolution layer, a fourth up-sampling transpose convolution layer and a first output convolution layer connected in sequence through a first input end; the input end of the first up-sampling transpose convolution layer serves as the input end of the first up-sampling module and is connected with the output end of the first residual module; the second output end of the first input convolution layer, the second output end of the first down-sampling convolution layer, the second output end of the second down-sampling convolution layer and the second output end of the third down-sampling convolution layer are connected with the second input end of the first output convolution layer, the second input end of the fourth up-sampling transpose convolution layer, the second input end of the third up-sampling transpose convolution layer and the second input end of the second up-sampling transpose convolution layer in one-to-one correspondence; the output end of the first output convolution layer serves as the output end of the first up-sampling module and outputs the no-light scene feature; The scene discriminator adopts a Markov discriminator structure and is composed of four stacked stride convolution layers, wherein the input end of the first stride convolution layer serves as the input end of the scene discriminator and is connected with the output end of the first up-sampling module, and the output end of the fourth stride convolution layer serves as the output end of the scene discriminator and constitutes the scene conversion generation network.

4. The three-dimensional image relighting generation method based on a generative adversarial network according to claim 3, characterized in that, The shadow estimation generator comprises a second down-sampling module, a second residual module and a second up-sampling module connected in sequence; The second down-sampling module comprises a second input convolution layer, a fifth down-sampling convolution layer, a sixth down-sampling convolution layer, a seventh down-sampling convolution layer and an eighth down-sampling convolution layer connected in sequence; the input end of the second input convolution layer serves as the input end of the second down-sampling module and inputs each image in the test data set; the output end of the eighth down-sampling convolution layer serves as the output end of the second down-sampling module and is connected with the input end of the second residual module; The second residual module comprises nine residual sub-modules connected in sequence, wherein the input end of the first residual sub-module serves as the input end of the second residual module and is connected with the output end of the second down-sampling module, and the output end of the last residual sub-module serves as the output end of the second residual module and is connected with the input end of the second up-sampling module; The second up-sampling module comprises a fifth up-sampling transpose convolution layer, a sixth up-sampling transpose convolution layer, a seventh up-sampling transpose convolution layer, an eighth up-sampling transpose convolution layer and a second output convolution layer connected in sequence; the input end of the fifth up-sampling transpose convolution layer serves as the input end of the second up-sampling module and is connected with the output end of the second residual module; the output end of the second output convolution layer serves as the output end of the first up-sampling module and outputs the shadow distribution estimation feature; The shadow discriminator adopts a Markov discriminator structure and is composed of four stacked stride convolution layers, wherein the input end of the first stride convolution layer serves as the input end of the shadow discriminator and is connected with the output end of the second up-sampling module, and the output end of the fourth stride convolution layer serves as the output end of the shadow discriminator and constitutes the shadow estimation generation network.

5. The three-dimensional graphic relighting generation method based on a generative adversarial network according to claim 1, characterized in that, The calculation expressions of the loss functions of the scene discriminator and the shadow discriminator are as follows: wherein, represents a loss function result of the discriminator, represents an input image, represents a shadow-free reference image, represents a mathematical expectation value conforming to a distribution designated by the input image and the shadow-free reference image, represents a discriminator output result based on the input image and the shadow-free reference image, represents a mathematical expectation value conforming to a distribution designated by the input image, represents a generator output result based on the input image, represents a discriminator output result based on the input image and the generator output result.

6. The three-dimensional graphic relighting generation method based on a generative adversarial network according to claim 1, characterized in that, The calculation expression of the loss function of the re-rendering network is as follows: wherein, wherein denotes a re-calibration feature map, denotes a re-illumination image, denotes a re-illumination balancing parameter, denotes a feature map extracted from the image using a VGG19 network.

7. The three-dimensional image relighting generation method based on a generative adversarial network according to claim 6, characterized in that, The step S6 comprises the following steps: S61, input the non-light scene feature and the shadow distribution estimation feature into a multi-scale perception module to obtain global information features and local information features comprising multi-scale information; S62, input the global information features and the local information features into a channel re-calibration module to sequentially perform feature homogenization, pooling and full connection on the global information features and the local information features, and perform element multiplication on matrices of the global information features and the local information features to obtain a re-calibration feature map; S63, input the re-calibration feature map into a rendering module to obtain a re-illumination image through mapping between a feature space and an image space.

Images