Single image material physical parameter editing method based on generative adversarial network

Abstract

The invention provides a single image material physical parameter editing method based on a generative adversarial network, and belongs to the technical field of computer vision, and the method comprises the steps: taking a target physical parameter as a pixel value channel, splicing the pixel value channel with an RGB channel of a target image, and obtaining an input tensor; inputting the input tensor into an encoder of a pre-trained generator to obtain image features output by the encoder; and inputting the image features into a decoder of the generator to obtain a result image output by the decoder. According to the method, physical parameters are embedded into the generative adversarial network, a physical model is used as parameterized representation of material attributes, the material physical attributes of an object in an original image can be modified through the generative adversarial neural network only by using a single image and expected material physical attributes as input, and the image quality is improved. And generating an image after the physical parameters of the appearance of the material are modified to the required values, thereby realizing control on the appearance of the material.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for editing physical parameters of a single image material based on a generative adversarial network. Background Art

[0002] In scenarios such as virtual reality, game development, and industrial design, it is a common functional requirement to adjust the appearance of a certain material. Editing the appearance of a material to obtain a variety of visual effects that suit different needs is an important part of business processes such as design and development. Commonly used material appearance editing is based on computer graphics or image processing algorithms, mainly forward rendering, reverse reconstruction, and image-based generation.

[0003] Forward rendering is a conventional method of generating images in computer graphics. It requires complete information such as 3D modeling, material parameters, lighting, geometric topology, etc. of objects in the scene, and calculates the image through the rendering algorithm. Forward rendering requires complete scene data, which is not applicable to the case where there is only an image. It cannot use arbitrary image input to edit the appearance properties of the object material in the image. In addition, the data-driven method requires a large amount of real measured material data as support, and the editing is also based on the existing measured parameters. It is basically only used in the conventional rendering process and cannot perform single image material editing.

[0004] The inverse reconstruction method can also be used for reconstruction without complete scene information. The material map reconstruction method can restore the material map of the object in the image based on the input image, and generate a new material map corresponding to the required appearance. Based on the reconstructed material, a new edited material image can be further rendered based on the reconstructed and edited material properties and ambient lighting object geometry information. However, the editing process of the material map reconstruction method also edits the material map. To obtain the image corresponding to the edited material appearance, complete scene information, including geometry, lighting, etc., is still required, and then rendered by a renderer.

[0005] Image generation-based methods do not reconstruct the entire scene or some reusable elements such as material maps. Instead, they edit the image directly at the pixel level through image generation technology, thereby editing the material in the image. The neural network model extracts scene information into latent vectors, which can be transformed into perceptual attributes with perceptual significance, such as glossiness and metallicity, so as to edit the image through perceptual attributes. Other work uses diffusion model networks to edit the physical parameters of materials.

[0006] Therefore, how to use generative adversarial networks to edit the physical parameters of a single image without complete scene information becomes a technical problem that needs to be solved urgently. Summary of the invention

[0007] The present invention provides a method for editing physical parameters of materials of a single image based on a generative adversarial network, so as to solve the defect in the prior art that physical parameters of a single image cannot be edited without complete scene information.

[0008] The present invention provides a method for editing physical parameters of a single image material based on a generative adversarial network, comprising the following steps: Taking the target physical parameter as a pixel value channel, concatenating the pixel value channel with the RGB channel of the target image to obtain an input tensor; Inputting the input tensor into the encoder of the pre-trained generator to obtain image features output by the encoder; The image features are input into the decoder of the generator to obtain a result image output by the decoder.

[0009] According to a single image material physical parameter editing method based on a generative adversarial network provided by the present invention, the encoder includes a plurality of first convolution blocks connected in sequence and a residual block, and each of the first convolution blocks includes a convolution layer, an instance regularization layer and a rectified linear unit activation function connected in sequence.

[0010] According to a single image material physical parameter editing method based on a generative adversarial network provided by the present invention, the decoder includes a plurality of upsampling convolution blocks connected in sequence and a second convolution block, and each of the upsampling convolution blocks includes an upsampling layer, a convolution layer, an instance normalization layer and a rectified linear unit activation function connected in sequence.

[0011] According to a single image material physical parameter editing method based on a generative adversarial network provided by the present invention, the upsampling convolution block in the decoder is jump-connected to the first convolution block in the encoder.

[0012] According to a method for editing physical parameters of a single image material based on a generative adversarial network provided by the present invention, the method for editing physical parameters of a single image material based on a generative adversarial network further includes: The image to be tested is input into a pre-trained discriminator to obtain a physical parameter group output by the discriminator.

[0013] According to a single image material physical parameter editing method based on a generative adversarial network provided by the present invention, the generator and the discriminator are trained based on the following steps: Inputting sample data into the generator to obtain a result image output by the generator; Input the sample data and the result image into a discriminator respectively, and obtain a discriminant result output by the discriminator, wherein the discriminant result includes a first prediction result and a second prediction result, wherein the first prediction result indicates whether the sample data or the result image is a real image, and the second prediction result is a physical parameter group corresponding to the sample data or the result image; Determining a countermeasure loss value based on the first prediction result; Based on the second prediction result, determining a multi-label regression loss value; Based on the result image, determining a reconstruction loss value; Determining a total loss value of the discriminator based on the adversarial loss value and the multi-label regression loss value; Determining a total loss value of the generator based on the adversarial loss value, the multi-label regression loss value, and the reconstruction loss value; Based on the total loss value of the generator and the total loss value of the discriminator, the model parameters of the generator and the model parameters of the discriminator are updated respectively until a preset stop condition is reached.

[0014] The present invention also provides a single image material physical parameter editing device based on a generative adversarial network, comprising the following modules: An input tensor calculation module is used to: use the target physical parameter as a pixel value channel, and splice the pixel value channel with the RGB channel of the target image to obtain an input tensor; An image feature extraction module, used to: input the input tensor into an encoder of a pre-trained generator to obtain image features output by the encoder; The result image determination module is used to: input the image features into the decoder of the generator to obtain the result image output by the decoder.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a single image material physical parameter editing method based on a generative adversarial network as described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for editing physical parameters of a single image material based on a generative adversarial network as described above is implemented.

[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for editing physical parameters of single image materials based on a generative adversarial network.

[0018] The single image material physical parameter editing method based on the generative adversarial network provided by the present invention uses the target physical parameter as the pixel value channel, splices the pixel value channel with the RGB channel of the target image to obtain an input tensor; inputs the input tensor into the encoder of the pre-trained generator to obtain the image features output by the encoder; inputs the image features into the decoder of the generator to obtain the result image output by the decoder. This scheme embeds physical parameters into the generative adversarial network, uses the physical model as the parameterized representation of the material properties, and can only use a single image and the expected material physical properties as input. The material physical properties of the object in the original image are modified through the generative adversarial neural network, and an image is generated after the material appearance physical parameters are modified to the desired values, thereby achieving precise control of the material appearance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of a method for editing physical parameters of a single image material based on a generative adversarial network provided by the present invention; Figure 2 It is a structural schematic diagram of the generator provided by the present invention; Figure 3 It is a schematic diagram of the structure of the discriminator provided by the present invention; Figure 4 is an exemplary effect diagram of editing physical parameters provided by the present invention; Figure 5 It is a structural schematic diagram of a single image material physical parameter editing device based on a generative adversarial network provided by the present invention; Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be a connection between two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] Figure 1 is a flow chart of a method for editing physical parameters of a single image material based on a generative adversarial network provided by the present invention, such as Figure 1 As shown, the method includes the following: S110, taking the target physical parameter as a pixel value channel, and concatenating the pixel value channel with the RGB channel of the target image to obtain an input tensor; S120, inputting the input tensor into an encoder of a pre-trained generator to obtain image features output by the encoder; S130, inputting the image features into the decoder of the generator to obtain a result image output by the decoder.

[0025] It should be noted that the executor of the task construction method provided in the embodiment of the present application can be a server, a computer device, such as a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc.

[0026] Figure 2 It is a schematic diagram of the structure of the generator provided by the present invention, such as Figure 2 As shown, in the embodiment of the present invention, the generator adopts an encoder-decoder architecture, the input includes the target image and the target physical parameters, and the edited image is output. In order to achieve the goal of conditional generation, the model structure regards each control parameter as a pixel value channel and connects them with the RGB channels of the original image to form the input tensor of the generator.

[0027] The single image material physical parameter editing method based on the generative adversarial network provided by the present invention uses the target physical parameter as the pixel value channel, splices the pixel value channel with the RGB channel of the target image to obtain an input tensor; inputs the input tensor into the encoder of the pre-trained generator to obtain the image features output by the encoder; inputs the image features into the decoder of the generator to obtain the result image output by the decoder. This scheme embeds physical parameters into the generative adversarial network, uses the physical model as the parameterized representation of the material properties, and can only use a single image and the expected material physical properties as input. The material physical properties of the object in the original image are modified through the generative adversarial neural network, and an image is generated after the material appearance physical parameters are modified to the desired values, thereby achieving precise control of the material appearance.

[0028] In an optional embodiment, the encoder includes a plurality of first convolution blocks connected in sequence and a residual block, and each of the first convolution blocks includes a convolution layer, an instance regularization layer, and a rectified linear unit activation function connected in sequence.

[0029] like Figure 2 As shown in Figure 1, the encoder consists of multiple blocks, each of which contains a convolutional layer, an instance normalization layer, and a ReLU activation function. The encoder maps the input data to a high-dimensional space (in the form of a latent code) after convolution and regularization of multiple first convolutional blocks to extract image features. Finally, the encoder uses a residual block to improve model performance and reduce overfitting problems.

[0030] The present invention provides a single image material physical parameter editing method based on a generative adversarial network. The encoder of the generator extracts target image features through multiple blocks, and uses residual blocks to improve encoder performance and reduce overfitting problems.

[0031] In an optional embodiment, the decoder includes a plurality of sequentially connected upsampling convolution blocks and a second convolution block, each of the upsampling convolution blocks respectively including a sequentially connected upsampling layer, a convolution layer, an instance normalization layer and a rectified linear unit activation function.

[0032] like Figure 2 As shown in Figure 1, after feature extraction, the latent vector extracted by the encoder is upsampled using bilinear interpolation to restore the resolution. The decoder part also consists of multiple blocks, each of which contains an upsampling layer, a convolutional layer, an instance normalization layer, and a ReLU.

[0033] Furthermore, the upsampling convolution block in the decoder is jump-connected to the first convolution block in the encoder.

[0034] like Figure 2 As shown, in order to better preserve image details, skip connections are used in the decoder to connect the corresponding tensors of the encoder and decoder, and the tensors in the layers of the encoder that are earlier in the data flow order are connected to the tensors in the layers of the decoder that are later in the data flow order. The connected tensors are the same in dimension to minimize the loss caused by feature extraction. The model uses a convolutional layer to restore the connected feature maps to the original number of feature maps of the current block. The final output of the decoder is an image of the expected effect after material editing.

[0035] Specifically, the encoder includes five first convolution blocks, which are convolution block 1, convolution block 2, convolution block 3, convolution block 4, and convolution block 5 from left to right; the decoder includes five upsampling convolution blocks, which are upsampling convolution 1, upsampling convolution 2, upsampling convolution 3, upsampling convolution 4, and upsampling convolution 5 from left to right; the five first convolution blocks of the encoder are jump-connected with the five upsampling convolution blocks of the decoder, that is, the output of convolution block 5 is connected to the output of upsampling convolution 5 and then input to upsampling convolution 4, the output of convolution block 4 is connected to the output of upsampling convolution 4 and then input to upsampling convolution 3, the output of convolution block 3 is connected to the output of upsampling convolution 3 and then input to upsampling convolution 2, the output of convolution block 2 is connected to the output of upsampling convolution 2 and then input to upsampling convolution 1, and the output of convolution block 1 is connected to the output of upsampling convolution 1 and then input to the convolution block of the decoder to obtain the edited image as the final output.

[0036] The single image material physical parameter editing method based on generative adversarial network provided by the present invention jumps the decoder and the encoder to alleviate the gradient disappearance, reduce the learning difficulty of each layer of the model, and promote multi-scale feature fusion.

[0037] In an optional embodiment, the single image material physical parameter editing method based on a generative adversarial network further includes: The image to be tested is input into a pre-trained discriminator to obtain a physical parameter group output by the discriminator.

[0038] Figure 3 It is a schematic diagram of the structure of the discriminator provided by the present invention, such as Figure 3 As shown in the figure, the PatchGAN architecture is used, which is suitable for the needs of high-resolution details under the material editing of a single image, and it can both judge the authenticity of the image and predict its physical parameters. The fully connected layer of the traditional discriminator is replaced by a convolutional layer. After being processed by multiple blocks of convolutional layers, the output of the discriminator is a matrix, in which each element represents a block of the image. At the same time, the discriminator needs to judge the conditions to generate results. For this multi-label regression task, the discriminator generates a set of predicted attributes corresponding to the input image, thereby realizing the loss evaluation of various labels.

[0039] Furthermore, the generator and the discriminator are trained based on the following steps: Inputting sample data into the generator to obtain a result image output by the generator; Input the sample data and the result image into a discriminator respectively, and obtain a discriminant result output by the discriminator, wherein the discriminant result includes a first prediction result and a second prediction result, wherein the first prediction result indicates whether the sample data or the result image is a real image, and the second prediction result is a physical parameter group corresponding to the sample data or the result image; Determining a countermeasure loss value based on the first prediction result; Based on the second prediction result, determining a multi-label regression loss value; Based on the result image, determining a reconstruction loss value; Determining a total loss value of the discriminator based on the adversarial loss value and the multi-label regression loss value; Determining a total loss value of the generator based on the adversarial loss value, the multi-label regression loss value, and the reconstruction loss value; Based on the total loss value of the generator and the total loss value of the discriminator, the model parameters of the generator and the model parameters of the discriminator are updated respectively until a preset stop condition is reached.

[0040] In the embodiment of the present invention, the adversarial loss is a loss value determined by maximizing the judgment probability of the real image while minimizing the judgment probability of the generated image, thereby ensuring that the generator generates a real image that its discriminator cannot distinguish.

[0041] In the embodiment of the present invention, the multi-label regression loss is a loss value determined by the discriminator to correctly predict the target attribute from the generated image to promote adversarial training.

[0042] In the embodiment of the present invention, the reconstruction loss is used to change the appearance of the material while keeping other features of the image unchanged, and the image reconstruction loss is introduced to improve detail recovery.

[0043] In the embodiment of the present invention, the model parameters are updated by adversarial loss, multi-label regression loss, and reconstruction loss, which ensures that the quality of the editing results and other attributes remain unchanged. Material editing only requires an image and the expected combination of physical parameters, realizing a pure machine process from training to prediction without human intervention.

[0044] For ease of understanding, the following is a specific implementation process from training dataset construction to image generator and discriminator training, and then to the final single image material physical parameter editing. The overall workflow is: first, render a variety of images with a variety of lighting, geometry, and physical parameter combinations based on the selected physical parameter model, and use the set physical parameter combination as the dataset; when training the model, for the generator, the input is the original image and the controlled parameter group, and its output should be the image after adjusting the material physical parameters. For the discriminator, it is necessary to determine whether the image is a real image and predict the corresponding physical parameters of the image. The training uses adversarial loss to improve image quality, multi-label regression loss to achieve accurate prediction and control of physical parameters, and reconstruction loss to ensure that other image information except the material remains unchanged after editing; the trained model can accept the input image and the required physical parameter combination, thereby realizing material appearance editing based on physical parameters.

[0045] During the training dataset construction phase, the training of the generative adversarial network requires a large number of images as datasets. Therefore, it is necessary to construct an image dataset with physical parameter annotations, including a variety of environmental lighting covering indoor and outdoor or artificial and natural light sources, a variety of geometric shapes covering flat or complex surfaces, and a variety of physical parameter setting combinations.

[0046] In this embodiment, the physics-based BRDF model used as the appearance attribute representation of the material includes but is not limited to the GGX BRDF model, the Cook-Torrance BRDF model, the Ward BRDF model, etc. Taking the GGX BRDF model as an example, the BRDF model parameter set is defined as the control parameter of the neural network, where is the total reflectance, defined as , and They represent the proportion of energy in specular reflection and diffuse reflection respectively, and the ratio remains unchanged here. Used to control the width of the microsurface normal distribution, which represents the roughness level of a homogeneous material surface and is directly related to perceived glossiness. Controls the overall brightness and color of surface reflections. Represents the Fresnel reflectivity at normal incidence, which is a good indicator of the appearance of a metallic or non-metallic material. Three physical parameters replace the perceptual parameters for material appearance editing.

[0047] A physically based renderer is used to generate image datasets with accurate physical parameter annotations. Each physical control parameter is sampled at multiple levels within a specified range and normalized. Different RGB color values ​​are multiplied by Samples are used to give all material appearance samples various diffuse colors to ensure data diversity and coverage. Each image is labeled with the physical parameters at the time of rendering.

[0048] In the image generator and discriminator training phase, the generator and discriminator networks are trained mainly by backpropagating and updating the model weights through the definition of loss functions. In this scheme, adversarial loss, multi-label regression loss, reconstruction loss, and the combined final loss are defined.

[0049] Adversarial loss: The generator needs to generate real images that its discriminator cannot distinguish, so adversarial loss is needed to maximize the judgment probability of real images while minimizing the judgment probability of generated images. In order to avoid mode collapse and gradient disappearance, WGAN-GP adversarial loss and gradient penalty are used to optimize the generator and discriminator. Adversarial loss is defined as follows: ; in, Defined as the probability that the image output by the discriminator is a real image, It is a discriminator The input image is Defined as a generator According to the input image and control parameters The generated image, is a hyperparameter used to control the weight of the gradient penalty term. Adversarial loss can improve the realism of the final image.

[0050] Multi-label regression loss: The discriminator needs to correctly predict the target attributes from the generated images to facilitate adversarial training. Therefore, for the vector of control parameters, the multi-label regression loss is defined by the mean square error (MSE) to achieve accurate prediction of physical parameters, as shown in the following formula: ; in, represents the attribute predicted by the discriminator, The generated image is used as input to the discriminator middle.

[0051] Reconstruction loss: In order to change the appearance of the material while keeping other features of the image unchanged, image reconstruction loss is introduced to improve detail recovery. The L1 norm is used to measure the pixel difference between the input image and the generated image, as shown in the following formula: ; Among them, Generator generates images and original control parameter combinations Input to the generator In the original image And in order to separate the target attributes from other attributes in the latent space, a cycle consistency loss is adopted to ensure that the geometric and lighting information of the edited image remains unchanged.

[0052] Final loss: The optimized discriminator objective function is as follows: ; The objective function of the generator is as follows: ; in, and is a hyperparameter used to balance the contributions of different losses during training.

[0053] During training, the generator is fed with an image and a control parameter group, the discriminator is fed with a real image or a generated image, the combined final loss is calculated, and the generator and discriminator are trained simultaneously based on the final loss.

[0054] In the material physical parameter editing stage of a single image, the trained model can be used to predict and edit the material physical parameters at the same time. For material physical parameter prediction, only the discriminator needs to be used. The input is an image, and the control parameter group is output after the discriminator convolution operation. Even if the image is not rendered using the corresponding physical parameter model, the control parameter group that fits the corresponding physical parameter model can be obtained. For material physical parameter editing, such as Figure 4 As shown in the figure, only the generator needs to be used. The input is a combination of an image and the required control parameters. The latent space vector is obtained through the convolution block and residual block of the generator. Then, the shallow features are skipped through upsampling and convolution blocks. The final output is an image with the corresponding physical parameters modified, thereby realizing the editing of material effects.

[0055] In summary, the present invention embeds the physical parameters of the rendering model into the latent space of the generator, including a discriminator with multi-label regression loss, and a structure in which the encoder and decoder in the generator correspond to the jump connection of the latent vector layer. Only a single image is needed to modify the physical parameters to realize material editing. No complete information such as object information, model or lighting in the scene is required. The editing of the material appearance is realized by editing the physical parameters, providing continuous and controllable material appearance editing capabilities. In addition, the model training process of the present invention does not rely on manually labeled data, and the data set generation saves a large amount of manual data labeling. The automatically generated data set is directly used. The training ensures the quality of the editing results and the invariance of other attributes through a combination of multiple loss functions. The use of physical parameters embedded in the generative adversarial network avoids the problems of semantic inconsistency and attribute coupling in manual labeling of perceptual attributes, and the data set can be automatically generated.

[0056] The following is a description of the single image material physical parameter editing device based on a generative adversarial network provided in an embodiment of the present application. The single image material physical parameter editing device based on a generative adversarial network described below and the single image material physical parameter editing method based on a generative adversarial network described above can be referenced to each other.

[0057] Figure 5 is a schematic diagram of the structure of a single image material physical parameter editing device based on a generative adversarial network provided by the present invention, such as Figure 5 As shown, the single image material physical parameter editing device based on the generative adversarial network may include but is not limited to; An input tensor calculation module 510 is used to: use the target physical parameter as a pixel value channel, and concatenate the pixel value channel with the RGB channel of the target image to obtain an input tensor; An image feature extraction module 520 is used to: input the input tensor into an encoder of a pre-trained generator to obtain image features output by the encoder; The result image determination module 530 is used to: input the image features into the decoder of the generator to obtain the result image output by the decoder.

[0058] It should be noted that the single image material physical parameter editing device based on the generative adversarial network provided in the embodiment of the present invention can execute the single image material physical parameter editing method based on the generative adversarial network described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0059] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a single image material physical parameter editing method based on a generative adversarial network, the method comprising: taking the target physical parameter as a pixel value channel, splicing the pixel value channel with the RGB channel of the target image, and obtaining an input tensor; Inputting the input tensor into the encoder of the pre-trained generator to obtain image features output by the encoder; The image features are input into the decoder of the generator to obtain a result image output by the decoder.

[0060] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0061] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the single image material physical parameter editing method based on the generative adversarial network provided by the above methods, the method comprising: taking the target physical parameter as a pixel value channel, splicing the pixel value channel with the RGB channel of the target image to obtain an input tensor; Inputting the input tensor into the encoder of the pre-trained generator to obtain image features output by the encoder; The image features are input into the decoder of the generator to obtain a result image output by the decoder.

[0062] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the single image material physical parameter editing method based on a generative adversarial network provided by the above methods, the method comprising: taking the target physical parameter as a pixel value channel, splicing the pixel value channel with the RGB channel of the target image to obtain an input tensor; Inputting the input tensor into the encoder of the pre-trained generator to obtain image features output by the encoder; The image features are input into the decoder of the generator to obtain a result image output by the decoder.

[0063] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for editing physical parameters of material in a single image based on a generative adversarial network, characterized in that: include: Taking the target physical parameter as a pixel value channel, concatenating the pixel value channel with the RGB channel of the target image to obtain an input tensor; Inputting the input tensor into the encoder of the pre-trained generator to obtain image features output by the encoder; The image features are input into the decoder of the generator to obtain a result image output by the decoder.

2. The method for editing physical parameters of material in a single image based on a generative adversarial network according to claim 1, characterized in that: The encoder includes a plurality of first convolution blocks connected in sequence and a residual block, and each of the first convolution blocks includes a convolution layer, an instance regularization layer and a rectified linear unit activation function connected in sequence.

3. The method for editing physical parameters of material in a single image based on a generative adversarial network according to claim 2, characterized in that: The decoder includes a plurality of sequentially connected upsampling convolution blocks and a second convolution block, and each of the upsampling convolution blocks includes a sequentially connected upsampling layer, a convolution layer, an instance normalization layer and a rectified linear unit activation function.

4. The method for editing physical parameters of material in a single image based on a generative adversarial network according to claim 3, characterized in that: The upsampling convolution block in the decoder is skip-connected to the first convolution block in the encoder.

5. The method for editing physical parameters of material in a single image based on a generative adversarial network according to any one of claims 1 to 4, characterized in that: The single image material physical parameter editing method based on the generative adversarial network also includes: The image to be tested is input into a pre-trained discriminator to obtain a physical parameter group output by the discriminator.

6. The method for editing physical parameters of material in a single image based on a generative adversarial network according to claim 5, characterized in that: The generator and the discriminator are trained based on the following steps: Inputting sample data into the generator to obtain a result image output by the generator; Input the sample data and the result image into a discriminator respectively, and obtain a discriminant result output by the discriminator, wherein the discriminant result includes a first prediction result and a second prediction result, wherein the first prediction result indicates whether the sample data or the result image is a real image, and the second prediction result is a physical parameter group corresponding to the sample data or the result image; Determining a countermeasure loss value based on the first prediction result; Based on the second prediction result, determining a multi-label regression loss value; Based on the result image, determining a reconstruction loss value; Determining a total loss value of the discriminator based on the adversarial loss value and the multi-label regression loss value; Determining a total loss value of the generator based on the adversarial loss value, the multi-label regression loss value, and the reconstruction loss value; Based on the total loss value of the generator and the total loss value of the discriminator, the model parameters of the generator and the model parameters of the discriminator are updated respectively until a preset stop condition is reached.

7. A single image material physical parameter editing device based on a generative adversarial network, characterized in that: include: An input tensor calculation module is used to: use the target physical parameter as a pixel value channel, and splice the pixel value channel with the RGB channel of the target image to obtain an input tensor; An image feature extraction module, used to: input the input tensor into an encoder of a pre-trained generator to obtain image features output by the encoder; The result image determination module is used to: input the image features into the decoder of the generator to obtain the result image output by the decoder.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for editing physical parameters of a single image material based on a generative adversarial network as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for editing physical parameters of a single image material based on a generative adversarial network as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for editing physical parameters of a single image material based on a generative adversarial network as described in any one of claims 1 to 6 is implemented.

Images