Image material migration method based on neural rendering and depth prior

Abstract

The invention relates to an image migration method based on neural rendering and depth priori, and the method comprises the steps: 1, obtaining a group of discrete multi-view images of the same object in a static scene, the camera coordinates and orientation of the corresponding images, and a target material map; 2, building an object three-dimensional reconstruction network; 3, building a generative adversarial network for training material migration; 4, training the three-dimensional reconstruction network in the step 2 by using the training data set processed in the step 1, and training the material migration network in the step 3 based on the output of the network in the step 2, the training data set in the step 1 and various loss functions until the network converges; step 5, for an image to be migrated, carrying out migration processing by using the image migration model trained in the step 4 to obtain a final migrated image; the migration image generated by the method has high-quality material details, and the problems that an existing image migration method is blurred, multiple view angles are inconsistent and the like can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image material migration method based on neural rendering and depth prior. Background Art

[0002] Image migration tasks usually migrate images based on existing image content and migration targets, so that the migrated image has information of both the target image and the original image. It is widely used in art, film, games, digital humans and other fields. Therefore, the research and improvement of image migration technology has great practical significance and application value.

[0003] To transfer images, many image transfer methods have been proposed, which can be mainly divided into two categories: traditional single 2D image-based transfer methods and neural rendering-based 3D image transfer methods. Traditional single 2D image-based transfer methods mainly focus on stroke-based scene transfer, text-driven image transfer, and attribute-based face transfer, which greatly reduces the difficulty of artistic creation. However, despite the great success of 2D image transfer and neural rendering techniques, similar transfer capabilities in the 3D field are still limited: (1) they require laborious annotations such as image masks and mesh vertices to achieve the desired operations; (2) they perform global style transfer while ignoring the semantic meaning of each object part; (3) they can transfer categories by learning textured 3D latent representations, or transfer at a coarse level with basic color assignment or object-level disentanglement, but it is difficult to transfer materials for objects with photo-realistic textures or out-of-distribution features.

[0004] With the development of neural rendering methods in 3D reconstruction, people have introduced 2D image migration methods to 3D images and made significant progress. The image migration method based on neural rendering can utilize image information from new perspectives, so as to understand the spatial structure of the object area to be migrated. However, the existing image migration algorithms based on neural rendering still have problems. For example, the manually designed loss function cannot effectively understand the target image and cannot fully utilize the semantic and structural features of the target image. In addition, the operation of existing network migration is laborious, and users cannot edit 3D scenes on a single 2D image in a convenient way. To make the results of multiple migrations consistent, manual pixel alignment or repeated editing from multi-view images is required. Therefore, when facing the multi-view migration image of a specified semantic area, how to ensure the consistency of the generated migration image material information with the target image and the consistency of the entire image under multiple perspectives is still a challenge. Summary of the invention

[0005] In response to the above-mentioned problems existing in the existing three-dimensional image migration methods, the present invention proposes an image material migration method based on neural rendering and depth prior. When faced with multi-view images, this method can fully extract the spatial structure information and global semantic information of the image data, enhance the model migration quality, and improve the multi-view consistency of the migrated image, so that the migrated image has more texture details.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for image material migration based on neural rendering and depth prior, the migration method comprises the following steps:

[0008] Step 1: Obtain a set of discrete multi-view images of the same object in a static scene, a semantic segmentation map of the corresponding image area to be subjected to material migration, the image after migration, the camera coordinates and camera orientation of the corresponding image, and the RGB map and normal map of the target material. The multi-view images can be composed of video frames extracted from real objects or random perspective rendering by modeling software. The camera coordinates and camera orientation of the corresponding images can be obtained by feature extraction by tools such as COLMAP or rendering output by modeling software. The material image comes from Internet resources;

[0009] Step 2: Building a 3D reconstruction network for the object. The 3D reconstruction network inputs the multi-view images in step 1 and the corresponding camera coordinates and orientations, and outputs the reconstructed image, depth, and semantic segmentation image of the object.

[0010] Step 3: Building a generative adversarial network for training material migration, the input of the generative adversarial network for material migration is the reconstructed image, depth image and semantic image mentioned in step 2, and the material map mentioned in step 1, performing material migration on the image, and directly outputting the migrated image;

[0011] Step 4: Use the training data set processed in step 1 and the 3D reconstruction network in step 2 to train, and then train the material migration network in step 3 based on the output of the network in step 2, the training data set in step 1 and multiple loss functions until the network converges;

[0012] Step 5: For images that require material migration, use the material migration model trained in step 4 to perform material migration to obtain the final migrated image.

[0013] Furthermore, the adversarial generative network for material migration in step 3 includes a generator and a discriminator. The generator is used to migrate images, and the discriminator makes the probability distribution of the generated migrated image material information approach that of the target material information, while ensuring the consistency of multiple perspectives. The generator includes three parts: an encoder, a constraint layer and a decoder. The NLayer discriminator mainly includes a multi-layer convolutional neural network, and uses 4×4 and 3×3 convolution kernels for feature extraction and judgment.

[0014] Furthermore, the specific structure of the generator is:

[0015] The encoder uses a spatial attention module to extract the spatial structure information of the depth image data and the image material information;

[0016] The constraint layer uses a gated neural network to constrain material properties for visual perception;

[0017] The decoder uses a spatial attention module to fuse spatial information and material information and outputs an image after material migration.

[0018] Furthermore, the loss function in step 4 includes two parts: the generator and the discriminator. The loss function of the generator consists of four parts: migration loss, perception loss, adversarial loss and material loss, and the loss function of the discriminator includes adversarial loss and material loss. The loss function formulas are as follows:

[0019] L gen =λ tra L tra +λ percep L percep +λ adv L adv,G +λ mat，G L mat

[0020] L dis =λ mat,D L mat +λ adv L adv,D

[0021] Among them, L gen and L dis Denote the loss functions of the generator and discriminator respectively, λ tra , percep , adv , mat，G and λ mat，D They represent the weights of transfer loss, perceptual loss, adversarial loss, generator material loss, and discriminator material loss respectively; L tra represents the migration loss, L percep represents the perceptual loss, L adv,Grepresents the adversarial loss of the generator, L mat Indicates material loss, L adv，D Represents the adversarial loss of the discriminator. The specific loss function formula is as follows:

[0022] The migration loss formula is as follows:

[0023]

[0024] Among them, I gt represents the original image before migration, I out Represents the output result of the migration model.

[0025] The perceptual loss function is calculated based on the pre-trained VGG-19 network, and the formula is as follows:

[0026]

[0027] Where N represents the total number of layers in the pre-trained network, and F i (x) represents the input image I gt The output feature of the i-th layer, F i (y) represents the output result I at layer i out .

[0028] The adversarial loss formula is as follows:

[0029]

[0030] Among them, L adv，G represents the adversarial loss of the generator, L adv，D represents the adversarial loss of the discriminator, I out Represents the output result of the migration model, I gt represents the original image before migration, P x (i) represents the true probability distribution.

[0031] Furthermore, the training process adds depth prior as a guide and adds a constraint layer to align spatial information under multiple perspectives, so that the decoding module in the generator can fully integrate the spatial structure information of the input data, enhance the model migration quality, and improve the multi-perspective consistency of the migrated image.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] (1) The present invention proposes an image material migration method based on neural rendering and depth prior. To address the semantic and structural inconsistency problems in existing image material migration methods, the present invention uses depth prior information and a material constraint module to achieve efficient material migration in a specified semantic area, thereby ensuring the quality and consistency of the migrated image.

[0034] (2) The present invention proposes an attention-based gated convolutional generative adversarial network (MST). The model consists of a spatial attention module (SMST) and a gated convolutional constraint block (CMST), which enables the network to fully extract the spatial structural features of the original image, the semantic features of the prior information, and the semantic features of the material constraint information at the encoding layer, further improving the semantic consistency of the migrated image.

[0035] (3) The present invention uses depth maps of three-dimensional images at different viewing angles as model priors. The depth prior enables the attention mechanism and constraint module to focus on key information, effectively obtain spatial structure information, and solve the quality problem of the migrated image. At the same time, an attention-based upsampling module is constructed to integrate the low-size features of the previous level and the constraint information of the constraint module to enhance the multi-view consistency of the migrated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0037] Figure 1 A flowchart of an image migration method based on neural rendering and depth prior provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a generative adversarial network for image migration according to an embodiment of the present invention;

[0039] Figure 3 , Figure 4 This is an effect diagram of material migration of a multi-view image of a three-dimensional object provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Figure 1 Flowchart of an image material migration method based on neural rendering and depth prior according to an embodiment of the present invention. Figure 1 As shown, the present invention provides an image material migration method based on neural rendering and depth prior, the method comprising the following steps:

[0042] Step 1: Obtain a set of discrete multi-view images of the same object in a static scene, the semantic segmentation map of the corresponding image area to be subjected to material migration, the migrated image, the camera coordinates and camera orientation of the corresponding image, and the RGB map and normal map of the target material.

[0043] The original image used in the present invention comes from the NeRF_Synthetic dataset. The semantic segmentation map and the migrated image are produced using the modeling software Blender. The BlenderNerf plug-in needs to be installed, and the viewing angle information of the original dataset needs to be imported, and the material node information of the material migration area needs to be changed. The node information of the semantic segmentation map is the self-luminous shader, and the remaining shaders are cancelled. The migrated image uses a principled shader, and an automated script is used to change the material map, normal map, metalness, roughness and high light refractive index. The metalness and roughness set by the present invention are 0%, 25%, 50%, 75%, and 100%, respectively, and then rendered according to the imported viewing angle information, and then all images are unified in resolution to 512×512.

[0044] Step 2: Build the object 3D reconstruction network.

[0045] The 3D reconstruction network inputs the multi-view image and the corresponding camera coordinates and orientation of step 1, and after passing through eight fully connected networks, it outputs the reconstructed image of the object through the color branch, the pixel density branch outputs the depth, and the semantic prediction branch outputs the semantic segmentation image.

[0046] Step 3: Build a generative adversarial network for training material transfer.

[0047] Figure 2 A schematic diagram of a generative adversarial network for image material migration provided in an embodiment of the present invention. The input of the generative adversarial network for material migration is the reconstructed image, depth image and semantic image mentioned in step 2, as well as the material map mentioned in step 1. The image is material migrated and the migrated image is directly output; the constructed image material migration adversarial network includes a generator and a discriminator. The generator is used to migrate the image, and the discriminator makes the migrated image have multi-view consistency and more material details. The generator includes three parts: an encoder, a constraint layer, and a decoder. The encoder uses a spatial attention module to extract the spatial structure information of the original image and prior information; the constraint layer uses a gated convolution module for the alignment of image data under multiple perspectives; the decoder uses a spatial attention module for the fusion of spatial information and material information, and outputs the image after material migration. The discriminator is an NLayer discriminator, which mainly includes a multi-layer convolutional neural network, and uses 4×4 and 3×3 convolution kernels for feature extraction and judgment.

[0048] like Figure 2As shown, the present invention is specifically implemented as follows: the network first accepts the image to be migrated, the semantic segmentation map of the migration area, the depth map of the migration image, the material map and the normal map. Then the migration image and the depth map are segmented according to the segmentation map, the corresponding area of ​​the migration image and the corresponding area of ​​the retained depth map are removed, and then they are added. The combined image is then sent to the encoder for feature extraction. The encoder is composed of 5 layers of SMST blocks, and each layer of SMST blocks uses a spatial attention module to capture the spatial structure information of the combined image and the multi-level information of the material map and the normal map. In addition, the present invention also fuses the image information and the material information through a constraint layer composed of several CMSTs to enhance the features. The CMST block uses a gated convolution module to extract the global semantics and structural information of the material information. Next, the decoder fuses the spatial information and the semantic information and outputs the final migration image. The decoder is composed of 4 layers of spatial attention convolution blocks and an upsampling module based on self-attention. A U-shaped skip connection is used. The skip connection fuses the features output by the encoder of the corresponding layer and the material information constrained by the CMST module and then sends them to the decoder corresponding to the encoder, so as to better migrate the image of the shallow features of the encoder. Finally, the features are mapped to the real image distribution through a two-dimensional convolution with a kernel size of 7×7 to obtain the migrated image.

[0049] Step 4: Use the training data set processed in step 1 to train the 3D reconstruction network in step 2, and then train the material migration network in step 3 based on the output of the network in step 2, the training data set in step 1 and multiple loss functions until the network converges.

[0050] The loss function in step 4 includes two parts: the generator and the discriminator. The loss function of the generator consists of four parts: transfer loss, perception loss, adversarial loss, and material loss. The loss function of the discriminator includes adversarial loss and material loss. The loss function formulas are as follows:

[0051] L gen =λ tra L tra +λ percep L percep +λ adv L adv,G +λ mat,G L mat

[0052] L dis =λ mat,D L mat +λ adv L adv,D

[0053] Among them, L gen and L disDenote the loss functions of the generator and discriminator respectively, λ tra , percep , adv , mat,G and λ mat,D They represent the weights of transfer loss, perceptual loss, adversarial loss, generator material loss, and discriminator material loss respectively; L tra represents the migration loss, L percep represents the perceptual loss, L adv,G represents the adversarial loss of the generator, L mat Indicates material loss, L adv,D represents the adversarial loss of the discriminator.

[0054] According to the image material migration method based on neural rendering and deep prior fusion according to claim 4, it is characterized in that the adversarial loss function can make the migrated image have multi-view consistency and more material details by introducing a discriminator for true and false judgment, and the specific calculation process of the loss function is as follows:

[0055] The migration loss formula is as follows:

[0056]

[0057] Among them, I gt represents the original image before migration, I out Represents the output result of the migration model.

[0058] The perceptual loss function is calculated based on the pre-trained VGG-19 network, and the formula is as follows:

[0059]

[0060] Where N represents the total number of layers in the pre-trained network, and F i (x) represents the input image I gt The output feature of the i-th layer, F i (y) represents the output result I at layer i out .

[0061] The adversarial loss formula is as follows:

[0062]

[0063] Among them, L adv,G represents the adversarial loss of the generator, L adv,D represents the adversarial loss of the discriminator, I out Represents the output result of the migration model, I gt Indicates that there is no original image before migration.

[0064] The training process adds depth prior as a guide and adds a constraint layer to align spatial information under multiple perspectives, so that the decoding module in the generator can fully integrate the spatial structure information of the input data, enhance the model migration quality, and improve the multi-perspective consistency of the migrated image.

[0065] Step 5: For the image to be migrated, use the image migration model trained in step 4 to perform migration processing to obtain the final migration image.

[0066] Figure 3 , Figure 4 This is a diagram showing the effect of image migration provided by an embodiment of the present invention.

[0067] like Figure 3 , Figure 4 As shown in the figure, the seat surface material migration results of the NeRF_Synthetic dataset are shown. After the introduction of the depth prior, the trained image migration generative adversarial network can obtain a synthetic image with clear details, clear contours and multi-view consistency. Specifically, the key parts such as the texture, highlights, and edges of the seat surface after migration all show high-quality details and multi-view consistency, which further proves the effectiveness of the image migration method based on neural rendering and depth prior proposed by the present invention.

[0068] The present invention adopts an attention-based gated convolution block, which is composed of two layers of gated convolution kernels and one layer of attention. In the different stages of the encoder-constrained layer-decoder, the present invention takes advantage of spatial attention and self-attention, which are used to learn depth information, constrain material information, and extract global semantic and structural information, respectively. In summary, the present invention achieves efficient migration of multi-view images of three-dimensional objects by combining spatial attention modules and self-attention modules, enhances the quality of model migration, and improves the multi-view consistency of migrated images.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than alignment limitations. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image material migration method based on neural rendering and deep prior fusion, characterized in that: The following steps are involved: Step 1: Obtain a set of discrete multi-view images of the same object in a static scene, a semantic segmentation map of the corresponding image area to be subjected to material migration, the image after migration, the camera coordinates and camera orientation of the corresponding image, and the RGB map and normal map of the target material. The multi-view images can be composed of video frames extracted from real objects or random perspective rendering by modeling software. The camera coordinates and camera orientation of the corresponding images can be obtained by feature extraction by tools such as COLMAP or rendering output by modeling software. The material image comes from Internet resources; Step 2: Building a 3D reconstruction network for the object. The 3D reconstruction network inputs the multi-view images in step 1 and the corresponding camera coordinates and orientations, and outputs the reconstructed image, depth, and semantic segmentation image of the object. Step 3: Building a generative adversarial network for training material migration, the input of the generative adversarial network for material migration is the reconstructed image, depth image and semantic image mentioned in step 2, and the material map mentioned in step 1, performing material migration on the image, and directly outputting the migrated image; Step 4: Use the training data set processed in step 1 and the 3D reconstruction network in step 2 to train, and then train the material migration network in step 3 based on the output of the network in step 2, the training data set in step 1 and multiple loss functions until the network converges; Step 5: For images that require material migration, use the material migration model trained in step 4 to perform material migration to obtain the final migrated image.

2. The material migration method based on neural rendering and depth prior fusion according to claim 1, characterized in that: The material migration adversarial generative network in step 3 includes a generator and a discriminator. The generator includes three parts: an encoder, a constraint layer and a decoder. The NLayer discriminator mainly includes a five-layer convolutional neural network, which uses 4×4 and 3×3 convolution kernels for feature extraction and judgment.

3. The method according to claim 2, characterized in that The generator comprises: The encoder adopts a spatial attention module and a spatial attention mechanism to extract the spatial structure information of the depth image data and the image material information; The constraint layer adopts a gated convolution module, which can align spatial information under multiple perspectives and also be used for material attribute constraints of visual perception through depth prior guidance; The decoder adopts a spatial attention module to fuse spatial information and material information and outputs an image after material migration.

4. The image material migration method based on neural rendering and depth prior fusion according to claim 1 is characterized in that: The loss function in step 4 includes two parts: the generator and the discriminator. The loss function of the generator consists of four parts: migration loss, perception loss, adversarial loss and material loss, and the loss function of the discriminator includes adversarial loss and material loss. The loss function formulas are as follows: L gen =λ tra L tra +λ percep L percep +λ adv L adv,G +λ mat,G L mat L dis =λ mat, D Lmat +λ adv L adv,D Among them, L gen and L dis Denote the loss functions of the generator and discriminator respectively, λ tra , percep , adv , mat,G and λ mat,D They represent the weights of transfer loss, perceptual loss, adversarial loss, generator material loss, and discriminator material loss respectively; L tra represents the migration loss, L percep represents the perceptual loss, L adv,G represents the adversarial loss of the generator, L mat Indicates material loss, L adv,D represents the adversarial loss of the discriminator.

5. The image material migration method based on neural rendering and depth prior fusion according to claim 4 is characterized in that: The adversarial loss function introduces a discriminator for true and false judgment, which can make the migrated image have multi-view consistency and more material details. The specific calculation process of the loss function is as follows: The migration loss formula is as follows: Among them, I gt represents the original image before migration, I out Represents the output result of the migration model. The perceptual loss function is calculated based on the pre-trained VGG-19 network, and the formula is as follows: Where N represents the total number of layers in the pre-trained network, and F i (x) represents the input image I gt The output feature of the i-th layer, F i (y) represents the output result I at layer i out . The adversarial loss formula is as follows: Among them, L adv,G represents the adversarial loss of the generator, L adv,D represents the adversarial loss of the discriminator, I out Represents the output result of the migration model, I gt represents the original image before migration, P x (i) represents the true probability distribution.

6. The method for image material migration based on neural rendering and depth prior according to claim 5, characterized in that: By adding depth prior, the training process can enable the decoding module in the generator to fully integrate the spatial structure information of the input data, enhance the model migration quality, and improve the multi-view consistency of the migrated image.

Images