Single image material recovery method based on CBAM and adversarial network

Abstract

The invention relates to the field of image generation material mapping, and discloses a single image material recovery method based on CBAM and an adversarial network, and the method comprises the following steps: S1, a network structure, S2, a CBAM attention module, and S3, a joint loss function. According to the method, a new joint loss function is provided, pixel-level loss and perception loss are combined by introducing the new joint loss function, so that high-level semantic features and pixel-level information are learned, loss based on human perception is introduced, the estimation quality is further improved, and the estimation accuracy is improved. An attention mechanism based on a spatial domain and a channel domain is introduced, the attention mechanism helps the network to improve attention on important features, reduce attention on less important features and enhance the ability of the network to capture related image details, and an end-to-end framework based on a CGAN is established; and the network can generate a more accurate SVBRDF material map which accords with expectation.

Description

Technical Field

[0001] The present invention relates to the field of image generation of material maps, and in particular to a single image material restoration method based on CBAM and an adversarial network. Background Art

[0002] In the real world, the visual effects of object surfaces are affected by factors such as lighting conditions, surface geometry, and material properties. In order to achieve more realistic rendering effects, this makes generating high-quality SVBRDF maps from a single image a challenging and urgent problem to be solved. Recently, methods for generating material maps from images captured by mobile devices (such as mobile phones) have become increasingly popular, as such methods can effectively reduce costs and improve efficiency compared to traditional methods.

[0003] The present invention proposes an end-to-end method for accurately estimating SVBRDF material parameters from a single image. Unlike some existing methods based on convolutional neural networks, these methods only use pixel-based loss functions (such as L1 Loss) to optimize network parameters. These methods are insufficient when dealing with materials with complex features. Therefore, the method proposed in the present invention breaks through this limitation by introducing a new joint loss function. The joint loss function introduced in the present invention enables the network to optimize parameters based on the high-level semantic features of the material image by adding perceptual loss. Not only that, a loss function that can capture global structure and shape features is also added to encourage the network to generate results that are similar to the target image in overall structure. This allows the network to not pay too much attention to the differences in the pixel level of the input image, thereby improving the perceptual accuracy and visual rationality of the estimated SVBRDF.

[0004] In addition, this method introduces an attention mechanism module CBAM based on the spatial domain and channel domain. Compared with some previous CNN-based methods, the addition of the attention module effectively improves the network's ability to segment each feature, thereby helping the network focus on more important features, such as changes in highlights and shadows or texture patterns, while reducing the impact of irrelevant and interfering features in the image. In general, the introduction of this attention mechanism significantly improves the performance of the network and its ability to extract complex detail features, thereby helping to improve the accuracy of SVBRDF estimation overall.

[0005] In order to further improve the authenticity of the generated results, this method introduces a conditional generative adversarial network (CGAN) in the network structure. Since CGAN introduces additional constraints in the network input, it can obtain more specific information during the training process and better meet the task requirements of the present invention. In the implementation of the present invention, the input target image is used as the constraint condition in this method, and the SVBRDF texture map generated by the generator is used as the texture map of the target image. Figure 1It is added to the training process of the discriminator to achieve more effective adversarial training. In order to improve the accuracy of each material map, this method introduces multiple discriminators, each of which is responsible for discriminating the corresponding SVBRDF material map. This multi-discriminator setting improves the overall fitting ability of the network and strives to ensure that the generated results are consistent with the expected properties. By making full use of conditional adversarial training and multiple discriminators, this method improves the quality and accuracy of the generated SVBRDF map and produces more realistic rendering results. To this end, a method for accurately estimating surface appearance SVBRDF material maps from a single image is disclosed. Summary of the invention

[0006] The present invention proposes a new joint loss function: by introducing a new joint loss function, the pixel-level loss is combined with the perceptual loss to learn high-level semantic features and pixel-level information, and introduces the loss based on human perception to further improve the quality of estimation, and introduces an attention mechanism based on the spatial domain and channel domain: this attention mechanism helps the network to increase its attention to important features and reduce its attention to less important features, thereby enhancing the network's ability to capture relevant image details, and establishes an end-to-end framework based on CGAN: by designing the overall CNN structure and introducing CGAN, the network is able to generate more accurate and expected SVBRDF material maps.

[0007] To achieve the above object, the present invention provides the following technical solution: a single image material restoration method based on CBAM and adversarial network, the method comprising the following steps:

[0008] S1. Network structure

[0009] By inputting the material parameters to be estimated and trained into the generator, the generator renders the generated SVBRDF material map into a new image for discrimination using random lighting, and multiple discriminators respectively determine the authenticity of each material map and rendered image. The input image is also added to the discriminator's judgment basis as a limiting condition of CGAN. The entire process optimizes the network parameters through the joint loss function set by the present invention.

[0010] (1) Generator

[0011] The generator of this method is mainly composed of an encoder and a decoder. Each layer of the encoder and the decoder corresponds to each other, with a total of 8 corresponding levels, and a splicing operation is used between the corresponding layers to transfer the results of feature extraction. This design minimizes the features after multi-layer network extraction.

[0012] (2) Discriminator

[0013] The discriminator is composed of three independent layer discriminators to form a joint discriminator with multi-scale discrimination function. The results of the layer discriminators are downsampled through the middle average pooling layer, and each independent layer discriminator judges the authenticity of the input features based on image features of different sizes. Then the results of the three layer discriminators are combined as the final judgment result.

[0014] S2, CBAM attention module

[0015] The attention mechanism has received extensive attention in deep learning. This method uses CBAM, which affects both spatial and channel features, to help the network optimize parameters. As a lightweight module, CBAM effectively affects spatial and channel information, and is very suitable for integration into CNN-based architectures and is widely used in various image and visual processing fields. By effectively utilizing the spatial and channel information of input features, CBAM further enhances the feature extraction capability of CNN networks. Experimental results show that CBAM effectively enhances intermediate features from multiple angles while suppressing irrelevant intermediate features. This makes CBAM very suitable for the task of the present invention, such as when the input sample image contains some interference, such as overexposed highlights and undersaturated shadows, which poses a challenge to CNN in extracting useful information. Therefore, the performance and efficiency of the network are significantly improved by introducing CBAM.

[0016] For the input feature map F, the feature passes through the channel attention module M c To obtain the channel feature map. The product of the channel feature map and F is represented as F1, and then F1 is input into the spatial attention module M s The spatial feature map is obtained from the spatial feature map, and the product of the spatial feature map and F1 is F2. F2 is input into the next layer of the network as the adjusted feature map. The whole process is described by formula (1-1) and formula (1-2). Represents element-wise dot product.

[0017]

[0018] In the channel attention module, the input feature map F passes through the maximum pooling layer and the average pooling layer at the same time. Then, the results of the two pooling operations enter the MLP layer, and then the two extracted features are combined and input to the sigmoid layer to obtain the channel attention weight map. The weight map is element-wise multiplied with the original input feature to obtain the channel attention feature map F1;

[0019] S3. Joint loss function

[0020] The loss function plays a crucial role in improving the effectiveness and efficiency of network learning. Previous methods usually rely on pixel-based L1 Loss to optimize the network by directly comparing the difference between the generated results and the real results. However, this method limits the network's ability to obtain features from deeper hierarchical structures.

[0021] In order to break this limitation, the present invention proposes a new joint loss function, which combines perceptual loss and pixel loss from different aspects, and introduces a loss function based on image structural similarity. By introducing this joint loss function, the network can consider more factors when optimizing parameters, which helps to generate higher quality and visually reasonable material maps.

[0022] The joint loss function of the present invention is expressed by formula (1-3):

[0023]

[0024] in, represents the total loss function, Respectively represent the loss functions for the four material maps (normal, diffuse, roughness, and highlight) of SVBRDF, represents the loss calculated using the rendering result of the generated material map, that is, the rendering loss, λ r Indicates the weight of rendering loss. In the process of experiment, it is found that r The setting of weights affects the quality of the final result. r Set to 1, 3, 10, find λ r Too low or too high will reduce the accuracy of the SVBRDF map. Therefore, in the implementation process of the present invention, λ r The weight of is set to 5.

[0025] Definition of the invention Expressed by formula (1-4):

[0026]

[0027] Among them, M i Represents the actual material map of the target image, Represents the corresponding material map generated by the generator, represents the loss function implemented by MS-SSIM, which is used to evaluate the structural similarity between the generated image and the target image, thereby enhancing the model's attention to global structural features and making the results more consistent with the perception of the human eye. It is a perceptual loss function that compares the similarity between features by directly comparing the differences in the features of each layer of the discriminator. In the implementation process, L1Loss is used to compare the differences in each layer. This loss function enables the network to optimize the model parameters from the feature perspective, thereby expanding the previous method of using only pixel-level loss as a loss function. Similarly, the present invention also uses L1Loss to directly calculate the similarity between the generated result and the target image. It is about the adversarial loss of the material map, which will be introduced in the following content. λ1, λ2, λ3 and λ4 are the weights of each loss function, which are set to 5, 2, 10 and 1 respectively. Because in the experiment, it was found that the loss function based on pixel comparison directly has a greater impact on the accuracy of the result, so a higher weight is given. Although the weight of the loss function based on the degree of feature similarity is not as high as the former, it still helps to improve the accuracy of the result.

[0028] For adversarial loss The optimization strategy of the present invention is expressed by formulas (1-5) and (1-6):

[0029]

[0030] The discriminator and generator use equations (1-5) and (1-6) to implement adversarial training strategies respectively, where G represents the generator and D represents the discriminator. This training strategy not only includes SVBRDF material mapping, but also uses the same strategy for rendering results. The generator's results and the target image's material mapping use the same lighting conditions, that is, randomly selecting point light sources on the upper hemisphere, and the rendering results are based on the Cook-Torrance model, so that the rendering results are more in line with physical rules and reduce unexpected errors.

[0031] Although directly evaluating the differences in material parameters has achieved good results, this strategy alone cannot ensure consistent results after rendering, so it is necessary to add rendering loss To ensure that material textures maintain high quality consistency with ground truth when combined, The definition is expressed by formula (1-7):

[0032]

[0033] and compared to, Joined It is a perceptual loss that uses the VGG pre-trained network as the evaluation standard. The images to be compared are input into the VGG network, and the overall perceptual difference between the images is calculated by comparing the feature differences extracted from each layer of the VGG network. In the implementation details of the invention, VGG19 is used as the pre-trained network model, and its weight parameter λ5 is set to 2.

[0034] Preferably, in step S1, the feature extraction layer of the encoder module first passes through a 4×4 convolution layer and a 3×3 convolution layer, each convolution layer is followed by an instance normalization layer and a Leaky ReLU activation function, the first part consists of a 4×4 convolution kernel with a step size of 2, an instance normalization layer and a Leaky ReLU layer, then, additional information is extracted through 1×1, 3×3, and 1×1 convolution layers, while reducing the parameters to be learned, each layer is followed by an instance normalization layer and a Leaky Relu activation function, then, the input features pass through a module (CBAM) that applies attention to channel and spatial features respectively to enhance the focus on valid features and suppress unnecessary information, the features from the attention module are then added to the input features of the previous part to prevent excessive feature loss, and finally, the merged features are input to the final standardization layer.

[0035] Preferably, in step S1, in the decoder, the present invention merges the encoder feature information and the decoder input features of the same level, and then inputs them into the upsampling layer, and then, after the normalization layer and the Dropout layer, outputs the features to the next layer for further decoding.

[0036] Preferably, the structure of each independent layer discriminator is as shown in Table 3.1. The layer discriminator includes a total of 6 convolutional layers of different sizes, which also include partial normalization layers and Leaky ReLU activation functions. In the first 4 convolutional layers of the layer discriminator, Leaky ReLU activation functions are designed for nonlinear transformation. Batch normalization layers and Leaky ReLU activation functions are added after the 5th convolutional layer, and the 6th convolutional layer maps the true probability of the input samples through convolution operations.

[0037] Table 3.1 Layer Discriminator Network Structure Parameters

[0038]

[0039] Preferably, in the step S2, in the spatial attention module, the input feature F1 is continuously passed through a maximum pooling layer and an average pooling layer, and then the result is applied to a 2×2 convolution layer to generate a spatial attention weight map. Finally, the spatial attention weight map is multiplied with the channel attention feature map F1 to obtain the final feature map F2 with comprehensive performance.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. Propose a new joint loss function: By introducing a new joint loss function, the pixel-level loss is combined with the perceptual loss to learn high-level semantic features and pixel-level information, and the loss based on human perception is introduced to further improve the quality of estimation;

[0042] 2. Introducing the attention mechanism based on spatial domain and channel domain: This attention mechanism helps the network to pay more attention to important features and reduce attention to less important features, thus enhancing the network's ability to capture relevant image details;

[0043] 3. An end-to-end framework based on CGAN was established: By designing the overall CNN structure and introducing CGAN, the network can generate more accurate and expected SVBRDF material maps. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A material estimation network structure diagram of the present invention;

[0045] Figure 2 It is a schematic diagram of the structure of the generator module of the present invention;

[0046] Figure 3 Schematic diagram of the structure of the discriminator module of the present invention;

[0047] Figure 4 It is a structural diagram of the CBAM attention module of the present invention;

[0048] Figure 5 It is a visualization result diagram of various baseline methods of the present invention on synthetic data;

[0049] Figure 6 It is a visualization result diagram of various baseline methods of the present invention on real pictures;

[0050] Figure 7 This is the ablation experiment of the present invention: the influence diagram of the CBAM module on the restoration effect;

[0051] Figure 8 This is the ablation experiment of the present invention: a diagram showing the influence of some loss functions on the results. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The present invention provides a single image material restoration method based on CBAM and adversarial network, and the method comprises the following steps:

[0054] S1. Network structure

[0055] like Figure 1 As shown, by inputting the material parameters to be estimated and trained into the generator, the generator renders the generated SVBRDF material map into a new image for discrimination using random lighting, and multiple discriminators respectively determine the authenticity of each material map and rendered image, and the input image is also added to the discriminator's judgment basis as a limiting condition of CGAN. The entire process optimizes the network parameters through the joint loss function set by the present invention.

[0056] (1) Generator

[0057] like Figure 2 As shown in the figure, the generator of this method is mainly composed of an encoder and a decoder. Each layer of the encoder and the decoder corresponds to each other, and there are 8 corresponding levels in total. A splicing operation is used between the corresponding layers to transfer the results of feature extraction. This design minimizes the features after multi-layer network extraction.

[0058] (2) Discriminator

[0059] like Figure 3 As shown in the figure, the discriminator is composed of three independent layer discriminators to form a joint discriminator with multi-scale discrimination function. The results of the layer discriminators are downsampled through the average pooling layer in the middle, and each independent layer discriminator judges the authenticity of the input features for image features of different sizes. Then the results of the three layer discriminators are combined as the final judgment result.

[0060] S2, CBAM attention module

[0061] The attention mechanism has received extensive attention in deep learning. This method uses CBAM, which affects both spatial and channel features, to help the network optimize parameters. As a lightweight module, CBAM effectively affects spatial and channel information, and is very suitable for integration into CNN-based architectures and is widely used in various image and visual processing fields. By effectively utilizing the spatial and channel information of input features, CBAM further enhances the feature extraction capability of CNN networks. Experimental results show that CBAM effectively enhances intermediate features from multiple angles while suppressing irrelevant intermediate features. This makes CBAM very suitable for the task of the present invention, such as when the input sample image contains some interference, such as overexposed highlights and undersaturated shadows, which poses a challenge to CNN in extracting useful information. Therefore, the performance and efficiency of the network are significantly improved by introducing CBAM.

[0062] The structure of CBAM is as follows Figure 4 As shown, for the input feature map F, the feature passes through the channel attention module M c To obtain the channel feature map. The product of the channel feature map and F is represented as F1, and then F1 is input into the spatial attention module M s The spatial feature map is obtained from the spatial feature map, and the product of the spatial feature map and F1 is F2. F2 is input into the next layer of the network as the adjusted feature map. The whole process is described by formula (1-1) and formula (1-2). Represents element-wise dot product.

[0063]

[0064] In the channel attention module, the input feature map F passes through the maximum pooling layer and the average pooling layer at the same time. Then, the results of the two pooling operations enter the MLP layer, and then the two extracted features are combined and input to the sigmoid layer to obtain the channel attention weight map. The weight map is element-wise multiplied with the original input feature to obtain the channel attention feature map F1;

[0065] S3. Joint loss function

[0066] The loss function plays a crucial role in improving the effectiveness and efficiency of network learning. Previous methods usually rely on pixel-based L1 Loss to optimize the network by directly comparing the difference between the generated results and the real results. However, this method limits the network's ability to obtain features from deeper hierarchical structures.

[0067] In order to break this limitation, the present invention proposes a new joint loss function, which combines perceptual loss and pixel loss from different aspects, and introduces a loss function based on image structural similarity. By introducing this joint loss function, the network can consider more factors when optimizing parameters, which helps to generate higher quality and visually reasonable material maps.

[0068] The joint loss function of the present invention is expressed by formula (1-3):

[0069]

[0070] in, represents the total loss function, Respectively represent the loss functions for the four material maps (normal, diffuse, roughness, and highlight) of SVBRDF, represents the loss calculated using the rendering result of the generated material map, that is, the rendering loss, λ r Indicates the weight of rendering loss. In the process of experiment, it is found that r The setting of weights affects the quality of the final result. r Set to 1, 3, 10, find λ r Too low or too high will reduce the accuracy of the SVBRDF map. Therefore, in the implementation process of the present invention, λ r The weight of is set to 5.

[0071] Definition of the invention Expressed by formula (1-4):

[0072]

[0073] Among them, M i Represents the actual material map of the target image, Represents the corresponding material map generated by the generator, represents the loss function implemented by MS-SSIM, which is used to evaluate the structural similarity between the generated image and the target image, thereby enhancing the model's attention to global structural features and making the results more consistent with the perception of the human eye. It is a perceptual loss function that compares the similarity between features by directly comparing the differences in the features of each layer of the discriminator. In the implementation process, L1Loss is used to compare the differences in each layer. This loss function enables the network to optimize the model parameters from the feature perspective, thereby expanding the previous method of using only pixel-level loss as a loss function. Similarly, the present invention also uses L1Loss to directly calculate the similarity between the generated result and the target image. It is about the adversarial loss of the material map, which will be introduced in the following content. λ1, λ2, λ3 and λ4 are the weights of each loss function, which are set to 5, 2, 10 and 1 respectively. Because in the experiment, it was found that the loss function based on pixel comparison directly has a greater impact on the accuracy of the result, so a higher weight is given. Although the weight of the loss function based on the degree of feature similarity is not as high as the former, it still helps to improve the accuracy of the result.

[0074] For adversarial loss The optimization strategy of the present invention is expressed by formulas (1-5) and (1-6):

[0075]

[0076] The discriminator and generator use equations (1-5) and (1-6) to implement adversarial training strategies respectively, where G represents the generator and D represents the discriminator. This training strategy not only includes SVBRDF material mapping, but also uses the same strategy for rendering results. The generator's results and the target image's material mapping use the same lighting conditions, that is, randomly selecting point light sources on the upper hemisphere, and the rendering results are based on the Cook-Torrance model, so that the rendering results are more in line with physical rules and reduce unexpected errors.

[0077] Although directly evaluating the differences in material parameters has achieved good results, this strategy alone cannot ensure consistent results after rendering, so it is necessary to add rendering loss To ensure that material textures maintain high quality consistency with ground truth when combined, The definition is expressed by formula (1-7):

[0078]

[0079] and compared to, Joined It is a perceptual loss that uses the VGG pre-trained network as the evaluation standard. The images to be compared are input into the VGG network, and the overall perceptual difference between the images is calculated by comparing the feature differences extracted from each layer of the VGG network. In the implementation details of the invention, VGG19 is used as the pre-trained network model, and its weight parameter λ5 is set to 2.

[0080] In this embodiment: In step S1, the feature extraction layer of the encoder module first passes through a 4×4 convolution layer and a 3×3 convolution layer, each convolution layer is followed by an instance normalization layer and a Leaky ReLU activation function, the first part consists of a 4×4 convolution kernel with a step size of 2, an instance normalization layer and a Leaky ReLU layer, then, additional information is extracted through 1×1, 3×3, and 1×1 convolution layers, while reducing the parameters to be learned, each layer is followed by an instance normalization layer and a Leaky Relu activation function, then, the input features pass through a module (CBAM) that applies attention to channel and spatial features respectively to enhance the focus on valid features and suppress unnecessary information, the features from the attention module are then added to the input features of the previous part to prevent excessive feature loss, and finally, the merged features are input to the final normalization layer.

[0081] In this embodiment: In step S1, in the decoder, the present invention merges the encoder feature information and the decoder input features of the same level, and then inputs them into the upsampling layer, and then, after the normalization layer and the Dropout layer, outputs the features to the next layer for further decoding.

[0082] In this embodiment: the structure of each independent layer discriminator is shown in Table 3.1. The layer discriminator contains a total of 6 convolutional layers of different sizes, which also include partial normalization layers and Leaky ReLU activation functions. In the first 4 convolutional layers of the layer discriminator, Leaky ReLU activation functions are designed for nonlinear transformation. Batch normalization layers and Leaky ReLU activation functions are added after the 5th convolutional layer, and the 6th convolutional layer maps the true probability of the input samples through convolution operations.

[0083] Table 3.1 Layer Discriminator Network Structure Parameters

[0084]

[0085] In this embodiment: In step S2, in the spatial attention module, the input feature F1 passes through a maximum pooling layer and an average pooling layer continuously, and then the result is applied to a 2×2 convolution layer to generate a spatial attention weight map. Finally, the spatial attention weight map is multiplied with the channel attention feature map F1 to obtain the final feature map F2 with comprehensive performance.

[0086] 3.3 Experimental Results and Analysis

[0087] 3.3.1 Dataset and Experimental Environment

[0088] The datasets used are mainly from Deschaintre. Each sample consists of an original image, a normal map, a diffuse map, a roughness map, and a specular map. The size of each map is 256×256, and there are about 200,000 images in total for training and testing.

[0089] The entire training process uses PyTorch as a deep learning framework to build the network. During the experiment, the network uses a batch size of 8 and uses an RTX 4080 GPU as the training hardware. In terms of model parameter settings, the Adam optimizer is used globally, and the learning rate is kept at 2×10 throughout the training process. -5 The parameter of the Leaky ReLU activation function is set to 0.2, while other hyperparameters keep the default values ​​provided by PyTorch.

[0090] 3.3.2 Result analysis

[0091] In order to verify the effectiveness of this method, some recent methods are selected for comparison, such as Deschaintre V, Aittala M, Durand F, et al. Single-image svbrdf capture witharendering-aware deep network[J].ACM Transactions on Graphics(ToG),2018,37(4):1-15, Deschaintre V, Aittala M, Durand F, et al. Flexible svbrdf capture with amulti-image deep network[C] / / Computer graphics forum.2019,38(4):1-13, Zhou X, Kalantari N K. Adversarial Single-Image SVBRDF Estimation with Hybrid Training[C] / / Computer Graphics Forum.2021,40(2):315-325 and Shi Z, Lin X, Song Y. Anattention-embedded GAN for SVBRDF recovery from a single image[J].ComputationalVisual Media, 2023, 9(3): 551-561, among which, the method of Deschaintre V, Aittala M, Durand F, et al. Flexible svbrdf capture with a multi-image deep network[C] / / Computer graphics forum. 2019, 38(4): 1-13 accepts multiple images as input. For fair comparison, only a single image is used as input when comparing with the literature Deschaintre V, Aittala M, Durand F, et al. Flexible svbrdf capturewith a multi-image deep network[C] / / Computer graphics forum. 2019, 38(4): 1-13. In order to more intuitively compare the effects of various methods, in addition to directly comparing the differences in SVBRDF maps, according to the literature Cook RL, Torrance K E.A reflectance model for computer graphics[J].ACM Transactions on Graphics(ToG),1982,1(1):7-24. The Cook-Torrance model adds a comparison chart of the rendering results and maintains the same lighting conditions for the rendering.

[0092] (1) Synthetic Data Comparison

[0093] The test dataset used is from Des

[2018] , Des

[2019] , Zhou

[2021] , Shi

[2023] and ours. The test dataset does not participate in the training of the model and is only used to test the results of the model. RMSE, LPIPS and SSIM are selected as indicators for evaluating image quality. The results of each method (including normal map, diffuse map, roughness map, base reflectivity map and rendering results) are evaluated independently. When comparing the renderings, the same lighting conditions are used. The specific comparison results are shown in Table 3.2, Table 3.3 and Table 3.4. The best value in each table is highlighted in bold.

[0094] The overall performance of this method is better, which is not only attributed to the role of the attention module, which enables the model to pay more attention to important features, so that the estimated SVBRDF has higher accuracy, but also the design of the joint loss function enables the network to constrain the optimization process from multiple perspectives, thus producing more realistic results.

[0095] Table 3.2 RMSE indicator evaluation results

[0096]

[0097] Table 3.3 LPIPS indicator evaluation results

[0098]

[0099] Table 3.4SSIM index evaluation results

[0100]

[0101] In order to make it easier to compare the differences between this method and other methods, some samples were randomly selected for visual comparison, such as Figure 5As shown. By comparison, it is found that when Deschaintre restores roughness and specular reflection maps, compared with this method, it shows more differences from the target map in both details and overall effects. This is also due to the introduction of the attention module and the new joint loss function, which effectively extracts input feature information and shows excellent performance in estimating global features. The method proposed by Deschaintre relies more on the information of multiple input samples, so when estimating various based on a single image, there is a gap with the target image. Zhou will encounter certain errors when estimating normals and roughness, and Shi's values ​​are easily overestimated when estimating specular reflections. Overall, this method performs better in restoring overall features and details.

[0102] (2) Comparison of real data

[0103] In order to compare the performance of this method on real photos, some images taken by cameras or smartphones are randomly selected from the literature Deschaintre V, Aittala M, Durand F, et al. Single-images vbrdf capture with a rendering-aware deep network[J]. ACM Transactions on Graphics (ToG), 2018, 37(4): 1-15. The experimental results are shown in Figure 6 As shown, in Figure 6 In , each corresponding column represents the result of different methods, and each row uses different lighting to render the result. Through comparison, it is found that this method can show more details and reflection effects when dealing with changes in light source conditions.

[0104] 3.3.3 Ablation Experiment

[0105] The present invention conducts a series of ablation experiments to verify the effectiveness of each module in the model, mainly covering the following aspects:

[0106] w / o CBAM attention module. In order to analyze the importance of the attention module in restoring the details of the texture map, the module is removed while keeping other structures unchanged. In each ablation experiment, the present invention keeps other conditions unchanged.

[0107] w / o In order to evaluate the effect of this loss function, The weight of is set to 0.

[0108] w / o In order to verify the importance of this loss function, The weight of is set to 0.

[0109] w / o In order to demonstrate the role of the VGG loss function in improving the quality of the results, the VGG19 network is removed and the The weight of is set to 0.

[0110] The present invention firstly conducts ablation experiments on the CBAM module. Figure 7 As shown in FIG. 1 , the present invention shows several examples to illustrate the effectiveness of the CBAM module. In the figure, the main differences between different methods are highlighted with red boxes. Figure 6 In the present invention, the first and third examples show that the CBAM module helps the network better extract useful information from the input image in the area with strong highlights, while the second example illustrates the help provided by the attention module used in the present invention in utilizing the information of the occluded area in the image.

[0111] exist Figure 8 In the present invention, additional samples used in other ablation experiments are shown.

[0112] Figure 8 (a) shows that there is a lack of This makes it more difficult to decompose the specular component from the input image, and in the second example, the normal map is also difficult to decompose, due to By using a pre-trained discriminator as a perception function, the present invention finds that its introduction is helpful for decomposing and utilizing input features.

[0113] exist Figure 8 In (b), by observing the diffuse reflectance images of the two samples, it can be seen that without using The network has more difficulty extracting information from higher brightness areas, resulting in black areas in these parts because the network cannot decompose further information from them. It is a loss function based on human perception, so it can better optimize errors that are easily observed from human perception.

[0114] Figure 8 (c) display, Helps improve the accuracy of SVBRDF, especially the roughness map. Since it is a single channel material texture map, the differences it presents are more perceptible to the pre-trained network.

[0115] The present invention uses RMSE and LPIPS to perform statistical analysis on the parameters of these ablation experiments, as shown in Table 3.5 and Table 3.6. The better values ​​in the table are highlighted in bold. It can be seen that each module and loss function affects the accuracy of material mapping and rendering results. By comparison, it is found that when using RMSE for analysis, The impact on Specular is small. When using LPIPS for statistics, The impact on Roughness is small. Overall, the joint loss function of this method achieves better performance in terms of numerical value.

[0116] Table 3.5 Ablation experiment: quantitative analysis of the impact of RMSE

[0117]

[0118] Table 3.6 Ablation experiment: quantitative analysis of the impact of LPIPS

[0119]

[0120] The above results show that: because traditional methods are time-consuming and have high requirements for professional equipment, there is now a tendency to use deep learning-based methods to restore the SVBRDF of materials. In the past, deep learning-based methods still left some shortcomings when solving this problem, such as the singleness of the loss function and the performance of the network structure. On this basis, the present invention proposes a new solution. By adding attention modules based on the spatial domain and the channel domain, the network can be more focused on learning important features. A new joint loss function is designed to enable the network to better optimize parameters from multiple angles. Finally, CGAN is introduced as an adversarial loss model to improve the efficiency of adversarial training and the performance of the generator. Finally, the present invention proves the effectiveness of this method through a series of comparative experiments from qualitative and quantitative perspectives and through multiple different evaluation criteria.

[0121] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0122] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A single image material restoration method based on CBAM and adversarial network, characterized by: The method comprises the following steps: S1. Network structure By inputting the material parameters to be estimated and trained into the generator, the generator renders the generated SVBRDF material map into a new image for discrimination using random lighting, and multiple discriminators respectively determine the authenticity of each material map and rendered image. The input image is also added to the discriminator's judgment basis as a limiting condition of CGAN. The entire process optimizes the network parameters through the joint loss function set by the present invention. (1) Generator The generator of this method is mainly composed of an encoder and a decoder. Each layer of the encoder and the decoder corresponds to each other, with a total of 8 corresponding levels, and a splicing operation is used between the corresponding layers to transfer the results of feature extraction. This design minimizes the features after multi-layer network extraction. (2) Discriminator The discriminator is composed of three independent layer discriminators to form a joint discriminator with multi-scale discrimination function. The results of the layer discriminators are downsampled through the middle average pooling layer, and each independent layer discriminator judges the authenticity of the input features based on image features of different sizes. Then the results of the three layer discriminators are combined as the final judgment result. S2, CBAM attention module The attention mechanism has received extensive attention in deep learning. This method uses CBAM, which affects both spatial and channel features, to help the network optimize parameters. As a lightweight module, CBAM effectively affects spatial and channel information, and is very suitable for integration into CNN-based architectures and is widely used in various image and visual processing fields. By effectively utilizing the spatial and channel information of input features, CBAM further enhances the feature extraction capability of CNN networks. Experimental results show that CBAM effectively enhances intermediate features from multiple angles while suppressing irrelevant intermediate features. This makes CBAM very suitable for the task of the present invention, such as when the input sample image contains some interference, such as overexposed highlights and undersaturated shadows, which poses a challenge to CNN in extracting useful information. Therefore, the performance and efficiency of the network are significantly improved by introducing CBAM. For the input feature map F, the feature passes through the channel attention module M c To obtain the channel feature map. The product of the channel feature map and F is represented as F1, and then F1 is input into the spatial attention module M s The spatial feature map is obtained from the spatial feature map, and the product of the spatial feature map and F1 is F2. F2 is input into the next layer of the network as the adjusted feature map. The whole process is described by formula (1-1) and formula (1-2). Represents element-wise dot product. In the channel attention module, the input feature map F passes through the maximum pooling layer and the average pooling layer at the same time. Then, the results of the two pooling operations enter the MLP layer, and then the two extracted features are combined and input to the sigmoid layer to obtain the channel attention weight map. The weight map is element-wise multiplied with the original input feature to obtain the channel attention feature map F1; S3. Joint loss function The loss function plays a crucial role in improving the effectiveness and efficiency of network learning. Previous methods usually rely on pixel-based L1 Loss to optimize the network by directly comparing the difference between the generated results and the real results. However, this method limits the network's ability to obtain features from deeper hierarchical structures. In order to break this limitation, the present invention proposes a new joint loss function, which combines perceptual loss and pixel loss from different aspects, and introduces a loss function based on image structural similarity. By introducing this joint loss function, the network can consider more factors when optimizing parameters, which helps to generate higher quality and visually reasonable material maps. The joint loss function of the present invention is expressed by formula (1-3): in, represents the total loss function, Respectively represent the loss functions for the four material maps (normal, diffuse, roughness, and highlight) of SVBRDF, represents the loss calculated using the rendering result of the generated material map, that is, the rendering loss, λ r Indicates the weight of rendering loss. In the process of experiment, it is found that r The setting of weights affects the quality of the final result. r Set to 1, 3, 10, find λ r Too low or too high will reduce the accuracy of the SVBRDF map. Therefore, in the implementation process of the present invention, λ r The weight of is set to 5. Definition of the invention Expressed by formula (1-4): Among them, M i Represents the actual material map of the target image, Represents the corresponding material map generated by the generator, represents the loss function implemented by MS-SSIM, which is used to evaluate the structural similarity between the generated image and the target image, thereby enhancing the model's attention to global structural features and making the results more consistent with the perception of the human eye. It is a perceptual loss function that compares the similarity between features by directly comparing the differences in the features of each layer of the discriminator. In the implementation process, L1Loss is used to compare the differences in each layer. This loss function enables the network to optimize the model parameters from the feature perspective, thereby expanding the previous method of using only pixel-level loss as a loss function. Similarly, the present invention also uses L1Loss to directly calculate the similarity between the generated result and the target image. It is about the adversarial loss of the material map, which will be introduced in the following content. λ1, λ2, λ3 and λ4 are the weights of each loss function, which are set to 5, 2, 10 and 1 respectively. Because in the experiment, it was found that the loss function based on pixel comparison directly has a greater impact on the accuracy of the result, so a higher weight is given. Although the weight of the loss function based on the degree of feature similarity is not as high as the former, it still helps to improve the accuracy of the result. For adversarial loss The optimization strategy of the present invention is expressed by formulas (1-5) and (1-6): The discriminator and generator use equations (1-5) and (1-6) to implement adversarial training strategies respectively, where G represents the generator and D represents the discriminator. This training strategy not only includes SVBRDF material mapping, but also uses the same strategy for rendering results. The generator's results and the target image's material mapping use the same lighting conditions, that is, randomly selecting point light sources on the upper hemisphere, and the rendering results are based on the Cook-Torrance model, so that the rendering results are more in line with physical rules and reduce unexpected errors. Although directly evaluating the differences in material parameters has achieved good results, this strategy alone cannot ensure consistent results after rendering, so it is necessary to add rendering loss To ensure that material textures maintain high quality consistency with ground truth when combined, The definition is expressed by formula (1-7): and compared to, Joined It is a perceptual loss that uses the VGG pre-trained network as the evaluation standard. The images to be compared are input into the VGG network, and the overall perceptual difference between the images is calculated by comparing the feature differences extracted from each layer of the VGG network. In the implementation details of the invention, VGG19 is used as the pre-trained network model, and its weight parameter λ5 is set to 2.

2. The single image texture restoration method based on CBAM and adversarial network according to claim 1, characterized in that: In the step S1, the feature extraction layer of the encoder module first passes through a 4×4 convolution layer and a 3×3 convolution layer, each convolution layer is followed by an instance normalization layer and a Leaky ReLU activation function, the first part consists of a 4×4 convolution kernel with a step size of 2, an instance normalization layer and a Leaky ReLU layer, then, additional information is extracted through 1×1, 3×3, and 1×1 convolution layers, while reducing the parameters to be learned, each layer is followed by an instance normalization layer and a Leaky Relu activation function, then, the input features pass through a module (CBAM) that applies attention to channel and spatial features respectively to enhance the focus on valid features and suppress unnecessary information, the features from the attention module are then added to the input features of the previous part to prevent excessive feature loss, and finally, the merged features are input to the final standardization layer.

3. The single image texture restoration method based on CBAM and adversarial network according to claim 1, characterized in that: In the step S1, in the decoder, the present invention combines the encoder feature information and the decoder input features of the same level, and then inputs them into the upsampling layer, and then, after the normalization layer and the Dropout layer, outputs the features to the next layer for further decoding.

4. The single image texture restoration method based on CBAM and adversarial network according to claim 1, characterized in that: The structure of each independent layer discriminator is shown in Table 3.

1. The layer discriminator contains a total of 6 convolutional layers of different sizes, which also include partial normalization layers and Leaky ReLU activation functions. In the first 4 convolutional layers of the layer discriminator, Leaky ReLU activation functions are designed for nonlinear transformation. Batch normalization layers and Leaky ReLU activation functions are added after the 5th convolutional layer, and the 6th convolutional layer maps the true probability of the input samples through convolution operations. Table 3.1 Layer Discriminator Network Structure Parameters 5. The single image texture restoration method based on CBAM and adversarial network according to claim 1, characterized in that: In the step S2, in the spatial attention module, the input feature F1 passes through a maximum pooling layer and an average pooling layer continuously, and then the result is applied to a 2×2 convolution layer to generate a spatial attention weight map. Finally, the spatial attention weight map is multiplied with the channel attention feature map F1 to obtain the final feature map F2 with comprehensive performance.

Images