Essential image decomposition method based on bilateral feature pyramid network and multi-scale identification

Abstract

The present invention relates to an essential image decomposition method based on a bilateral feature pyramid network and multi-scale identification. Two parallel generative adversarial networks are used as the main network to reconstruct the reflection map and the illumination map respectively. For the generative network, a strategy of local frequency-fraction feature fusion is proposed to respectively realize the selection and retention of high-frequency reflection features and low-frequency illumination features. At the same time, a multi-scale adaptive combination module is added to the discriminator to adaptively evaluate the contribution of multi-scale features, strengthen the identification effect and improve the generation effect. The method of the present invention performs well on a variety of data sets. In the MPI-Sintel data set, the reconstruction mean square error of the present invention is reduced by 13.26% compared with the optimal results of other methods; in the ShapeNet data set, the reconstruction mean square error of the present invention is reduced by 26.09% compared with the optimal results of other methods.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to an essential image decomposition method based on a bilateral feature pyramid network and multi-scale identification. Background Art

[0002] During the object imaging process, many environmental factors will affect the scene appearance to varying degrees, including different light intensity, different light incident angles, shadows, and posture changes. These changes will eventually appear in the image. Different images will be collected for the same target object in different environmental conditions, which also brings certain difficulties to the subsequent image understanding tasks.

[0003] To solve this problem, we need to extract features that do not change with environmental factors, that is, essential feature images. Essential features refer to features that are inherent to the object itself and are not affected by environmental factors. For an object, its inherent features include color, texture, and material. These inherent features will not change with changes in environmental factors. If we can separate the essential feature information such as color, texture, and material of the object from the image from the environmental information, and filter out the image components affected by the environment, we can get a more accurate description of the feature information of the object. This approach is conducive to improving the performance of other image processing tasks.

[0004] The decomposition of the essential features of an image is one of the low-level tasks in the field of computer vision. Essential feature extraction is to extract inherent features. It decomposes the image into two parts: a reflection map with color, texture, and material, and an illumination map with shape information and illumination information. The reflection map does not change with changes in environmental factors, so the decomposed reflection map can be used as input for other image processing tasks, which will greatly reduce the difficulty of image analysis and make image processing have the robust characteristics of illumination invariance. The reflection map and illumination map of an image, such as Figure 1 At the same time, with the continuous deepening of research on deep learning methods, the image analysis algorithm based on the essential characteristics of convolutional neural networks has made great progress in real-time and accuracy, which has also laid a solid theoretical foundation for improving the robustness of advanced image processing tasks such as unmanned driving and accelerating its industrial application.

[0005] The solution to essential image decomposition can be divided into the following two categories: the first is the optimization method based on explicit constraints, and the second is the deep learning method based on implicit constraints.

[0006] Most optimization methods based on explicit constraints use prior constraints of the essential image to solve the optimization problem in the constraint domain. The performance of the algorithm depends on the rationality of the prior constraints and the convergence performance of the convex optimization function. Reasonable prior constraints and good optimization functions can prevent the model from converging to the local optimal solution. Optimization methods based on explicit constraints do not require labels, but their applications are limited. In reality, the lighting conditions are very complex, and problems such as highlights, occlusions, and specular reflections may occur. The proposed prior constraints cannot fully handle these conditions. Implicit constraints obtained based on label learning can generalize these situations better than explicit constraints, and are also the current mainstream research direction. Summary of the invention

[0007] Technical issues to be solved

[0008] In view of the problem that existing methods lack communication and guidance for features between different frequencies, the present invention provides an essential image decomposition method based on bilateral feature pyramid network and multi-scale identification. The bilateral feature pyramid network is innovatively introduced so that the reflection map and the illumination map can obtain feature information that is beneficial to the reconstruction of the other.

[0009] Technical Solution

[0010] An essential image decomposition method based on bilateral feature pyramid network and multi-scale identification, characterized by the following steps:

[0011] Step 1: Build a training image sample library;

[0012] Step 2: construct a generator network; the generator network includes a reflection map generator and an illumination map generator, the reflection map generator and the illumination map generator differ in structure only in the skip connection channel, the backbone network of the generator adopts the U-Net network structure as a template, with a total of 5 layers, and the channels corresponding to each convolution layer are 16, 32, 64, 128 and 256 respectively; for the skip connection channel, the present invention adopts a top-to-bottom symmetrical structure; the reflection map generator first calculates the intermediate features from low frequency to high frequency, and then calculates the output features from high frequency to low frequency in combination with the intermediate features; the illumination map generator first calculates the intermediate features from high frequency to low frequency, and then calculates the output features from low frequency to high frequency in combination with the intermediate features;

[0013] Step 3: construct a discriminator network; the discriminator network consists of a four-layer convolutional neural network, the convolutional layer step size is 2, the size of each convolutional layer is halved, the channel changes of the four convolutional layers are 3 to 64, 64 to 128, 128 to 256, 256 to 512, and the output of each convolutional layer is compressed into a single-channel feature probability map; when the discriminator is judged to be true, all single-channel feature probability maps are close to 1, and when it is judged to be false, all single-channel feature probability maps are close to 0;

[0014] Step 4: Define the loss function;

[0015] The generator loss is defined as shown in formula (1):

[0016] L G =L GAN-G +L mse +L cos +L bf +L feat (1)

[0017] Among them, L GAN-G represents the intrinsic loss function, L mse represents the mean square error function, L cos represents the cosine loss function, L feat represents the feature loss function;

[0018] Intrinsic loss function L GAN-G The calculation formula is shown in formula (2):

[0019]

[0020] Among them, W i represents the normalized weight parameter of the i-th layer, i represents the network layer number, fake_output i Indicates the probability that the output image is false, and ones indicates the probability is 1;

[0021] Mean square error function L mse The calculation formula is shown in formula (3):

[0022]

[0023] Among them, fake_image i Represents the output of the reciprocal i-th layer feature map of the decoder, true_image i Indicates the image label that is scaled i times;

[0024] Cosine loss function L cos The calculation formula is shown in formula (4):

[0025]

[0026] Among them, fake_region i Indicates the i-th region of the generated image, true_region i Represents the i-th block area of ​​the generated image;

[0027] Cross bilateral filtering loss L bf The calculation formula is shown in formula (5)-(6):

[0028]

[0029]

[0030]

[0031] Among them, L bf represents the double-sideband filtering loss, bf represents the double-sideband filtering, C represents the label image, {A, S} represents the reflection map and the illumination map respectively, and J p represents the output of the bilateral filter, C p Represents the value of the pth pixel of the label image, N p represents the total number of p pixels and neighboring pixels, W p represents the normalized weight, q represents the neighbor pixel position of p, N(p) represents the neighbor pixel position set of the p-th pixel, represents the spatial Gaussian kernel, p represents the position of the pth pixel, represents the range Gaussian kernel, C q Represents the value of neighbor pixel q;

[0032] L feat The calculation formula is shown in formula (8):

[0033]

[0034] Where l represents the lth layer of the VGG network, F l Indicates the number of channels of the feature map of the lth layer, H l represents the height of the feature map of the lth layer, W represents the width of the feature map of the lth layer, Represents the feature activation value of the lth layer;

[0035] The discriminator loss is defined as shown in formula (9):

[0036]

[0037] Among them, L2 represents l1 loss, y i represents the ground-truth image, f(x i ) represents the image after super resolution;

[0038] Step 5: Network training

[0039] When the reflectance map generator or the illumination map generator is trained, the reflectance map or the illumination map output by the reflectance map generator or the illumination map generator is input into the discriminator, and the discriminator compares the input reflectance map or the illumination map with the label image, and outputs the probability that the reflectance map or the illumination map is consistent with the label image;

[0040] Step 6: Input the original image to be processed into the reflection map generator or illumination map generator obtained in step 5 respectively, and the output image is the reflection map or illumination map obtained by decomposing the original image.

[0041] Further technical solution of the present invention: Step 1 is specifically:

[0042] A certain number of images are randomly selected from the test image data set, and then M small images of size N*N are randomly sampled from each image. These M small images are flipped horizontally or upside down to obtain M new small images, so that each image obtains 2*M small images; the small images obtained after performing the above operations on all the extracted images constitute the training image sample library.

[0043] A further technical solution of the present invention: Step 4 is specifically: a reflection map generator and a discriminator are combined to train the reflection map generator, and an illumination map generator and a discriminator are combined to train the illumination map generator; the present invention uses MIT and MPI data sets to train the generator network, adopts the SGD optimization method to update the network parameters, stops training when the loss function value is minimized, and obtains the final trained network.

[0044] A computer system, characterized in that it comprises: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0045] A computer-readable storage medium, characterized in that it stores computer-executable instructions, which are used to implement the above method when executed.

[0046] Beneficial Effects

[0047] The present invention provides an essential image decomposition method based on a bilateral feature pyramid network and multi-scale identification. The proposed network structure uses two parallel generative adversarial networks as the main network to reconstruct the reflection map and the illumination map respectively. For the generation network, the present invention proposes a strategy of local frequency-fraction feature fusion to respectively select and retain high-frequency reflection features and low-frequency illumination features. At the same time, the present invention adds a multi-scale adaptive combination module to the discriminator to adaptively evaluate the contribution of multi-scale features, strengthen the identification effect and improve the generation effect. Furthermore, the present invention constructs a variety of loss functions to constrain the generation results and promote the training of the network. The algorithm proposed in the present invention performs well on a variety of data sets. In the MPI-Sintel data set, the reconstruction mean square error of the present invention is reduced by 13.26% compared with the best results of other methods; in the ShapeNet data set, the reconstruction mean square error of the present invention is reduced by 26.09% compared with the best results of other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0049] Figure 1 Schematic diagram of reflection map and illumination map;

[0050] Figure 2 Schematic diagram of the intrinsic decomposition network based on bilateral feature pyramid and multi-scale identification;

[0051] Figure 3 This is a schematic diagram of the BIFPN-A network structure;

[0052] Figure 4 Schematic diagram of the BIFPN-B network structure;

[0053] Figure 5 Schematic diagram of the discriminator network structure;

[0054] Figure 6 A schematic diagram of the MPI-Sintel data set;

[0055] Figure 7 This is a schematic diagram of ShapeNet’s intrinsic image;

[0056] Figure 8 This is a schematic diagram for comparing test results of different module combinations;

[0057] Fig. 9 This is a schematic diagram of the network structure of the first BIFPN module;

[0058] Fig.10 This is a schematic diagram of the network structure of the second BIFPN module;

[0059] Fig.11 It is a schematic diagram for comparing the essential image decomposition of the present invention and other methods under image segmentation;

[0060] Fig.12 This is a schematic diagram for comparing the essential image decomposition of the present invention and other methods under scene segmentation;

[0061] Fig.13 This is a schematic diagram of the comparison of local area effects under scene segmentation;

[0062] Fig.14 This is a schematic diagram comparing the effects of some test data of the ShapeNet dataset. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0064] The present invention proposes an essential image decomposition method based on bilateral feature pyramid and multi-scale identification, and its network structure is as follows: Figure 2 As shown, in terms of the generator, taking the U-Net algorithm as the prototype, the present invention innovatively adds a deformed bilateral feature pyramid module to the jump layer of U-Net, and sends the encoder features to the decoder after enhanced selection, thereby improving the effect of essential image decomposition. In the discriminator, the present invention adds a multi-scale adaptive combination module to predict at multiple feature scales, enhance the identification effect, and thus improve the generation effect. On the one hand, the constraint of frequency decomposition is added to the jump connection of the reflection map U-Net, so that the network learns the importance of different features and obtains more suitable features. On the other hand, frequency decomposition and frequency compression are added to the jump connection of the illumination map, which not only obtains a more suitable feature map but also solves the problem of many high-frequency components in the illumination map. The following steps are included:

[0065] Step 1: Build a training image sample library

[0066] (1) Build a training image sample library

[0067] The present invention randomly extracts a certain number of images from a test image data set, and then randomly samples M small images of size N*N from each image, and flips the M small images horizontally or upside down to obtain new M small images, so that each image obtains 2*M small images; the small images obtained after performing the above operations on all the extracted images constitute a training image sample library.

[0068] Step 2: Build the generator

[0069] (1) Build a reflection map generator

[0070] The generator network is based on the U-Net network structure. A bilateral feature pyramid network is introduced in the jump connection layer from the encoder to the decoder of U-Net to enhance the encoder's ability to decompose effective features from the original image and suppress the generation of invalid features. The network adopts a symmetrical structure, and both the encoder and decoder have 5 layers. Each layer of the encoder consists of a downsampling layer and a convolution layer. The channels corresponding to the convolution layers are 16, 32, 64, 128 and 256, respectively. Each layer of the decoder consists of an upsampling layer and a convolution layer. Except for the last convolution layer of the encoder and the last convolution layer of the decoder, which do not use batch normalization layers, other convolution layers uniformly add batch normalization layers to the convolution operation to accelerate network training. According to prior knowledge, the activation functions of the first few layers of the generator network use the Leaky-ReLU structure, and the activation function of the final output layer uses Tanh for activation. At this time, an image is input into the reflection map generator, and the output is the reflection map of the input image.

[0071] The BIFPN-A module is used to generate a reflection map, and the BIFPN-B module is used to generate an illumination map. The BIFPN module is different from the general skip connection layer. Its input is not just one channel, but five skip connection channels. The frequency of the channels increases from top to bottom. Both BIFPN modules are composed of three BIFPN blocks stacked together. BIFPN Blocks are specifically divided into two types: reflection map end and illumination map end. The block structure of the BIFPN-A module is as follows: Figure 3 As shown in the figure, it keeps the operation of calculating the intermediate features from low frequency to high frequency, and then calculating the output features from high frequency to low frequency by combining the intermediate features. This is because it allows the high-frequency features to guide the synthesis of low-frequency features, making the output result contain richer high-frequency features. However Figure 3 The Block divides the five paths into high-frequency paths and medium-low-frequency paths. The high-frequency path contains the lower two layers, and the medium-low-frequency path contains the upper three layers. When calculating the intermediate features, the two paths need to be calculated separately. The advantage of this is that the high-frequency features and the medium-low-frequency features can be isolated to a certain extent, and they are not simply merged together.

[0072] (2) Constructing the illumination map generation graph

[0073] For the Block of BIFPN-B, its structural diagram is as follows Figure 4 It and Figure 3In comparison, it just exchanges the upsampling and downsampling operations. It first calculates the intermediate features from high frequency to low frequency, and then calculates the output features from low frequency to high frequency by combining the intermediate features. The 5 features of different scales input to the two BIFPN networks will first be input into the BIFPN Block together. First, these features of different scales need to unify the channels to 64 when performing operations. Then, the network will allocate the input features according to different weights and fuse the features of different scales, and finally obtain the fused features. Compared with BIFPN-A, BIFPN-B adds the final channel compression. It consists of a 1×1 convolution layer and a Leaky ReLU activation layer. This 1×1 convolution layer reduces the number of channels of the input feature to one-eighth of the original, so that the illumination map removes high-frequency features as much as possible without losing important high-frequency information.

[0074] Step 3: Construct the discriminator

[0075] The discriminator consists of a four-layer convolutional neural network. Its network structure is as follows Figure 5 When the reflectance map generator or the illumination map generator is trained, the reflectance map or the illumination map output by the reflectance map generator or the illumination map generator is input into the discriminator, and the discriminator compares the input reflectance map or the illumination map with the label image, and outputs the probability that the reflectance map or the illumination map is consistent with the label image.

[0076] The reflectance map generator is combined with the discriminator to train the reflectance map generator. The illumination map generator is combined with the discriminator to train the illumination map generator.

[0077] Step 4: Define the loss function

[0078] (1) Define the generator loss as shown in formula (1):

[0079] L G =L GAN-G +L mse +L cos +L bf +L feat (1)

[0080] Among them, L GAN-G represents the intrinsic loss function, L mse represents the mean square error function, L cos represents the cosine loss function, L feat Represents the feature loss function.

[0081] Intrinsic loss function L GAN-G The calculation formula is shown in formula (2):

[0082]

[0083] Among them, W i represents the normalized weight parameter of the i-th layer, i represents the network layer number, fake_output i Indicates the probability that the output image is fake, and ones indicates the probability is 1.

[0084] Mean square error function L mse The calculation formula is shown in formula (3):

[0085]

[0086] Among them, fake_image i Represents the output of the reciprocal i-th layer feature map of the decoder, true_image i Indicates the image label that is scaled i times.

[0087] Cosine loss function L cos The calculation formula is shown in formula (4):

[0088]

[0089] Among them, fake_region i Indicates the i-th region of the generated image, true_region i Represents the i-th block area of ​​the generated image.

[0090] Cross bilateral filtering loss L bf The calculation formula is shown in formula (5)-(6):

[0091]

[0092]

[0093]

[0094] Among them, L bf represents the double-sideband filtering loss, bf represents the double-sideband filtering, C represents the label image, {A, S} represents the reflection map and the illumination map respectively, and J p represents the output of the bilateral filter, C p Represents the value of the pth pixel of the label image, N p represents the total number of p pixels and neighboring pixels, W p represents the normalized weight, q represents the neighbor pixel position of p, N(p) represents the neighbor pixel position set of the p-th pixel, represents the spatial Gaussian kernel, p represents the position of the pth pixel, represents the range Gaussian kernel, C q Represents the value of neighbor pixel q.

[0095] L feat The calculation formula is shown in formula (8):

[0096]

[0097] Where l represents the lth layer of the VGG network, F l Indicates the number of channels of the feature map of the lth layer, H l represents the height of the feature map of the lth layer, W represents the width of the feature map of the lth layer, Represents the feature activation value of the lth layer;

[0098] (2) Define the discriminator loss as shown in formula (9):

[0099]

[0100] Among them, L2 represents l1 loss, y i represents the ground-truth image, f(x i ) represents the image after super-resolution.

[0101] Step 5: Network training

[0102] Use the training image sample library constructed in step 1 to train the combination of the reflection map generator and discriminator in step 2 and the combination of the illumination map generator and discriminator respectively, use the Adam optimization method to update the network parameters, stop training when the loss function value defined in step 3 is the smallest, and obtain the final reflection map generator and illumination map generator;

[0103] Step 6: Input the original image to be processed into the reflection map generator or illumination map generator obtained in step 5 respectively, and the output image is the reflection map or illumination map obtained by decomposing the original image.

[0104] In order to enable those skilled in the art to better understand the present invention, the present invention is described in detail below in conjunction with specific embodiments.

[0105] Embodiment 1:

[0106] Step 1: Build a training image sample library

[0107] The present invention uses the MPI image dataset based on complex scenes and the ShapeNet image dataset based on single synthetic objects. There are 9 common scenes in the MPI dataset, each of which has two subcategories and 50 images in each subcategory. Figure 6 Part of the MPI-Sintel dataset is shown. There are two segmentation methods when constructing the training image sample library, one is based on the image-split method and the other is the scene-split method.

[0108] In the image-split mode, half of the images are extracted from each of the 18 small classes in the image dataset. The size of each image is 1024x436. Ten small images of 256x256 are randomly sampled from the image. Then the small images are flipped horizontally or upside down, so that 20 small images are obtained for each image. The total number of training data sets is 9000 (18x25x20) small images of 256x256. The test data set uses the original image size, with a total of 450 (18x25) large images of 1024x436.

[0109] In the scene-split mode, one small class is selected for training from each large class, and the other small class is used for testing. The two defective small classes "bandage 1" and "shaman 3" are removed. The method of obtaining small images is the same as that of the image-split mode. The training data set is 9000 (9x50x20) small images of size 256x256, and the test data set is 350 (7x50) large images of size 1024x436.

[0110] The ShapeNet dataset is a large-scale dataset of 3D shapes. It is a computer-generated dataset in which each image provides a reflectance map, illumination map, surface normal, depth, and scene lighting conditions. The images in the ShapeNet dataset are fully aligned, and for the intrinsic image decomposition task, only the illumination map and reflectance map are needed. ShapeNet has more than 3 million images of more than 3,000 categories, and each category has data of different objects, different angles, and different lighting. Figure 7 Shows intrinsic images from the ShapeNet dataset.

[0111] Step 2: Build the generator network

[0112] according to Figure 3 and Figure 4 , the method in step 2 is used to construct the reflection map generator and the illumination map generator. The encoder of the reflection map generator U-Net network uses the convolution layer, batch normalization layer, and LeakyRelu activation function layer as downsampling blocks. The step size of the convolution layer is 2. After each convolution operation, the size of the feature map is halved. The output of each activation function layer in the encoder is skipped and connected to the frequency decomposition submodule. The channel of the encoder changes to [3, 32, 64, 128, 256]. The output of the frequency decomposition submodule is sent to the decoder, and the step size of the decoder convolution layer is 1.

[0113] The channel compression submodule of the light map generator is performed through a convolutional layer with a step size of 1. The size of the feature map is not changed, and the number of channels is compressed in different proportions. If there are fewer high-frequency components in the light map, the compression ratio is large; if there are more low-frequency components, the compression ratio is small.

[0114] Step 3: Build the Discriminator Network

[0115] The discriminator is a four-layer convolutional neural network with a convolutional layer stride of 2. The size of each convolutional layer is halved. The channels of the four convolutional layers change from 3 to 64, 64 to 128, 128 to 256, and 256 to 512 respectively. The output of each convolutional layer is compressed into a single-channel feature probability map. When the discriminator judges as true, all single-channel feature probability maps are close to 1, and when it is judged as false, all single-channel feature probability maps are close to 0.

[0116] Step 4: Construct the loss function

[0117] According to formulas (1)-(9), the present invention calculates the generator loss function, and the weights of the first and last layers of the generator network in the inherent loss are set to 4, and the weights of the middle two layers are set to 1.

[0118] When calculating the mean square error, the present invention takes the feature maps of the last three layers of the decoder to generate a complete, half, and one quarter of the original image, respectively, and constrains the three different scales, with the weights of the three scales being 1, 0.8, and 0.6, respectively.

[0119] In order to better maintain the edge features when calculating the cosine loss and keep the edges of the generated image and the label image consistent, the input image is divided into 4 blocks to ensure that the cosine similarity of each block and the corresponding label block is consistent.

[0120] When calculating the discriminator loss function, the first and last layers have a weight of 4, and the middle two layers have a weight of 1.

[0121] Step 5: Network training

[0122] The training is performed using samples from the training image sample library. Different generators and discriminators are used for the reflectance map and illumination map, respectively. The network models of the illumination map and the reflectance map are the same and are trained separately. The Adam optimization method is used to optimize the network. The generator and the discriminator need to use different Adam optimizers. The optimizer parameters betas are set to (0.5, 0.999), the learning rate is 0.0005, the weight_decay is 0.0001, and the batchsize is 20. The generator and the discriminator are trained alternately (TTUR). The number of training times for the discriminator is 5 to 1 compared to the number of training times for the generator.

[0123] Step 6: Experimental results and analysis

[0124] In order to comprehensively evaluate the effect of the algorithm proposed in the present invention, the present invention first analyzes the effect of the bilateral feature pyramid module of the skip connection layer, and then compares and evaluates it in terms of visualization effect and quantitative indicators on the MPI-Sintel essential image dataset.

[0125] (1) Evaluation of generator and discriminator modules

[0126] In order to measure the effectiveness of the modules in the generator and discriminator, the present invention is evaluated on the MPI-Sintel dataset with complex scenes. Taking image segmentation as an example, a set of comparative experiments are designed to evaluate the generator's skip connection layer bilateral feature pyramid module (BIFPN) and the discriminator's adaptive combination module (AC).

[0127] (a) Without BIFPN: The bilateral feature pyramid network of the generator is removed, and the features of the encoder are directly passed to the decoder;

[0128] (b) Without AC: the multi-scale adaptive combination module without the discriminator;

[0129] (c)With All: Using both the bilateral feature pyramid module and the adaptive combination module.

[0130] With other variables unchanged, the present invention trains the above three networks, and the final result indicators on the MPI-Sintel data set are shown in Table 1.

[0131] Table 1 Comparison of generator and discriminator modules (the black ones are the best)

[0132]

[0133] From Table 1, we can see that the results of using the skip connection layer bilateral feature pyramid module (BIFPN) and the adaptive combination module (AC) are better than not using one of the modules in three indicators. Among them, the structural similarity is improved the most, and the predicted image is closer to the label, followed by the mean square error, which also maintains a high accuracy in the overall pixel area, and finally the local mean square error indicator is improved less. Figure 8The following table shows some results of the comparative experiments of three networks on the MPI-Sintel image segmentation dataset. There are four columns in the figure. The first column is the original image, and the second, third and fourth columns correspond to the three experimental results in Table 4-1. It can be seen from the four images in the second column that the results generated by the network without the bilateral feature pyramid module are not as good as those in the fourth column. The reflection images in the first and second rows of the second column still contain some shadows, which is because the lack of the bilateral feature pyramid module causes the illumination information to not be completely removed. The illumination images in the third and fourth rows of the second column still seem to contain some texture information, indicating that the high-frequency feature information inside is not completely removed. It can be seen from the four images in the third column that the results generated by the network without the multi-scale adaptive module are not as good as those in the fourth column. It can be clearly seen that some pixels of the original image are still retained in the four corners of the third and fourth images in the third column. The results using the multi-scale adaptive combination module are better in both global information recovery and local detail recovery.

[0134] According to the above indicators and visualization results, it can be concluded that the introduction of the bilateral feature pyramid module in the skip connection layer proposed by the present invention and the use of the multi-scale adaptive combination module in the discriminator have obvious effects on the essential image decomposition, significantly improve the effectiveness and consistency of the image, and also have better reconstruction effects on the local details of the image. After fully explaining the role of the bilateral feature pyramid module and the multi-scale adaptive combination module, the present invention continues to explore the internal results of the bilateral feature pyramid module. Here, the present invention makes changes based on the original bilateral feature pyramid module to find the best decomposition network. Therefore, the present invention proposes several different bilateral feature pyramid modules, trains them, obtains the decomposition results and evaluates them.

[0135] The first network is Fig. 9 As shown in Figure 1. This network is consistent with the classic bilateral feature pyramid network, and is also a five-layer network. First, the lowest frequency feature is continuously upsampled and fused with the second-highest frequency feature to obtain the intermediate feature. Then, starting from the highest frequency feature, it is continuously downsampled and fused with the intermediate feature to obtain the output feature information. Among them, the sub-network blocks are connected in series twice to obtain the final network structure.

[0136] The second network is Fig.10 As shown in the figure. This network is the opposite of the first network structure. It continuously downsamples the highest frequency feature and fuses it with the second lowest frequency feature to obtain the intermediate feature. Then, starting from the lowest frequency feature, it continuously upsamples and fuses it with the intermediate feature to obtain the output feature information. The final network structure is also composed of 3 repeated sub-network blocks.

[0137] The third and fourth networks are the networks finally adopted by the present invention, namely Figure 3 and Figure 4. These two networks are the evolution of the first two networks. Considering that the proportions of high- and low-frequency feature components in the reflection map and the illumination map are different. The high-frequency features in the reflection map account for a high proportion, and the low-frequency features in the illumination map account for a high proportion. Therefore, if the high-frequency features and the low-frequency features can be properly separated, the decomposition effect of the image will be improved to a certain extent. Therefore, the third network has made some modifications compared to the first network, treating the two high-frequency pathways as a group and the three low-frequency pathways as a group. The high-frequency group and the low-frequency group independently generate intermediate features, and the subsequent downsampling operation combines the two groups. Similarly, the fourth network has made some modifications relative to the second network. It is also divided into two groups of high-frequency and low-frequency channels, and generates intermediate features respectively, and finally connects them at the output features by upsampling.

[0138] Structure 1: The reflection map uses network 1 + the illumination map uses network 1 and channel compression;

[0139] Structure 2: The reflection map uses network 2 + the illumination map uses network 2 and channel compression;

[0140] Structure 3: The reflection map uses network 1 + the illumination map uses network 2 and channel compression;

[0141] Structure 4: The reflection map uses network 3 + the illumination map uses network 4 and channel compression.

[0142] With other variables unchanged, the present invention trains the above four network combinations on the MPI-Sintel image segmentation dataset, and the final indicators are shown in Table 2.

[0143] Table 2 Comparative experimental results of different jump connection modules

[0144]

[0145]

[0146] It can be seen from the above table that compared with structure 1, the mean square error of the reflection map of structure 1 is lower than that of structure 2, while the mean square error of the illumination map of structure 2 is lower than that of structure 1. Because the reflection map requires a lot of high-frequency feature information. Structure 1 happens to have a path from high-frequency features down-sampling to low-frequency features, so that the reflection map end is guided by the high-frequency components when reconstructing the low-frequency components, thereby generating a better reflection map. Similarly, the result of structure 2 is exactly the opposite of structure 1. It has a path from low-frequency features up-sampling to high-frequency features, so that the illumination map end is guided by the low-frequency components when reconstructing the high-frequency components. The generated illumination map contains enough low-frequency feature information, and a better illumination map is obtained. Then, if the structure of the reflection map end in structure 1 and the structure of the illumination map end in structure 2 are integrated, will the effect be better than that of structure 1 and structure 2? Based on this conjecture, the present invention designs structure 3 and conducts experiments. Finally, the experimental results are consistent with the conjecture, and the three indicators of structure 3 are better than the previous two combinations. Although the results generated by structure 3 are better than those of structure 1 and structure 2, the network structure is too complex. Then, if the effect is guaranteed, further simplifying the network structure becomes the next research topic. The present invention notes that the essence of the essential image decomposition problem is to distinguish the high-frequency features from the low-frequency features of the image. Therefore, the structure of directly connecting channels of all scales in the first three networks is not in line with actual theory. Therefore, the present invention attempts to modify the jump connection modules of the reflection map end and the illumination map end, and simply divide all channels into high-frequency channels and medium-low-frequency channels. Figure 3 and 4 It can be seen that the high-frequency channels are the two channels below, while the medium- and low-frequency channels are the three channels above. The intermediate features are only decomposed and calculated in the two main channels, and the two main channels are connected when calculating the output features of each overlapping block. Experiments have shown that this approach can not only simplify the network structure and ensure the decomposition effect, but also conform to theoretical practice.

[0147] (2) Quantitative analysis of loss function

[0148] In order to evaluate the impact of different loss functions on the final results, the present invention still uses the MPI-Sintel data set for training and testing, adopts exactly the same parameters, fixes the random seed, removes the selected loss function, and keeps other loss functions unchanged, and designs multiple groups of comparative experiments for evaluation.

[0149] (a) Without VGG: remove VGG perceptual loss;

[0150] (b)Without Muti-Scale: remove multi-scale loss;

[0151] (c) Without Bf: removes double-sideband filtering loss;

[0152] (d) Without Cos: removes local cosine loss;

[0153] The present invention conducts experiments on datasets based on image segmentation and scene segmentation, and the specific quantitative indicators are shown in Tables 3 and 4.

[0154] Table 3. Comparison of quantitative indicators of loss function under image segmentation

[0155]

[0156] Table 4 Comparison of loss function quantitative indicators under scene division

[0157]

[0158] From the results in the table, we can see that different loss functions have different promoting effects on the decomposition effect of the essential image. In addition, the datasets of image segmentation and scene segmentation will also affect the decomposition results. Among them, VGG perceptual loss has the greatest promoting effect on the final result, because it targets the loss of feature space and can strongly constrain features of different scales. The Cos loss contributes the least in the image segmentation dataset, but is second only to VGG perceptual loss in the scene segmentation dataset, indicating that Cos has a greater role in generalized scenes. Multi-scale loss has a promoting effect in both image segmentation and scene segmentation. Finally, the double-sideband filter loss also has a promoting effect in graphic segmentation, but has a negative impact on the structural similarity index of the illumination map in scene segmentation, which may be caused by the weak generalization of the adaptive constraints of the double-sideband filter.

[0159] (3) Analysis of experimental results of MPI-Sintel dataset based on image segmentation

[0160] The present invention plans to use the proposed network to compare with recent work. In order to ensure accuracy, the present invention uses the same data set and test method as Fan et al. to conduct experiments. The present invention compares the quantitative indicators of various previous methods, and the results are shown in Table 5. From the perspective of quantitative indicators, the method proposed by the present invention is better than other methods in terms of mean square error indicators, but worse in local mean square error and structural similarity of illumination maps.

[0161] Table 5 Comparison of quantitative indicators of various methods and this method under image segmentation

[0162]

[0163] In terms of visualizing the results, Fig.11The essential image decomposition diagrams of various methods and the present invention in the image segmentation data set are shown. From the results, the illumination map generated by the Barron method based on manually designed features is over-smoothed, its high-frequency features are almost lost, and the shadows of the reflection map are not completely removed. The Chen method based on manual features also has a huge deviation in generating colors. Its reflection map is also over-smoothed, and a lot of high-frequency detail information is decomposed into the illumination map. The methods in rows 4 to 5 in the figure are based on deep learning, and it can be seen that they are all better than the methods based on manual features. The image decomposed by the MSCR method has many blurred pixel blocks, the smoothness and consistency of the image are poor, and the local details are also poorly restored. The reflection map generated by Fan's method performs well in smoothness and consistency, but there is still room for improvement in the recovery of some local details. The present invention is better than the above methods in image smoothness, texture consistency and detail recovery. In particular, compared with Fan's method, for example, the hair of the characters in the first column of the figure and the texture features behind the tasks in the third column are closer to the labels, while Fan's method performs poorly. Although the present invention is superior to all other methods in terms of quantitative indicators and visualization results, it is still a certain distance away from the label image. For example. In the sixth row of the second column, there is still a certain gap between the hair details in the illumination map generated by the present invention and the label, which is consistent with the results in the table, indicating that the illumination map end of the present invention has not completely removed high-frequency information such as texture features, and further improvement is needed.

[0164] (4) Analysis of experimental results of MPI-Sintel dataset based on scene division

[0165] The image decomposition task of the MPI-Sintel dataset based on scene division is a very difficult challenge, because the scenes in the test set and the training set are completely different, which further tests the generalization ability of the method in this chapter. In order to ensure accuracy, the present invention also uses the same dataset and test method as Fan et al. to conduct experiments. The final quantitative index results of the experiment are shown in Table 6. It can be seen that the present invention has better performance than the previous method in all indicators.

[0166] Table 6 Comparison of quantitative indicators of various methods and the present invention under scene division

[0167]

[0168] In terms of visualizing the results, Fig.12 The essential image decomposition diagrams of other deep learning methods and the present invention in the scene segmentation dataset are shown. It can be seen that the present invention is much better in overall detail and color restoration than the previous methods. Fig.13Three pixel areas with the most obvious differences were selected for analysis of the above test image. In the first column, it can be seen that the characters restored by the MSCR method are very blurry and the details are not obvious, while the color restoration of Fan's method is problematic, and the skin and clothes are not smooth enough. The present invention restores the details of skin and clothes very well. In the second column, it can be seen that the canvas of MSCR is still blurry, with better color, but the details are completely lost. Fan's method has better details, but poor color restoration, and the canvas is mixed with other gray features, while the canvas restored by the present invention has normal color and the details around the canvas are also very clear. The wall in the third column is similar to the above two columns. The present invention is clearer at the edge and has good color restoration. It can be seen from the comprehensive visualization effect and specific quantitative indicators that the present invention is superior to all compared algorithms.

[0169] (5) Analysis of experimental results of ShapeNet dataset

[0170] In order to verify the universality of the method proposed in this invention on different data sets, this invention also trains and tests on the ShapeNet data set. This invention selects a subset of the ShapeNet data set, which has 100k image data. The ratio of the training set to the test set is 9:1. The method and parameters used are exactly the same as those of the MPI-Sintel data set. The specific quantitative indicators are shown in Table 7. The visual effect diagram of ShapeNet is shown in Fig.14 As shown in the figure, it can be seen that the method proposed in the present invention also achieves the best results on the ShapeNet dataset, and the improvement is very large. The visualization effect is very close to the label, which proves that the method not only has good results in complex scenes, but also has good results in datasets based on single objects.

[0171] Table 7 Comparison of quantitative indicators of various methods and the present invention in ShapeNet dataset

[0172]

[0173] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. An essential image decomposition method based on bilateral feature pyramid network and multi-scale identification, characterized in that Here are the steps: Step 1: Build a training image sample library; Step 2: Construct a generator network; the generator network includes a reflection map generator and an illumination map generator. The reflection map generator and the illumination map generator differ in structure only in the skip connection channel. The backbone network of the generator adopts the U-Net network structure as a template, with a total of 5 layers, and the channels corresponding to each convolution layer are 16, 32, 64, 128 and 256 respectively; for the skip connection channel, the present invention adopts a top-to-bottom symmetrical structure; the reflection map generator first calculates the intermediate features from low frequency to high frequency, and then calculates the output features from high frequency to low frequency in combination with the intermediate features; The illumination map generator first calculates the intermediate features from high frequency to low frequency, and then calculates the output features from low frequency to high frequency by combining the intermediate features; Step 3: construct a discriminator network; the discriminator network consists of a four-layer convolutional neural network, the convolutional layer step size is 2, the size of each convolutional layer is halved, the channel changes of the four convolutional layers are 3 to 64, 64 to 128, 128 to 256, 256 to 512, and the output of each convolutional layer is compressed into a single-channel feature probability map; when the discriminator is judged to be true, all single-channel feature probability maps are close to 1, and when it is judged to be false, all single-channel feature probability maps are close to 0; Step 4: Define the loss function; The generator loss is defined as shown in formula (1): L G = L GAN-G + L mse + L cos + L bf + L feat (1) Among them, L GAN-G represents the intrinsic loss function, L mse represents the mean square error function, L cos represents the cosine loss function, L feat represents the feature loss function; Intrinsic loss function L GAN-G The calculation formula is shown in formula (2): Among them, W k represents the normalized weight parameter of the kth layer, k represents the network layer number, fake_output k Indicates the probability that the output image is false, and ones indicates the probability is 1; Mean square error function L mse The calculation formula is shown in formula (3): Among them, fake_image i Represents the output of the reciprocal i-th layer feature map of the decoder, true_image i represents the image label of the same size as the output of the feature map of the i-th layer of the decoder; Cosine loss function L cos The calculation formula is shown in formula (4): Among them, fake_reion j Indicates the jth region of the generated image, true_region j Represents the j-th block area of ​​the generated image; Cross bilateral filtering loss L bf The calculation formula is shown in formula (5)-(6): Among them, L bf represents the double-sideband filtering loss, bf represents the double-sideband filtering, C represents the label image, {A, S} represents the reflection map and the illumination map respectively, and J p represents the output of the bilateral filter, C p Represents the value of the pth pixel of the label image, N p represents the total number of p pixels and neighboring pixels, W p represents the normalized weight, q represents the neighbor pixel position of p, N(p) represents the neighbor pixel position set of the p-th pixel, represents the spatial Gaussian kernel, p represents the position of the pth pixel, represents the range Gaussian kernel, C q Represents the value of neighbor pixel q; L feat The calculation formula is shown in formula (8): Where l represents the lth layer of the VGG network, F l Indicates the number of channels of the feature map of the lth layer, H l represents the height of the feature map of the lth layer, W l represents the width of the feature map of the lth layer, Represents the feature activation value of the lth layer; The discriminator loss is defined as shown in formula (9): Among them, L2 represents l1 loss, y i represents the ground-truth image, f(x i ) represents the image after super resolution; Step 5: Network training When the reflectance map generator or the illumination map generator is trained, the reflectance map or the illumination map output by the reflectance map generator or the illumination map generator is input into the discriminator, and the discriminator compares the input reflectance map or the illumination map with the label image, and outputs the probability that the reflectance map or the illumination map is consistent with the label image; Step 6: Input the original image to be processed into the reflection map generator or illumination map generator obtained in step 5 respectively, and the output image is the reflection map or illumination map obtained by decomposing the original image.

2. The essential image decomposition method based on bilateral feature pyramid network and multi-scale identification according to claim 1 is characterized in that Step 1 is as follows: A certain number of images are randomly selected from the test image data set, and then M small images of size N*N are randomly sampled from each image. These M small images are flipped horizontally or upside down to obtain M new small images, so that each image obtains 2*M small images; the small images obtained after performing the above operations on all the extracted images constitute the training image sample library.

3. The essential image decomposition method based on bilateral feature pyramid network and multi-scale identification according to claim 1 is characterized in that Step 4 is specifically as follows: the reflection map generator and the discriminator are combined to train the reflection map generator, and the illumination map generator and the discriminator are combined to train the illumination map generator; the present invention uses MIT and MPI data sets to train the generator network, adopts the SGD optimization method to update the network parameters, stops training when the loss function value is minimized, and obtains the final trained network.

4. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

5. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

Images