An image harmonization method based on multi-view image feature fusion

Abstract

This invention discloses an image harmonization method that can simultaneously use multiple perspective images to adjust the color of a portrait. The method consists of three modules: a semantic parsing module, a harmonization backbone module, and a feature fusion module. The semantic parsing module extracts semantic features from images and outputs feature maps that are embedded in the harmonization backbone module. The harmonization backbone module is a symmetrical network structure consisting of an encoder and a decoder. The fusion module fuses the feature maps of different images through the harmonization backbone module encoder and then feeds them into the decoder to produce the final harmonized image.

Description

Technical Field

[0001] The invention discloses an image harmonization method based on multi-view image feature fusion, and belongs to the field of image synthesis. Background Art

[0002] Image harmonization is a common operation in image editing. In image synthesis, because the foreground and background are captured under different environmental conditions (such as lighting and angle), significant visual inconsistencies can occur between the two after synthesis, reducing the realism of the synthesized image. Therefore, image harmonization aims to adjust the appearance of foreground objects in the synthesized image to match the background, ensuring a consistent visual style across the entire image and enhancing the realism of the synthesized image. With the rise of the metaverse, the demand for reconstructing real people in virtual worlds and creating realistic visual images has increased dramatically. However, traditional 3D reconstruction methods require specialized equipment to meticulously scan the person, then import the scanned data into 3D software for modeling and rendering. This is not only expensive, time-consuming, and inconvenient, but also limits the widespread adoption of metaverse-related applications among ordinary users. Therefore, it is necessary to use image harmonization methods to extract the appearance characteristics of people from their images and perform realistic synthesis of the person image with the background image in the virtual world.

[0003] Image harmonization methods are primarily divided into two categories: those based on manually extracted features and those based on deep learning. Manually extracted feature-based methods primarily rely on human knowledge to extract low-level image appearance statistics and adjust them to ensure low-level continuity between the foreground and background of the synthesized image. Manually extracted feature-based methods require prior knowledge, but the design of prior knowledge is often complex and subjective. Therefore, deep learning-based methods, which do not require manual prior knowledge, show greater potential for development. Deep learning-based methods can be further categorized into two types: one uses an encoder-decoder neural network that takes the synthesized image and the corresponding mask as input and directly outputs a harmonized image; the other uses a neural network to generate several filters based on the image to be synthesized and then applies these filters to the original image to obtain the final harmonized image. However, existing deep learning-based methods only extract foreground and background features from a single-view image. This limits the network to understanding the relationship between the foreground and background within a single image and fails to fully understand the overall characteristics of the image, resulting in poor synthesis results. Summary of the Invention

[0004] To address the above issues, the present invention provides an image harmonization method that can simultaneously utilize features from multiple perspectives to adjust the color of the foreground of a composite image. This method constructs a feature fusion and harmonization network that simultaneously extracts features from multiple images captured from different angles and fuses these extracted features to obtain complete scene information that cannot be fully reflected by a single image, thereby producing the final harmonized image result. The network consists of three modules: a semantic parsing module, a harmonization backbone module, and a feature fusion module. The technical solutions proposed by the present invention are as follows:

[0005] An image harmonization method based on multi-view image feature fusion simultaneously extracts features from several images taken from different shooting angles and fuses the extracted features to obtain complete scene information that cannot be fully reflected by a single image, thereby generating a final harmonized image result. The method specifically includes the following steps:

[0006] S1. Obtain image data: Obtain N sets of synthetic images and real image pairs from the same scene from different shooting angles from the dataset. For each set of synthetic image and real image pair, the real image is directly shot, the background of the real image remains unchanged, and the appearance of the foreground object is artificially modified to obtain a synthetic image. The foreground and background of the synthetic image have certain visual style differences. At the same time, for each synthetic image, there is a mask image that distinguishes the foreground and background. The N synthetic images are respectively used I c0 , I c1 ,……,I c(N-1) To represent, the corresponding mask is M0, M1, ..., M N-1 To represent, the corresponding real image is represented by I r0 , I r1 ,……,I r(N-1) To express;

[0007] S2. Construct a feature fusion and harmonization network. The network consists of three parts: a semantic parsing module, a harmonization backbone module, and a feature fusion module. The semantic parsing module is responsible for extracting semantic features from the image and outputting feature maps to embed into the harmonization backbone module. The harmonization backbone module is an encoder-decoder structure network. The image is input into the encoder to obtain a series of feature maps with decreasing sizes and increasing number of channels. The feature fusion module fuses different images through the last layer of feature maps of the harmonization backbone module encoder and then sends them to the decoder. In the decoder, the size of the feature map increases and the number of channels decreases, and finally a harmonized image with the same size as the input and 3 channels is obtained.

[0008] S3. The network takes a combination of N synthetic images and masks as input and obtains N harmonized images. The harmonized images and the real images constitute pixel-level loss, and the encoder output features and the fused features constitute L1 loss. The weighted sum of the above two losses is used as the total loss function to constrain the network training process, continuously update the network parameters, and generate the optimal model;

[0009] S4. Input the test set into the optimal model to generate a harmonized image.

[0010] Furthermore, the semantic parsing module in step S2 includes two parts, namely a high-resolution learning representation part and an object context representation part. The high-resolution learning representation part includes a first-stage module, a second-stage module, a third-stage module, and a fourth-stage module connected in sequence: the first stage is composed of a high-resolution convolution block, and the second, third, and fourth stages are composed of high-resolution convolution blocks with increasing numbers of parallel branches; the object context part representation is composed of an object attention block and a batch normalization random drop block. The object attention block uses the principle of the attention mechanism to process the feature map output by the high-resolution learning representation part, and then sends it to the batch normalization drop block for batch normalization and random drop. The final feature map is sent to the second layer of the encoder of the harmonization backbone module;

[0011] The harmonization backbone module in step S2 consists of an encoder and a decoder, and is a symmetrical structure. Each convolution block of the encoder and decoder includes a convolution layer, a batch normalization layer, and an ELU activation layer. The two symmetrical convolution blocks are connected using a skip connection.

[0012] The feature fusion module in step S2 consists of a convolutional layer, a batch normalization layer, a random dropout layer and a LeakyReLU activation layer.

[0013] Furthermore, the specific process of step S2 is: synthesize the image I c0 , I c1 ,……,I c(N-1) In turn, they are sent to the semantic parsing module, and then the corresponding masks M0, M1, ..., M N-1 After one layer of convolution, it is added to the feature map output by the first layer of the semantic parsing module, and subsequent semantic parsing processing is performed to obtain N semantic feature maps; the synthesized image I c0 , I c1 ,……,I c(N-1) and the corresponding masks M0, M1, ..., M N-1After being connected in the channel dimension, they are sent to the encoder of the harmonization backbone module in turn. The corresponding semantic feature map is connected with the third layer feature map of the harmonization backbone module encoder in the channel dimension and sent to the next layer of the encoder. The last layer feature map output by the N encoders is connected in the channel dimension and sent to the feature fusion module for processing to obtain a feature map with the same number of channels as the encoder output feature map, which is sent to the first layer of the decoder. Finally, the decoder outputs the harmonization result, and then the foreground object is segmented according to the corresponding mask and pasted to the corresponding position of the original image to be synthesized to obtain the final harmonized image I h0 , I h1 ...I h(N-1) , the process can be expressed by the following formula:

[0014] semantic_feature i =Semantic_Parsing_Module(I ci ,M i ) i=0,1,……,N-1

[0015] encoder_feature i =Encoder(Cat(I ci ,M i ), semantic_feature i ) i=0,1,……,N-1

[0016]

[0017] I result_i =Decoder(Encoder(Cat(I ci ,M i ), semantic_feature i ), fusion_feature)i=0,1,...,N-1

[0018]

[0019] Among them, Semantic_Parsing_Module represents the semantic parsing module, Encoder represents the encoder of the harmonized backbone module, Decoder represents the decoder of the harmonized backbone module, Cat represents the channel dimension connection operation, encoder_feature i Represents the feature map output by the encoder, Fusion_Moudle represents the feature fusion module, and fusion_feature represents the feature map output by the feature fusion module.

[0020] Furthermore, step S3 specifically includes:

[0021] Pixel-level reconstruction loss function, the formula is

[0022]

[0023] Among them, L rec represents the pixel-level reconstruction loss, I hi Represents the harmonized image predicted by the network, I r Represents the real image, M represents the mask, h represents the height of the image, and w represents the width of the image;

[0024] The L1 loss of the encoder output feature and the fusion feature is:

[0025]

[0026] Among them, L fea Represents L1 loss, encoder_feature i Represents the feature map output by the encoder, and fusion_feature represents the feature map output by the feature fusion module;

[0027] The loss function used by the entire network during training is the weighted sum of the above two loss functions:

[0028] Loss = L rec +λL fea

[0029] Loss represents the loss function of the entire network, and λ represents the artificially set weight parameter, which is set to 0.1 here.

[0030] Compared with other existing technologies, the present invention has the following beneficial effects:

[0031] 1. By using a pre-trained semantic parsing module to extract the semantic features of the input image and embedding them into the harmonized backbone module, the harmonized backbone module obtains richer semantic information and enhances the feature representation ability of the network.

[0032] 2. The features of multiple perspective images are fused through the feature fusion module, so that the network has a more comprehensive understanding of the scene content of the input image, improving the performance and prediction ability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the process of the multi-view feature fusion image harmonization method according to the present invention;

[0034] Figure 2 Schematic diagram of the principle of the multi-view feature fusion image harmonization method according to the present invention. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly visible, the present invention is further described below with reference to the accompanying drawings.

[0036] See also Figure 1 and Figure 2 The present invention provides an image harmonization method capable of simultaneously utilizing the features of multiple perspective images to adjust the color of the foreground of a composite image, comprising the following steps:

[0037] S1. Obtain image data: Obtain N sets of synthetic image and real image pairs from the same scene taken from different shooting angles from the dataset. For each set of synthetic image and real image pair, the real image is directly captured. The background of the real image remains unchanged, and the appearance of the foreground object is artificially modified to obtain a synthetic image. The foreground and background of the synthetic image have certain visual style differences. At the same time, for each synthetic image, there is a mask image that distinguishes the foreground and background. The N synthetic images are respectively used I c0 , I c1 ...I c(N-1) To indicate, the corresponding mask is M0, M1...M N-1 To represent, the corresponding real image is represented by I r0 , I r1 ...I r(N-1) To express;

[0038] S2. Construct a feature fusion and harmonization network. The network consists of three parts: a semantic parsing module, a harmonization backbone module, and a feature fusion module. The semantic parsing module is responsible for extracting semantic features from the image and outputting feature maps that are embedded in the harmonization backbone module. The harmonization backbone module is an encoder-decoder network. The image is input into the encoder, which produces a series of feature maps with decreasing size and increasing number of channels. The feature fusion module fuses the feature maps of different images through the last layer of the harmonization backbone module encoder and then feeds it into the decoder. In the decoder, the size of the feature map increases and the number of channels decreases, ultimately resulting in a harmonized image with the same size as the input and 3 channels.

[0039] S3: The network takes N synthetic images and masks as input and generates N harmonized images. The harmonized images and the real images form a pixel-level loss, and the encoder output features and the fused features form an L1 loss. The weighted sum of the two losses is used as the total loss function to constrain the network training process, continuously update the network parameters, and generate the optimal model.

[0040] S4. Put the test set into the optimal model to generate a harmonized image.

[0041] Step S1 specifically includes:

[0042] Obtain image data: Obtain N sets of synthetic images and real image pairs from the dataset, which are taken from different shooting angles in the same scene. For each set of synthetic image and real image pair, the real image is directly captured. The background of the real image remains unchanged, and the appearance of the foreground object is artificially modified to obtain a synthetic image. The foreground and background of the synthetic image have certain visual style differences. At the same time, for each synthetic image, there is a mask image that distinguishes the foreground and background. The N synthetic images are respectively used I c0 , I c1 ...I c(N-1) To indicate, the corresponding mask is M0, M1...M N-1 To represent, the corresponding real image is represented by I r0 , I r1 ...I r(N-1) The synthetic image and the real image are both 3-channel RGB images, with each pixel value between 0 and 255. The mask is a 1-channel black and white image, with each pixel having only two values: 0 or 1. If the value is 1, it means that the pixel at that position in the image to be harmonized is part of the foreground. If the value is 0, it means that the pixel at that position in the image to be harmonized is part of the background.

[0043] Step S2 specifically includes:

[0044] N RGB composite images I c0 , I c1 ...I c(N-1) The size of each image is modified to 256*256, and then normalized using the normalization parameters [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225]. This converts the RGB image into a tensor of size (3, 256, 256). N tensors are sequentially fed into the semantic parsing module. The semantic parsing module consists of two parts: a high-resolution learning representation part and an object context representation part. The high-resolution learning representation part includes a first-stage module, a second-stage module, a third-stage module, and a fourth-stage module connected in sequence. The first stage consists of a high-resolution convolution block, and the second, third, and fourth stages consist of high-resolution convolution blocks with increasing numbers of parallel branches. The object context representation consists of an object attention block and a batch normalization random dropout block. The object attention block uses the principle of the attention mechanism to process the feature map output by the high-resolution learning representation. It then sends it to the batch normalization dropout block for batch normalization and random dropout, and finally obtains a feature map with a shape of (128, 64, 64). The process of this part can be expressed as follows:

[0045] semantic_featurei =Semantic_Parsing_Module(I ci ,M i ) i=0,1,……,N-1

[0046] N composite images I c0 , I c1 ,……,I c(N-1) And the corresponding masks M0, M1, ..., M N-1 After concatenating them along the channel dimension, N feature maps of shape (4, 256, 256) are generated and fed into the encoder of the harmonized backbone module. The encoder of the harmonized backbone module consists of seven basic convolutional blocks, each of which includes a convolutional layer, a batch normalization layer, and an ELU activation layer. After the first three layers, the resulting feature map is concatenated along the channel dimension with the feature map output by the semantic parsing module, and the next four layers are processed. The process for this part can be represented as follows:

[0047] encoder_feature i =Encoder(Cat(I ci ,M i ), semantic_feature i ) i=0,1,……,N-1

[0048] Since N composite images I c0 , I c1 ,……,I c(N-1) And the corresponding masks M0, M1, ..., M N-1 They are connected in the channel dimension and then sent to the encoder in sequence. Therefore, the last layer of the encoder will output N feature maps. These N feature maps are connected in the channel dimension and sent to the feature fusion module for processing to obtain a fused feature as the input of the first layer of the decoder. The feature maps of other layers of the encoder are connected to the feature maps of the corresponding layer of the decoder through jump connections as the input of the next layer. Finally, the decoder outputs the harmonized result, and then the foreground object is segmented according to the corresponding mask and pasted to the corresponding position of the original synthetic image to obtain the final harmonized image. The process can be expressed by the following formula:

[0049]

[0050] I result_i =Decoder(Encoder(Cat(I ci ,M i ), semantic_feature i ), fusion_feature)i=0,1,...,N-1

[0051]

[0052] Step S3 specifically includes:

[0053] Since the final harmonized image I h0 , I h1 ,……,I h(N-1) The background part and I c0 , I c1 ,……,I c(N-1) The foreground parts of different images are completely consistent, and the sizes of the foreground parts are significantly different. Therefore, the foreground normalized mean square error is used as the pixel-level reconstruction loss function. The calculation formula of the pixel-level reconstruction loss function is as follows:

[0054]

[0055] Among them, L rec represents the pixel-level reconstruction loss, I hi Represents the harmonized image predicted by the network, I r Represents the real image, M represents the mask, h represents the height of the image, and w represents the width of the image;

[0056] The L1 loss calculation formula of the encoder output feature and the fusion feature is as follows:

[0057]

[0058] Among them, L fea Represents L1 loss, encoder_feature i Represents the feature map output by the encoder, and fusion_feature represents the feature map output by the feature fusion module;

[0059] The loss function used by the entire network during training is the weighted sum of the above two loss functions:

[0060] Loss = L rec +λL fea

[0061] Loss represents the loss function of the entire network, and λ represents the artificially set weight parameter, which is set to 0.1 here.

[0062] Table 1 shows the comparison between the proposed method and other methods in terms of the three image reconstruction evaluation indicators: MSE, fMSE, and PSNR.

[0063] Table 1

[0064]

[0065] MSE stands for Mean Square Error, and smaller values ​​indicate a smaller difference between the harmonized image and the true image. fMSE calculates the MSE of only the foreground. PSNR stands for Peak Signal-to-Noise Ratio, and larger values ​​indicate better image quality. Table 1 shows that our method achieves the best results in all three metrics: MSE, fMSE, and PSNR. This indicates that our method can generate results that are closer to the true image than other methods, and therefore outperforms them.

Claims

1. An image harmonization method based on multi-view image feature fusion. This method simultaneously extracts features from multiple images taken from different angles and fuses the extracted features to obtain complete scene information that cannot be fully reflected by a single image, thereby producing a final harmonized image result. The method specifically includes the following steps: S1. Obtain image data: Obtain N sets of synthetic images and real image pairs from the same scene from different shooting angles from the dataset. For each set of synthetic image and real image pair, the real image is directly shot, the background of the real image remains unchanged, and the appearance of the foreground object is artificially modified to obtain a synthetic image. The foreground and background of the synthetic image have certain visual style differences. At the same time, for each synthetic image, there is a mask image that distinguishes the foreground and background. The N synthetic images are respectively used 、 、……、 To represent, the corresponding mask is used 、 、……、 To represent, the corresponding real image is represented by 、 To express; S2. Construct a feature fusion and harmonization network. The network consists of three parts: semantic parsing module, harmonization backbone module and feature fusion module. The semantic parsing module is responsible for extracting semantic features from the image and outputting feature maps to be embedded in the harmonized backbone module. The semantic parsing module consists of two parts: a high-resolution learning representation part and an object context representation part. The high-resolution learning representation part includes a first-stage module, a second-stage module, a third-stage module, and a fourth-stage module connected in sequence: the first stage consists of a high-resolution convolution block, and the second, third, and fourth stages consist of high-resolution convolution blocks with increasing numbers of parallel branches; the object context representation consists of an object attention block and a batch normalization random drop block. The object attention block uses the principle of the attention mechanism to process the feature map output by the high-resolution learning representation part, and then sends it to the batch normalization drop block for batch normalization and random drop, and the final feature map is sent to the second layer of the encoder of the harmonized backbone module; The harmonization backbone module consists of an encoder and a decoder, and is a symmetrical structure. Each convolution block of the encoder and decoder includes a convolution layer, a batch normalization layer, and an ELU activation layer. The two symmetrical convolution blocks are connected using a skip connection. The image is input to the encoder to obtain a series of feature maps with decreasing size and increasing number of channels. The feature fusion module fuses the feature maps of different images through the last layer of the harmonization backbone module encoder and then sends them to the decoder. In the decoder, the size of the feature map is continuously increased and the number of channels is continuously reduced, and finally a harmonized image with the same size as the input and 3 channels is obtained. The feature fusion module consists of a convolutional layer, a batch normalization layer, a random dropout layer, and a LeakyReLU activation layer. S3. The network takes a combination of N synthetic images and masks as input and obtains N harmonized images. The harmonized images and the real images constitute pixel-level loss. The encoder output features and the fused features constitute L1 loss. The weighted sum of the two losses is used as the total loss function to constrain the network training process, continuously update the network parameters, and generate the optimal model. S4. Input the test set into the optimal model to generate a harmonized image.

2. The image harmonization method based on multi-view image feature fusion according to claim 1, characterized in that: The specific process of step S2 is: synthesize the image 、 、……、 In turn, they are sent to the semantic parsing module, and then the corresponding masks are 、 After one layer of convolution, it is added to the feature map output by the first layer of the semantic parsing module, and subsequent semantic parsing processing is performed to obtain N semantic feature maps; the synthesized image 、 、……、 and the corresponding mask 、 After being connected in the channel dimension, they are sent to the encoder of the harmonization backbone module in turn. The corresponding semantic feature map is connected with the third layer feature map of the harmonization backbone module encoder in the channel dimension and sent to the next layer of the encoder; the last layer feature map output by the N encoders is connected in the channel dimension and sent to the feature fusion module for processing to obtain a feature map with the same number of channels as the encoder output feature map, which is sent to the first layer of the decoder. Finally, the decoder outputs the harmonization result, and then the foreground object is segmented according to the corresponding mask and pasted to the corresponding position of the original image to be synthesized to obtain the final harmonized image. 、 , the process is expressed by the following formula: in Represents the semantic parsing module, represents the encoder of the harmonized backbone module, represents the decoder of the harmonized backbone module, Cat represents the channel dimension connection operation, represents the feature map output by the encoder, represents the feature fusion module, Represents the feature map output by the feature fusion module.

3. The image harmonization method based on multi-view image feature fusion according to claim 1, characterized in that: Step S3 specifically includes: Pixel-level reconstruction loss function, the formula is in, represents the pixel-level reconstruction loss, represents the harmonized image of the network predictions, represents the real image, represents the mask, Represents the height of the image, Represents the width of the image; The L1 loss of the encoder output feature and the fusion feature is: in, represents L1 loss, represents the feature map output by the encoder, Represents the feature map output by the feature fusion module; The loss function used by the entire network during training is a weighted sum of two loss functions: Represents the loss function of the entire network, Represents the manually set weight parameter, which is set to 0.1 here.

Images