A method and system for harmonizing synthetic images based on foreground reference images

Abstract

The present invention relates to a method and system for harmonizing a synthetic image based on a foreground reference image. The method comprises: processing the original synthetic image using its foreground mask and a pre-trained target detection model to obtain a reference image for the foreground image; performing image pre-processing on the true label image, the original synthetic image, and the reference image to obtain a training set; constructing a dual-branch synthetic image harmonization network based on the foreground reference image, the network consisting of three parts: a background feature extraction and fusion module, a foreground reference image feature extraction and fusion module, and a dual-branch feature fusion fine-tuning module; constructing a target loss function for the synthetic image harmonization network; training the synthetic image harmonization network using training set images; performing foreground reference image extraction on a new synthetic image, then inputting the new synthetic image into the trained synthetic image harmonization model to output a harmonized image. The method and system are advantageous for improving the quality of synthetic image harmonization.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a method and system for harmonizing a synthetic image based on a foreground reference image. Background Art

[0002] Image synthesis is a fundamental operation in computer graphics. Cutting out a portion of an image, called the foreground, is done through a process called cropping. Inserting the foreground into another image is called image synthesis. This technique has widespread applications in everyday life, such as augmented reality, artistic creation, e-commerce advertising, graphic design, movie posters, background replacement for portraits, and online group photos. Furthermore, to prevent image synthesis from being used to create fake news and spread online rumors, synthetic image detection technology is particularly important. However, synthesized images often appear quite dissonant. Humans, having seen numerous real-world examples, can readily identify whether a given synthetic image is "real." To make the synthesized image appear more "realistic," two operations are required: first, extracting the foreground object by calculating a mask, and second, adjusting the foreground's appearance relative to the new background. The first problem has received considerable attention in recent years. As for the second, after professional image adjustment to adjust brightness, contrast, hue, and saturation, most synthetic images can appear relatively harmonious. Adjusting composite images to make them look harmonious is a challenging task for ordinary users, and it takes considerable time and effort for professionals. Therefore, it is particularly important to automatically achieve the harmony of composite images.

[0003] In recent years, deep learning methods have been widely used in image processing and computer vision. Zhu et al. first applied deep learning to the task of synthetic image harmonization. They trained a convolutional neural network to predict the visual realism of images in terms of color, lighting, and texture to distinguish synthetic images from real images. They then incorporated the predicted realism scores into a color optimization function to adjust the appearance of the foreground. Deep learning requires large datasets of paired synthetic and real images for training. However, image harmonization requires human experts to process the synthetic images to produce high-quality harmonized images as comparison real images, which is not feasible with large-scale data collection. Tsai et al. extracted the foreground of real images from semantic segmentation datasets, altered the style of the foreground using color adjustment methods, and then stitched them back together to create a synthetic image, with the original image serving as the real image. Using this method, Tasi et al. generated a large number of paired datasets. They also designed an end-to-end deep convolutional neural network that leveraged the mask information contained in the semantic segmentation dataset to guide the harmonization process of the synthetic images. Based on the method of Tasi et al., Cong et al. created and released the first large-scale open-source dataset for synthetic image harmonization. They also introduced the concept of domain, treating the synthetic image harmonization task as a domain transfer task, and designed a domain verification discriminator to shift the domain of the foreground to match the domain of the background, resulting in a harmonious image. Guo et al. believed that the disharmony between the foreground and background stems from their inherent reflectivity and lighting differences. Based on this, they proposed an autoencoder to separately harmonize the reflectance and lighting of synthetic images. For the foreground reflection, a reflection consistency loss is used to keep it unaffected by the background while adjusting it. For the foreground lighting, an illumination consistency loss is used to learn the change from foreground lighting to background lighting. Ling et al. defined the synthetic image harmonization task as a style transfer problem from background to foreground and proposed a region-aware adaptive instance normalization module to explicitly express the visual style from the background and then adaptively apply it to the foreground.

[0004] Most current synthetic image harmonization deep learning models simply leverage the powerful feature extraction capabilities of deep learning to extract background features from the synthetic image and then fuse them into the foreground. Furthermore, some studies divide the feature information in the background into illumination and reflection information, extracting them separately. In reality, the background region of a synthetic image is more complex than the foreground, containing information from the near, mid, and distant views, as well as high-light and low-light areas. The background is often composed of a variety of objects, including people, vehicles, plants, and streets. Compared to the background, the foreground has a single object and a more consistent overall style. Therefore, simply extracting information from the background will cause unimportant features to be integrated into the foreground, while important features may be ignored. Summary of the Invention

[0005] The object of the present invention is to provide a method and system for harmonizing a composite image based on a foreground reference image, which method and system are conducive to improving the quality of the harmonization of the composite image.

[0006] To achieve the above object, the present invention adopts a technical solution: a method for harmonizing a composite image based on a foreground reference image, comprising the following steps:

[0007] Step A: Process the original composite image using the foreground mask of the original composite image and the target detection pre-trained model to obtain a reference image of the foreground image, and perform image preprocessing on the true label image, the original composite image, and the reference image to obtain a training set;

[0008] Step B: constructing a dual-branch synthetic image harmonization network based on the foreground reference image, wherein the synthetic image harmonization network consists of three parts: a background feature extraction and fusion module, a foreground reference image feature extraction and fusion module, and a dual-branch feature fusion fine-tuning module;

[0009] Step C, constructing the target loss function of the synthetic image and harmonization network;

[0010] Step D: using the training set images to train the synthetic image harmonization network to obtain a trained synthetic image harmonization model;

[0011] Step E: extract the foreground reference image from the new composite image, then input it into the trained composite image harmonization model, and output the harmonized image.

[0012] Furthermore, the step A specifically includes the following steps:

[0013] Step A1: resize the true label image, the original synthetic image, and the foreground mask to a uniform H×W pixel size;

[0014] Step A2: Use the foreground mask to set the pixel values ​​of the background part of the original synthetic image to 0 to obtain a foreground image, then use the yolov5 target detection pre-trained model to extract a single foreground target in the foreground image, then invert the foreground mask to obtain a background mask, use the background mask to set the pixel values ​​of the foreground part of the original synthetic image to 0 to obtain a background image, then use the yolov5 target detection pre-trained model to extract multiple background targets in the background image, resize all targets to H×W pixel size, calculate the cosine similarity between the foreground target and each background target, and use the background target with the highest value as the reference image of the original synthetic image; the formula for calculating the cosine similarity value cos(θ) is as follows:

[0015]

[0016] Where A and B represent the vectorized foreground and background targets respectively, and n represents the number of pixels in the image, i.e. H×W;

[0017] Step A3: Perform a uniform random flip operation on all the original synthetic images to be trained, the reference images, and the foreground masks to enhance the original synthetic images and the reference images;

[0018] Step A4: Normalize all original composite images and reference images. Given an original composite image or reference image I(i, j), the normalized image is Calculate normalized values The formula is as follows:

[0019]

[0020] Where (i, j) represents the position of the pixel.

[0021] Furthermore, the step B specifically includes the following steps:

[0022] Step B1: construct a background feature extraction and fusion module, which consists of an encoder, a decoder, and an output convolution block, and is used to extract background features of the synthesized image and fuse them with foreground features;

[0023] Step B2: constructing a foreground reference image feature extraction and fusion module, which consists of an encoder, two alignment modules, two residual blocks, a reference image feature fusion layer, and a decoder, and is used to extract foreground reference image features and fuse them with foreground features;

[0024] Step B3: Construct a dual-branch feature fusion fine-tuning module, which consists of three convolution blocks and is used to fuse dual-branch features and perform fine-tuning.

[0025] Furthermore, the specific implementation steps of step B1 are as follows:

[0026] Step B11: Construct a background feature extraction and fusion module. The input of the background feature extraction and fusion module is the normalized real label image I and the original synthetic image. and foreground mask M, original composite image The size of the foreground mask M is H×W×3, the size of the foreground mask M is H×W×1, and the output size is H×W×3 harmonized image The background feature extraction and fusion module consists of an encoder, a decoder and an output convolution block. The encoder contains 8 convolution blocks and the decoder contains 7 deconvolution blocks.

[0027] Step B12: Construct the encoder used in step B11. The encoder includes 8 convolution blocks. Input into the convolution block of the encoder in sequence, and the tth convolution block obtains the feature map X t , the size of X1 to X4 is The sizes from X5 to X8 are Where t represents the tth convolutional block in the encoder; the encoder outputs a feature map X8 of size 1×1×512;

[0028] Step B13: Construct the decoder used in step B11. The decoder includes 7 deconvolution blocks. For the first deconvolution block in the decoder, the feature map X8 obtained in step B12 is fed into the first deconvolution block of the decoder to obtain the feature map The feature map After being concatenated with the feature map X7 along the channel dimension, it is fed into the second deconvolution block of the decoder. Similarly, the input of the t-th deconvolution block in the decoder is the output feature of the previous deconvolution block. And the output feature X of the corresponding encoder 9-t Splicing along the channel dimension, the output is a feature map arrive The size is arrive The size is The final output size of the decoder is Feature map The feature map The concatenation with the feature map X1 along the channel dimension is fed into the final output convolution block to obtain a harmonized image of size H×W×3.

[0029] Step B14: Construct the convolution block in the encoder in step B12, wherein the convolution block in the encoder is composed of a convolution layer, a regularization layer, and an activation layer in sequence; the convolution layer uses a convolution kernel with a size of 4×4, a stride of 2, and a padding of 1; the regularization layer uses an instance regularization algorithm; and the activation layer uses a LeakyRelu activation function; in particular, the first convolution block in the encoder does not include a regularization layer;

[0030] Step B15: constructing a deconvolution block in the decoder in step B13, wherein the deconvolution block in the decoder is composed of a deconvolution layer, a background feature fusion layer, and an activation layer in sequence; the deconvolution layer uses a deconvolution kernel with a size of 4×4, a stride of 2, and a padding of 1, and the activation layer uses a ReLU activation function;

[0031] The last three deconvolution blocks in the encoder are composed of a deconvolution layer, a background feature fusion layer, an activation layer, and a channel attention layer in sequence; the channel attention layer consists of a convolution with a kernel size of 1×1 and a stride of 1 and a sigmoid function; the calculation formula is as follows:

[0032]

[0033] Among them, Conv1 represents the convolution with a convolution kernel size of 1×1, and BFIN represents the background feature fusion layer;

[0034] Step B16: Construct a background feature fusion layer used in step B15. The input of the background feature fusion layer is the feature map F output by the deconvolution layer in step B15 and a foreground mask of a size corresponding to F. The foreground feature map and the background feature map are obtained using the foreground mask. The mean μ and standard deviation σ of the foreground feature map are calculated by channel, and the mean γ and standard deviation β of the background feature map are calculated by channel. Then, the feature map F is normalized. The mean μ of the foreground feature map is subtracted from the feature map F, and then divided by the standard deviation σ of the foreground feature map. Then, the feature map F is multiplied by the mean γ of the background feature map, and then the standard deviation β of the background feature map is added. Finally, the foreground feature map and the background feature map are spliced ​​using the foreground mask to obtain a feature map that integrates the background features. The calculation formula is as follows:

[0035]

[0036] Where F represents the feature map, γ represents the mean of the background feature map, β represents the standard deviation of the background feature map, μ represents the mean of the foreground feature map, and σ represents the standard deviation of the foreground feature map;

[0037] Step B17: Construct the output convolution block used in step B11; the output convolution block consists of a deconvolution with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and a Tanh activation function in sequence; the output convolution block outputs a harmonized image of size H×W×3

[0038] Furthermore, the specific implementation steps of step B2 are as follows:

[0039] Step B21: Construct a foreground reference image feature extraction and fusion module. The input of the foreground reference image feature extraction and fusion module is the normalized real label image I and the original synthetic image. and reference images Original composite image and reference images The size of the foreground reference image is H×W×3, and the harmonized image of the output size is H×W×3; the foreground reference image feature extraction and fusion module is composed of an encoder, two alignment modules, two residual blocks, a reference image feature fusion layer and a decoder in sequence;

[0040] Step B22: Construct the encoder used in step B21, wherein the encoder includes 5 convolution blocks; and reference images Input into the convolution block of the encoder, and the synthetic image feature map is obtained by the tth convolution block and reference image feature map The size of each feature map is Each convolution block is composed of a convolution layer, an activation layer, and a regularization layer in sequence; the convolution layer uses a convolution kernel size of 4×4, a step size of 2, and a padding of 1. The activation layer uses the LeakyRelu activation function, and the regularization layer uses the instance regularization algorithm. The output of the encoder is five sets of feature maps of the synthetic image obtained by the five convolution blocks. arrive and five sets of feature maps of the reference image arrive

[0041] Step B23: construct two alignment modules used in step B21; the inputs of the two alignment modules are the five sets of feature maps of the composite image output by step B22. arrive and five sets of feature maps of the reference image arrive Each alignment module consists of five convolutions with a kernel size of 1x1. The input of each convolution is a corresponding set of feature maps from the five sets of input feature maps. The five sets of output feature maps of the convolution layer are adjusted to 32 and 32 in width and height by difference, and spliced ​​along the channel dimension. The final output is a synthetic image feature map of size 32×32×1024. and reference image feature map

[0042] Step B24, construct the two residual blocks used in step B21; the two residual blocks are composed of a convolution layer, an activation layer, a regularization layer, a convolution layer and a regularization layer in sequence, the convolution layer uses a convolution kernel size of 3×3, a step size of 1, and a padding of 1, the activation layer uses a ReLU activation function, and the regularization layer uses an instance regularization algorithm; the inputs of the two residual blocks are the synthetic image feature maps and reference image feature map Output size of synthetic image feature map is 32×32×1024 and reference image feature map

[0043] Step B25: Construct a reference image feature fusion layer; the input of the reference image feature fusion layer is the synthetic image feature map obtained in step B23. and reference image feature map Calculate the mean μ and standard deviation σ of the synthetic image feature map by channel, and calculate the mean γ and standard deviation β of the reference image feature map by channel, and then regularize the synthetic image feature map The synthetic image feature map Subtract the mean μ of the synthetic image feature map and divide it by the standard deviation σ of the synthetic image feature map, and then multiply it by the reference image feature map The mean γ of the reference image is added with the standard deviation β of the reference image feature map to obtain the feature map that combines the reference image features. The calculation formula is as follows:

[0044]

[0045] Then the feature map Input into the decoder, and the final output is a harmonized image with a size of H×W×3

[0046] Step B26: Construct the decoder used in step B21; the decoder includes three deconvolution blocks, each of which is composed of a deconvolution layer, a regularization layer, and an activation layer in sequence; the deconvolution layer uses a deconvolution kernel with a size of 4×4, a stride of 2, and a padding of 1; the regularization layer uses an instance regularization algorithm; and the activation layer uses a ReLU activation function;

[0047] Step B27. Construct the convolution block in the encoder in step B22, where the convolution block in the encoder is composed of a convolution layer, a regularization layer, and an activation layer in sequence; the convolution layer uses a convolution kernel size of 4×4, a step size of 2, and a padding of 1; the regularization layer uses an instance regularization algorithm; and the activation layer uses a LeakyRelu activation function; in particular, the first convolution block in the encoder does not contain a regularization layer.

[0048] Furthermore, the specific implementation steps of step B3 are as follows:

[0049] Step B31: Construct a dual-branch feature fusion fine-tuning module; the input of the dual-branch feature fusion fine-tuning module is the output obtained in step B1 and feature maps The output from step B2 And the foreground mask M; the feature map The deconvolution kernel is 3×3, and the feature map of size 256×256×16 is obtained. Will M and After splicing along the channel dimension and feeding into the fine-tuning module, the final harmonized image I′ is obtained;

[0050] Step B32: Construct the fine-tuning module used in step B31; the fine-tuning module consists of three convolution blocks, and the convolution block consists of a convolution layer, a regularization layer, and an activation layer in sequence; the convolution layer uses a convolution kernel size of 3×3, a step size of 1, and a padding of 1; the regularization layer uses a batch normalization algorithm, and the activation layer uses an ELU activation function.

[0051] Furthermore, the step C specifically includes the following steps:

[0052] Step C1: Construct the network target loss function, which is expressed as follows:

[0053] l=||I′-I||1

[0054] Among them, l is the L1 loss, I′ is the constructed synthetic image and the final output of the harmonization network, I is the real label image, and ||.||1 is the absolute value operation.

[0055] Furthermore, the step D specifically includes the following steps:

[0056] Step D1: randomly divide the corresponding true label images, original synthetic images, reference images, and foreground masks into multiple batches, each batch containing N groups of images;

[0057] Step D2: input the image pair into the image harmonization network described in step B to obtain the final harmonized image;

[0058] Step D3: Based on the target loss function of the image harmonization network, the back propagation method is used to calculate the gradient of each parameter in the image enhancement network, and the Adam optimization method is used to update the parameters, where the learning rate is changed from 2×10 -4 Slowly reduce to 2×10 -5 ;

[0059] Step D4: Repeat the image harmonization network training steps from step D1 to step D3 in batches. The entire process is iterated a set number of times, the network parameters are saved, and the image harmonization network training process is completed.

[0060] Furthermore, the specific implementation method of step E is: for a new synthetic image and foreground mask, the foreground target and background target are extracted using the foreground mask and the yolov5 target detection pre-training model, and then the cosine similarity is calculated to obtain a reference image, and the synthetic image, the reference image and the foreground mask are input into the image harmonization network described in step B to obtain a harmonized image.

[0061] The present invention also provides a synthetic image and harmonization system based on a foreground reference image, comprising a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0062] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes a method and system for harmonizing a synthetic image based on a foreground reference image. This method and system not only utilize the background features of the synthetic image, but also effectively utilize the features of textures and semantic blocks similar to or identical to the foreground to supplement the details and authenticity of the image harmonization, preventing the foreground image from being changed simply according to the background, and can be more suitable for scenes with more complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flowchart of a method implementation according to an embodiment of the present invention.

[0064] Figure 2 It is a diagram of the background feature extraction and fusion module in an embodiment of the present invention.

[0065] Figure 3 It is a reference graph feature extraction and fusion module diagram in an embodiment of the present invention.

[0066] Figure 4 2 is a diagram of a dual-branch and feature fusion fine-tuning module in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0068] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0069] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0070] like Figure 1 As shown, this embodiment provides a method for harmonizing a composite image based on a foreground reference image, comprising the following steps:

[0071] Step A: Use the foreground mask of the original synthetic image and the target detection pre-trained model to process the original synthetic image to obtain a reference image of the foreground image, and perform image preprocessing on the real label image, the original synthetic image and the reference image to obtain a training set.

[0072] In this embodiment, step A specifically includes the following steps:

[0073] Step A1: resize the true label image, original synthesized image, and foreground mask to a uniform H×W pixel size.

[0074] Step A2: Use the foreground mask to set the pixel values ​​of the background part of the original composite image to 0 to obtain a foreground image, then use the yolov5 target detection pre-trained model to extract a single foreground target in the foreground image, then invert the foreground mask to obtain a background mask, use the background mask to set the pixel values ​​of the foreground part of the original composite image to 0 to obtain a background image, then use the yolov5 target detection pre-trained model to extract multiple background targets in the background image, resize all targets to H×W pixel size, calculate the cosine similarity between the foreground target and each background target, and use the background target with the highest value as the reference image of the original composite image. The formula for calculating the cosine similarity value cos(θ) is as follows:

[0075]

[0076] Among them, A and B represent the vectorized foreground target and background target respectively, and n represents the number of pixels in the image, that is, H×W.

[0077] Step A3: Perform a uniform random flip operation on all original synthetic images to be trained, reference images, and foreground masks to enhance the original synthetic images and reference images.

[0078] Step A4: Normalize all original composite images and reference images. Given an original composite image or reference image I(i, j), the normalized image is Calculate normalized values The formula is as follows:

[0079]

[0080] Where (i, j) represents the position of the pixel.

[0081] Step B: construct a dual-branch synthetic image and harmonization network based on the foreground reference image. The synthetic image harmonization network consists of three parts: background feature extraction and fusion module, foreground reference image feature extraction and fusion module, and dual-branch feature fusion fine-tuning module.

[0082] In this embodiment, step B specifically includes the following steps:

[0083] Step B1: Construct a background feature extraction and fusion module. The background feature extraction and fusion module consists of an encoder, a decoder, and an output convolution block. The module can effectively extract the background features of the synthetic image and fuse them with the foreground features.

[0084] Specifically, the implementation steps of step B1 are as follows:

[0085] Step B11: Construct background feature extraction and fusion module, such as Figure 2 As shown, the input of the background feature extraction and fusion module is the normalized real label image I and the original synthetic image and foreground mask M, original composite image The size of the foreground mask M is H×W×3, the size of the foreground mask M is H×W×1, and the output size is H×W×3 harmonized image The background feature extraction and fusion module consists of an encoder, a decoder and an output convolution block. The encoder contains 8 convolution blocks and the decoder contains 7 deconvolution blocks.

[0086] Step B12: Construct the encoder used in step B11. The encoder includes 8 convolution blocks. Input into the convolution block of the encoder in sequence, and the tth convolution block obtains the feature map X t , the size of X1 to X4 is The sizes from X5 to X8 are Where t represents the tth convolutional block in the encoder. The encoder outputs a feature map X8 of size 1×1×512.

[0087] Step B13: Construct the decoder used in step B11. The decoder contains 7 deconvolution blocks. Taking the first deconvolution block in the decoder as an example, the feature map X8 obtained in step B12 is fed into the first deconvolution block of the decoder to obtain the feature map The feature map After being concatenated with the feature map X7 along the channel dimension, it is fed into the second deconvolution block of the decoder. Similarly, the input of the t-th deconvolution block in the decoder is the output feature of the previous deconvolution block. And the output feature X of the corresponding encoder 9-t Splicing along the channel dimension, the output is a feature map arrive The size is arrive The size is The final output size of the decoder is Feature map The feature map The concatenation with the feature map X1 along the channel dimension is fed into the final output convolution block to obtain a harmonized image of size H×W×3.

[0088] Step B14: Construct the convolutional block in the encoder from step B12. The convolutional block in the encoder consists of a convolutional layer, a regularization layer, and an activation layer in sequence. The convolutional layer uses a 4×4 kernel, a stride of 2, and a padding of 1. The regularization layer uses the instance regularization algorithm, and the activation layer uses the LeakyReLU activation function. In particular, the first convolutional block in the encoder does not include a regularization layer.

[0089] Step B15: Construct the deconvolution block in the decoder in step B13. The deconvolution block in the decoder is composed of a deconvolution layer, a background feature fusion layer, and an activation layer in sequence. The deconvolution layer uses a deconvolution kernel with a size of 4×4, a stride of 2, and a padding of 1. The activation layer uses a ReLU activation function.

[0090] In particular, the last three deconvolution blocks in the encoder are composed of a deconvolution layer, a background feature fusion layer, an activation layer, and a channel attention layer in sequence. The channel attention layer consists of a convolution with a kernel size of 1×1 and a stride of 1 and a sigmoid function. The calculation formula is as follows:

[0091]

[0092] Among them, Conv1 represents the convolution with a convolution kernel size of 1×1, and BFIN represents the background feature fusion layer.

[0093] Step B16: Construct the background feature fusion layer used in step B15. The input of the background feature fusion layer is the feature map F output by the deconvolution layer in step B15 and the foreground mask of the size corresponding to F. The foreground feature map and the background feature map are obtained using the foreground mask. The mean μ and standard deviation σ of the foreground feature map are calculated by channel, and the mean γ and standard deviation β of the background feature map are calculated by channel. Then, the feature map F is normalized. The mean μ of the foreground feature map is subtracted from the feature map F, and then divided by the standard deviation σ of the foreground feature map. The feature map F is then multiplied by the mean γ of the background feature map and then added with the standard deviation β of the background feature map. Finally, the foreground feature map and the background feature map are spliced ​​using the foreground mask to obtain a feature map that integrates the background features. The calculation formula is as follows:

[0094]

[0095] Among them, F represents the feature map, γ represents the mean of the background feature map, β represents the standard deviation of the background feature map, μ represents the mean of the foreground feature map, and σ represents the standard deviation of the foreground feature map.

[0096] Step B17: Construct the output convolution block used in step B11. The output convolution block consists of a deconvolution with a convolution kernel size of 4×4, a stride of 2, and a padding of 1, and a Tanh activation function in sequence. The output convolution block outputs a harmonized image of size H×W×3.

[0097] Step B2: Construct a foreground reference image feature extraction and fusion module. The foreground reference image feature extraction and fusion module consists of an encoder, two alignment modules, two residual blocks, a reference image feature fusion layer and a decoder. This module can effectively extract foreground reference image features and fuse them with foreground features.

[0098] Specifically, the implementation steps of step B2 are as follows:

[0099] Step B21: Construct a foreground reference image feature extraction and fusion module, such as Figure 3 As shown, the input of the foreground reference image feature extraction and fusion module is the normalized real label image I and the original synthetic image and reference images Original composite image and reference images The size of is H×W×3, and the harmonized image of size is H×W×3 is output. The foreground reference image feature extraction and fusion module is composed of an encoder, two alignment modules, two residual blocks, a reference image feature fusion layer and a decoder in sequence.

[0100] Step B22: Construct the encoder used in step B21, which includes 5 convolution blocks. and reference images Input into the convolution block of the encoder, and the synthetic image feature map is obtained by the tth convolution block and reference image feature map The size of each feature map is Each convolution block is composed of a convolution layer, an activation layer, and a regularization layer in sequence. The convolution layer uses a convolution kernel size of 4×4, a stride of 2, and a padding of 1. The activation layer uses the LeakyRelu activation function, and the regularization layer uses the instance regularization algorithm. The output of the encoder is five sets of feature maps of the synthesized image obtained by the five convolution blocks. arrive and five sets of feature maps of the reference image arrive

[0101] Step B23: Construct two alignment modules used in step B21. The inputs of the two alignment modules are the five sets of feature maps of the composite image outputted in step B22. arrive and five sets of feature maps of the reference image arrive Each alignment module consists of five convolutions with a kernel size of 1x1. The input of each convolution is a corresponding set of feature maps from the five sets of input feature maps. The five sets of output feature maps of the convolution layer are adjusted to 32 and 32 in width and height by difference, and spliced ​​along the channel dimension. The final output is a synthetic image feature map of size 32×32×1024. and reference image feature map

[0102] Step B24: Construct the two residual blocks used in step B21. The two residual blocks are composed of a convolution layer, an activation layer, a regularization layer, a convolution layer, and a regularization layer in sequence. The convolution layer uses a convolution kernel size of 3×3, a step size of 1, and a padding of 1. The activation layer uses the ReLU activation function, and the regularization layer uses the instance regularization algorithm. The inputs of the two residual blocks are the synthetic image feature maps. and reference image feature map Output size of synthetic image feature map is 32×32×1024 and reference image feature map

[0103] Step B25: Construct a reference image feature fusion layer. The input of the reference image feature fusion layer is the synthetic image feature map obtained in step B23. and reference image feature map Calculate the mean μ and standard deviation σ of the synthetic image feature map by channel, and calculate the mean γ and standard deviation β of the reference image feature map by channel, and then regularize the synthetic image feature map The synthetic image feature map Subtract the mean μ of the synthetic image feature map and divide it by the standard deviation σ of the synthetic image feature map, and then multiply it by the reference image feature map The mean γ of the reference image is added with the standard deviation β of the reference image feature map to obtain the feature map that combines the reference image features. The calculation formula is as follows:

[0104]

[0105] Then the feature map Input into the decoder, and the final output is a harmonized image with a size of H×W×3

[0106] Step B26: Construct the decoder used in step B21. The decoder contains three deconvolution blocks, each consisting of a deconvolution layer, a regularization layer, and an activation layer in sequence. The deconvolution layer uses a 4×4 kernel, a stride of 2, and a padding of 1. The regularization layer uses the instance regularization algorithm, and the activation layer uses the ReLU activation function.

[0107] Step B27: Construct the convolutional block in the encoder from step B22. The convolutional block in the encoder consists of a convolutional layer, a regularization layer, and an activation layer, in order. The convolutional layer uses a 4×4 kernel, a stride of 2, and a padding of 1. The regularization layer uses the instance regularization algorithm, and the activation layer uses the LeakyReLU activation function. In particular, the first convolutional block in the encoder does not contain a regularization layer.

[0108] Step B3: Construct a dual-branch feature fusion fine-tuning module. The dual-branch feature fusion fine-tuning module consists of three convolution blocks. This module can effectively fuse dual-branch features and perform fine-tuning.

[0109] Specifically, the implementation steps of step B3 are as follows:

[0110] Step B31: Construct a dual-branch feature fusion fine-tuning module, such as Figure 4 The input of the dual-branch feature fusion fine-tuning module is the output obtained in step B1. and feature maps The output from step B2 And the foreground mask M. The feature map The deconvolution kernel is 3×3, and the feature map of size 256×256×16 is obtained. Will M and After splicing along the channel dimension and sending it to the fine-tuning module, the final harmonized image I′ is obtained.

[0111] Step B32: Construct the fine-tuning module used in step B31. The fine-tuning module consists of three convolutional blocks, each consisting of a convolutional layer, a regularization layer, and an activation layer in sequence. The convolutional layer uses a 3×3 kernel, a stride of 1, and a padding of 1. The regularization layer uses batch normalization, and the activation layer uses the ELU activation function.

[0112] Step C: Construct the target loss function of the synthetic image and harmonization network.

[0113] In this embodiment, step C specifically includes the following steps:

[0114] Step C1: Construct the network target loss function, which is expressed as follows:

[0115] l=||I′-I||1

[0116] Among them, l is the L1 loss, I′ is the constructed synthetic image and the final output of the harmonization network, I is the real label image, and ||.||1 is the absolute value operation.

[0117] Step D: Use the training set images to train the synthetic image harmonization network to obtain a trained synthetic image harmonization model.

[0118] In this embodiment, step D specifically includes the following steps:

[0119] Step D1: randomly divide the corresponding true label images, original synthetic images, reference images and foreground masks into multiple batches, each batch contains N groups of images.

[0120] Step D2: Input the image pair into the image harmonization network described in step B to obtain the final harmonized image.

[0121] Step D3: Based on the target loss function of the image harmonization network, the back propagation method is used to calculate the gradient of each parameter in the image enhancement network, and the Adam optimization method is used to update the parameters, where the learning rate is changed from 2×10 -4 Slowly reduce to 2×10 -5 .

[0122] Step D4: Repeat the image harmonization network training steps from step D1 to step D3 in batches. The entire process is iterated 200 times. The network parameters are saved to complete the image harmonization network training process.

[0123] Step E: extract the foreground reference image from the new composite image, then input it into the trained composite image harmonization model, and output the harmonized image.

[0124] In this embodiment, the specific implementation method of step E is: for a new synthetic image and foreground mask, the foreground target and background target are extracted using the foreground mask and the yolov5 target detection pre-training model, and then the cosine similarity is calculated to obtain a reference image, and the synthetic image, the reference image and the foreground mask are input into the image harmonization network described in step B to obtain a harmonized image.

[0125] This embodiment also provides a synthetic image harmonization system based on a foreground reference image, comprising a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0126] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0127] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0130] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A method for harmonizing a synthetic image based on a foreground reference image, characterized in that: The following steps are involved: Step A: Process the original composite image using the foreground mask of the original composite image and the target detection pre-trained model to obtain a reference image of the foreground image, and perform image preprocessing on the true label image, the original composite image, and the reference image to obtain a training set; Step B: constructing a dual-branch synthetic image harmonization network based on the foreground reference image, wherein the synthetic image harmonization network consists of three parts: a background feature extraction and fusion module, a foreground reference image feature extraction and fusion module, and a dual-branch feature fusion fine-tuning module; Step C, constructing the target loss function of the synthetic image and harmonization network; Step D: using the training set images to train the synthetic image harmonization network to obtain a trained synthetic image harmonization model; Step E: extracting a foreground reference image from the new composite image, then inputting the trained composite image harmonization model to output a harmonized image; The step A specifically comprises the following steps: Step A1: resize the true label image, the original synthetic image, and the foreground mask to a uniform H×W pixel size; Step A2: Use the foreground mask to set the pixel values ​​of the background part of the original synthetic image to 0 to obtain a foreground image, then use the yolov5 target detection pre-trained model to extract a single foreground target in the foreground image, then invert the foreground mask to obtain a background mask, use the background mask to set the pixel values ​​of the foreground part of the original synthetic image to 0 to obtain a background image, then use the yolov5 target detection pre-trained model to extract multiple background targets in the background image, resize all targets to H×W pixel size, calculate the cosine similarity between the foreground target and each background target, and use the background target with the highest value as the reference image of the original synthetic image; the formula for calculating the cosine similarity value cos(θ) is as follows: Where A and B represent the vectorized foreground and background targets respectively, and n represents the number of pixels in the image, i.e. H×W; Step A3: Perform a uniform random flip operation on all the original synthetic images to be trained, the reference images, and the foreground masks to enhance the original synthetic images and the reference images; Step A4: Normalize all original composite images and reference images. Given an original composite image or reference image I(i, j), the normalized image is Calculate normalized values The formula is as follows: Where (i, j) represents the position of the pixel.

2. The method for harmonizing a composite image based on a foreground reference image according to claim 1, characterized in that: The step B specifically comprises the following steps: Step B1: construct a background feature extraction and fusion module, which consists of an encoder, a decoder, and an output convolution block, and is used to extract background features of the synthesized image and fuse them with foreground features; Step B2: constructing a foreground reference image feature extraction and fusion module, which consists of an encoder, two alignment modules, two residual blocks, a reference image feature fusion layer, and a decoder, and is used to extract foreground reference image features and fuse them with foreground features; Step B3: Construct a dual-branch feature fusion fine-tuning module, which consists of three convolution blocks and is used to fuse dual-branch features and perform fine-tuning.

3. The method for harmonizing a composite image based on a foreground reference image according to claim 2, characterized in that: The specific implementation steps of step B1 are as follows: Step B11: Construct a background feature extraction and fusion module. The input of the background feature extraction and fusion module is the normalized real label image I and the original synthetic image. and foreground mask M, original composite image The size of the foreground mask M is H×W×3, the size of the foreground mask M is H×W×1, and the output size is H×W×3 harmonized image The background feature extraction and fusion module consists of an encoder, a decoder and an output convolution block. The encoder contains 8 convolution blocks and the decoder contains 7 deconvolution blocks. Step B12: Construct the encoder used in step B11. The encoder includes 8 convolution blocks. Input into the convolution block of the encoder in sequence, and the tth convolution block obtains the feature map X t , the size of X1 to X4 is The sizes from X5 to X8 are Where t represents the tth convolutional block in the encoder; the encoder outputs a feature map X8 of size 1×1×512; Step B13: Construct the decoder used in step B11. The decoder includes 7 deconvolution blocks. For the first deconvolution block in the decoder, the feature map X8 obtained in step B12 is fed into the first deconvolution block of the decoder to obtain the feature map The feature map After being concatenated with the feature map X7 along the channel dimension, it is fed into the second deconvolution block of the decoder. Similarly, the input of the t-th deconvolution block in the decoder is the output feature of the previous deconvolution block. And the output feature X of the corresponding encoder 9-t Splicing along the channel dimension, the output is a feature map arrive The size is arrive The size is The final output size of the decoder is Feature map The feature map The concatenation with the feature map X1 along the channel dimension is fed into the final output convolution block to obtain a harmonized image of size H×W×3. Step B14: Construct the convolution block in the encoder in step B12, wherein the convolution block in the encoder is composed of a convolution layer, a regularization layer, and an activation layer in sequence; the convolution layer uses a convolution kernel with a size of 4×4, a stride of 2, and a padding of 1; the regularization layer uses an instance regularization algorithm; and the activation layer uses a LeakyRelu activation function; the first convolution block in the encoder does not include a regularization layer; Step B15: constructing a deconvolution block in the decoder in step B13, wherein the deconvolution block in the decoder is composed of a deconvolution layer, a background feature fusion layer, and an activation layer in sequence; the deconvolution layer uses a deconvolution kernel with a size of 4×4, a stride of 2, and a padding of 1, and the activation layer uses a ReLU activation function; The last three deconvolution blocks in the encoder are composed of a deconvolution layer, a background feature fusion layer, an activation layer, and a channel attention layer in sequence; the channel attention layer consists of a convolution with a kernel size of 1×1 and a stride of 1 and a sigmoid function; the calculation formula is as follows: Among them, Conv1 represents the convolution with a convolution kernel size of 1×1, and BFIN represents the background feature fusion layer; Step B16: Construct a background feature fusion layer used in step B15. The input of the background feature fusion layer is the feature map F output by the deconvolution layer in step B15 and a foreground mask of a size corresponding to F. The foreground feature map and the background feature map are obtained using the foreground mask. The mean μ and standard deviation σ of the foreground feature map are calculated by channel, and the mean γ and standard deviation β of the background feature map are calculated by channel. Then, the feature map F is normalized. The mean μ of the foreground feature map is subtracted from the feature map F, and then divided by the standard deviation σ of the foreground feature map. Then, the feature map F is multiplied by the mean γ of the background feature map, and then the standard deviation β of the background feature map is added. Finally, the foreground feature map and the background feature map are spliced ​​using the foreground mask to obtain a feature map that integrates the background features. The calculation formula is as follows: Where F represents the feature map, γ represents the mean of the background feature map, β represents the standard deviation of the background feature map, μ represents the mean of the foreground feature map, and σ represents the standard deviation of the foreground feature map; Step B17: Construct the output convolution block used in step B11; the output convolution block consists of a deconvolution with a convolution kernel size of 4×4, a step size of 2, and a padding of 1, and a Tanh activation function in sequence; the output convolution block outputs a harmonized image of size H×W×3 4. The method for harmonizing a composite image based on a foreground reference image according to claim 3, wherein: The specific implementation steps of step B2 are as follows: Step B21: Construct a foreground reference image feature extraction and fusion module. The input of the foreground reference image feature extraction and fusion module is the normalized real label image I and the original synthetic image. and reference images Original composite image and reference images The size of the foreground reference image is H×W×3, and the harmonized image of the output size is H×W×3; the foreground reference image feature extraction and fusion module is composed of an encoder, two alignment modules, two residual blocks, a reference image feature fusion layer and a decoder in sequence; Step B22: Construct the encoder used in step B21, wherein the encoder includes 5 convolution blocks; and reference images Input into the convolution block of the encoder, and the synthetic image feature map is obtained by the tth convolution block and reference image feature map The size of each feature map is Each convolution block is composed of a convolution layer, an activation layer, and a regularization layer in sequence; the convolution layer uses a convolution kernel size of 4×4, a step size of 2, and a padding of 1. The activation layer uses the LeakyRelu activation function, and the regularization layer uses the instance regularization algorithm. The output of the encoder is five sets of feature maps of the synthetic image obtained by the five convolution blocks. arrive and five sets of feature maps of the reference image arrive Step B23: construct two alignment modules used in step B21; the inputs of the two alignment modules are the five sets of feature maps of the composite image output by step B22. arrive and five sets of feature maps of the reference image arrive Each alignment module consists of five convolutions with a kernel size of 1x1. The input of each convolution is a corresponding set of feature maps from the five sets of input feature maps. The five sets of output feature maps of the convolution layer are adjusted to 32 and 32 in width and height by difference, and spliced ​​along the channel dimension. The final output is a synthetic image feature map of size 32×32×1024. and reference image feature map Step B24, construct the two residual blocks used in step B21; the two residual blocks are composed of a convolution layer, an activation layer, a regularization layer, a convolution layer and a regularization layer in sequence, the convolution layer uses a convolution kernel size of 3×3, a step size of 1, and a padding of 1, the activation layer uses a ReLU activation function, and the regularization layer uses an instance regularization algorithm; the inputs of the two residual blocks are the synthetic image feature maps and reference image feature map Output size of synthetic image feature map is 32×32×1024 and reference image feature map Step B25: construct a reference image feature fusion layer; The input of the reference image feature fusion layer is the synthetic image feature map obtained in step B23 and reference image feature map Calculate the mean μ and standard deviation σ of the synthetic image feature map by channel, and calculate the mean γ and standard deviation β of the reference image feature map by channel, and then regularize the synthetic image feature map The synthetic image feature map Subtract the mean μ of the synthetic image feature map and divide it by the standard deviation σ of the synthetic image feature map, and then multiply it by the reference image feature map The mean γ of the reference image is added with the standard deviation β of the reference image feature map to obtain the feature map that combines the reference image features. The calculation formula is as follows: Then the feature map Input into the decoder, and the final output is a harmonized image with a size of H×W×3 Step B26: Construct the decoder used in step B21; the decoder includes three deconvolution blocks, each of which is composed of a deconvolution layer, a regularization layer, and an activation layer in sequence; the deconvolution layer uses a deconvolution kernel with a size of 4×4, a stride of 2, and a padding of 1; the regularization layer uses an instance regularization algorithm; and the activation layer uses a ReLU activation function; Step B27. Construct the convolution block in the encoder in step B22, where the convolution block in the encoder is composed of a convolution layer, a regularization layer, and an activation layer in sequence; the convolution layer uses a convolution kernel size of 4×4, a step size of 2, and a padding of 1; the regularization layer uses an instance regularization algorithm, and the activation layer uses a LeakyRelu activation function; the first convolution block in the encoder does not contain a regularization layer.

5. The method for harmonizing a composite image based on a foreground reference image according to claim 4, characterized in that: The specific implementation steps of step B3 are as follows: Step B31: construct a dual-branch feature fusion fine-tuning module; The input of the dual-branch feature fusion fine-tuning module is the output obtained in step B1 and feature maps The output from step B2 And the foreground mask M; the feature map The deconvolution kernel is 3×3, and the feature map of size 256×256×16 is obtained. Will M and After splicing along the channel dimension and feeding into the fine-tuning module, the final harmonized image I′ is obtained; Step B32: Construct the fine-tuning module used in step B31; the fine-tuning module consists of three convolution blocks, and the convolution block consists of a convolution layer, a regularization layer, and an activation layer in sequence; the convolution layer uses a convolution kernel size of 3×3, a step size of 1, and a padding of 1; the regularization layer uses a batch normalization algorithm, and the activation layer uses an ELU activation function.

6. The method for harmonizing a composite image based on a foreground reference image according to claim 1, characterized in that: The step C specifically comprises the following steps: Step C1: Construct the network target loss function, which is expressed as follows: l=||I′-I||1 Among them, l is the L1 loss, I′ is the constructed synthetic image and the final output of the harmonization network, I is the real label image, and ‖.‖1 is the absolute value operation.

7. The method for harmonizing a composite image based on a foreground reference image according to claim 1, characterized in that: The step D specifically comprises the following steps: Step D1: randomly divide the corresponding true label images, original synthetic images, reference images, and foreground masks into multiple batches, each batch containing N groups of images; Step D2: input the image pair into the image harmonization network described in step B to obtain the final harmonized image; Step D3: Based on the target loss function of the image harmonization network, the back propagation method is used to calculate the gradient of each parameter in the image enhancement network, and the Adam optimization method is used to update the parameters, where the learning rate is changed from 2×10 -4 Slowly reduce to 2×10 -5 ; Step D4: Repeat the image harmonization network training steps from step D1 to step D3 in batches. The entire process is iterated a set number of times, the network parameters are saved, and the image harmonization network training process is completed.

8. The method for harmonizing a composite image based on a foreground reference image according to claim 1, characterized in that: The specific implementation method of step E is as follows: for a new synthetic image and foreground mask, the foreground target and background target are extracted using the foreground mask and the yolov5 target detection pre-trained model, and then the cosine similarity is calculated to obtain a reference image. The synthetic image, the reference image and the foreground mask are input into the image harmonization network described in step B to obtain a harmonized image.

9. A synthetic image and harmonization system based on a foreground reference image, characterized in that: The method comprises a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method according to any one of claims 1 to 8 can be implemented.

Images