Synthetic image harmonization model training method, harmonization method and device

Abstract

The present application provides a kind of synthetic image and harmonization model training method, harmonization method and device, the synthetic image and harmonization model training method adopts mask in training process to distinguish synthetic image foreground and background, respectively calculates the global illumination estimation of synthetic image foreground and background, and the global illumination estimation of synthetic image foreground, the global illumination estimation of synthetic image background and the color style feature of synthetic image are fused and are obtained by decoder upsampling coordinated image.Meanwhile, the present application coordinates the illumination and color style feature of synthetic image from global angle, and is trained based on generative adversarial network, utilizes the joint loss function of constructing relative loss and adversarial loss to carry out parameter iteration, improves the harmonization effect of synthetic image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a synthetic image harmonization model training method and a harmonization method and device. BACKGROUND

[0002] Image synthesis is a technique of extracting the foreground of an image and pasting it to the background of another image. Due to the difference in light and scene between the two images, there is a difference between the foreground and the background of the synthesized image, which leads to the phenomenon that the visual effect of the foreground and the background of the synthesized image is not harmonious, and the specified image synthesis quality requirement cannot be met.

[0003] The traditional method is based on manual extraction of image features, and adjusts the foreground and background of the synthesized image on this basis to realize the harmonization of the synthesized image. However, this way has a small number of features collected and a shallow semantic level, and it is difficult to achieve satisfactory results after harmonization processing. In addition, there is also a scheme in the prior art that performs global stylization processing on the foreground and background of the synthesized image through a deep learning method, but this scheme does not consider the lighting features of the foreground and the background, so the coordinated effect is not ideal.

[0004] Therefore, there is an urgent need for a synthetic image harmonization method to coordinate the foreground color style features of the synthesized image with the lighting and color style features of the background of the synthesized image from a global perspective, and to improve the realism of the synthesized image. SUMMARY

[0005] In view of this, the embodiments of the present application provide a synthetic image harmonization model training method and a harmonization method and device to solve the problem that the foreground and the background of the synthesized image are inconsistent in terms of lighting and color style features.

[0006] One aspect of the present application provides a synthetic image harmonization model training method, which comprises the following steps:

[0007] Obtain a training sample set, the training sample set comprising a plurality of samples, each sample comprising a synthetic image and a real image, the synthetic image distinguishing foreground and background by using a mask, the foreground and the background of the synthetic image not being harmonized, and the real image being obtained by harmonizing the foreground of the synthetic image;

[0008] The generator and the discriminator constitute a generative adversarial network, the generator comprises an encoder, an illumination fusion module and a decoder connected in sequence; the encoder comprises a style feature encoder and an illumination estimation module connected in parallel; in the generator, the synthesized image in a single sample is input into the style feature encoder to collect color style features of the synthesized image; the mask is used to distinguish the foreground and the background of the synthesized image and input into the illumination estimation module to calculate a first global illumination estimation of the foreground of the synthesized image and a second global illumination estimation of the background of the synthesized image respectively; the color style features, the first global illumination estimation and the second global illumination estimation are input into the illumination fusion module to calculate a fusion feature, and the color style features and the fusion feature are superimposed in channels and then input into the decoder to obtain a coordinated image; wherein the first global illumination estimation is calculated according to illumination features of the corresponding synthesized image foreground in three primary color channels, and the second global illumination estimation is calculated according to illumination features of the corresponding synthesized image background in three primary color channels; the discriminator comprises consecutive multiple convolution layers, and an activation function layer and a batch normalization layer are connected after each convolution layer.

[0009] The generator is trained by using the training sample set, a relative loss is calculated according to the real image and the corresponding coordinated image in each sample, the real image and the corresponding coordinated image in each sample are input into the discriminator to calculate a discrimination loss, a joint loss is calculated according to the relative loss and the discrimination loss, and the generator is iteratively parameterized according to the joint loss, and the trained generator is used as a synthesized image harmonization model.

[0010] In some embodiments, the style feature encoder comprises three layers of convolution layers, four layers of residual modules and two layers of convolution layers connected in sequence.

[0011] In some embodiments, the illumination estimation module comprises two first branches and second branches with the same structure, the first branch and the second branch each comprise five layers of convolution layers, the first branch is used to extract the first global illumination estimation, and the second branch is used to extract the second global illumination estimation.

[0012] In some embodiments, based on the mask, the foreground and the background of the synthesized image are distinguished and input into the illumination estimation module, and the first global illumination estimation of the foreground of the synthesized image and the second global illumination estimation of the background of the synthesized image are calculated respectively, comprising:

[0013] The mask is used to distinguish the foreground and the background of the synthesized image.

[0014] The foreground of the synthetic image is input into the first branch, the foreground is extracted into a plurality of foreground patches according to a set size, a four-channel feature map is extracted for each foreground patch, the first three channels extract three groups of illumination features of each foreground patch in three primary color channels respectively, and normalized processing is performed to obtain the illumination color of the corresponding foreground patch, and the fourth channel is used to calculate the contribution weight of the corresponding foreground patch; the illumination color and the contribution weight corresponding to each foreground patch are multiplied and accumulated to obtain the first global illumination estimation;

[0015] The background of the synthetic image is input into the second branch, the background is extracted into a plurality of background patches according to a set size, a four-channel feature map is extracted for each background patch, the first three channels extract three groups of illumination features of each background patch in three primary color channels respectively, and normalized processing is performed to obtain the illumination color of the corresponding background patch, and the fourth channel is used to calculate the contribution weight of the corresponding background patch; the illumination color and the contribution weight corresponding to each background patch are multiplied and accumulated to obtain the second global illumination estimation.

[0016] In some embodiments, the illumination fusion module is composed of a three-layer multi-layer perception machine; and the decoder is composed of three up-sampling layers and three convolutional layers alternately.

[0017] In some embodiments, the color style feature, the first global illumination estimation and the second global illumination estimation are input into the illumination fusion module to calculate a fusion feature, and the calculation formula is as follows:

[0018]

[0019] wherein, represents the fusion feature, represents the foreground of the synthetic image, represents the background of the synthetic image, represents the first global illumination estimation, represents the second global illumination estimation, e n represents the color style feature.

[0020] In some embodiments, a relative loss is calculated according to a real image and a corresponding coordinated image in each sample, and the calculation formula is as follows:

[0021]

[0022] wherein, represents the relative loss, represents the coordinated image, I represents the real image, and ||·||1 represents the L1 norm;

[0023] Input the real image and the corresponding harmonious image in each sample into the discriminator and calculate the discriminant loss, the calculation formula is:

[0024]

[0025] Wherein, The discriminant loss is represented by D(·), and the processing flow of the discriminator is represented by D(·), The harmonious image calculated by the generator based on the synthesized image And the mask M, The average value is represented by I, and the real image is represented by I;

[0026] Calculate the joint loss according to the relative loss and the discriminant loss, the calculation formula is:

[0027]

[0028] Wherein, The joint loss is represented by L, The relative loss is represented by L, The discriminant loss is represented by D, and the hyperparameter is represented by lambda.

[0029] In another aspect, the present application also provides a synthesized image harmonization method, comprising:

[0030] Obtain the synthesized image to be harmonized, and the synthesized image distinguishes the foreground and the background by using the mask;

[0031] Input the synthesized image into the synthesized image harmonization model in the synthesized image harmonization model training method described above, and obtain the harmonious image of the synthesized image.

[0032] In another aspect, the present application also provides an electronic device, comprising a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device realizes the steps of the above method.

[0033] In another aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by the processor to realize the steps of the above method.

[0034] The beneficial effects of the present application are at least:

[0035] The synthetic image harmonization model training method, harmonization method and device, the color style feature and global illumination estimation of the synthetic image are respectively acquired by the parallel style feature encoder and the illumination estimation module of the encoder, the color style feature and global illumination estimation of the foreground and background of the synthetic image are fused, the synthetic image is harmonized from the color style and global illumination, and the harmonization degree of coordinating the foreground and the background of the image is improved.

[0036] Further, the foreground of the synthetic image is extracted as a plurality of foreground patches, the background of the synthetic image is extracted as a plurality of background patches, the global illumination estimation is formed on the basis of the local illumination features of the foreground and the background, the more detailed illumination difference between the foreground and the background can be mined, the illumination of the foreground and the background of the synthetic image is coordinated from a global perspective, and the harmonization degree of the coordinated image is improved.

[0037] Further, in the synthetic image harmonization model training, the generator and the discriminator are alternately trained to continuously carry out confrontation, and the joint loss constructed by introducing the relative loss and the adversarial loss is used for parameter adjustment of the generator, so that the generator has better harmonization capability, and the generated coordinated image is more realistic.

[0038] Additional advantages, objects, and features of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0039] It will be understood by those skilled in the art that the objects and advantages of the present application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. The components in the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the application. For purposes of clarity and a consistent approach, portions of the drawings may have been exaggerated from the remainder, and are intended as an aid for understanding the embodiments.

[0041] Figure 1 The structure schematic diagram of the generator in the synthetic image harmonization model training method of an embodiment of the application.

[0042] Figure 2A comparison diagram of the harmonious image processed by the synthesis image and harmonization model, DIH, S2AM and DoveNet respectively and the real image and the synthesis image. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments and drawings. Herein, the illustrative embodiments of the present application and their descriptions are used to explain the present application but not as a limitation of the present application.

[0044] It should be noted that, in order not to obscure the present application due to unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0045] It should be emphasized that the term "comprises / comprising" as used herein is used to indicate the presence of a feature, element, step or component but does not preclude the presence or addition of one or more other features, elements, steps or components.

[0046] Image synthesis is one of the most common but critical operations in image editing, which aims to extract the foreground of an image and paste it onto the background of another image. However, usually due to the difference in light and scene between the two images, the appearance of the two images is different, resulting in the phenomenon of image disharmony. To solve this problem, professional personnel usually need to use image editing software to adjust the appearance of the foreground to improve the realism of the synthesized image.

[0047] In this image editing task, to generate a synthesized image with a realistic visual effect, professional designers usually need to carefully observe the differences between images with the help of image editing tools (such as Photoshop), adjust the color and lighting features of the pasted foreground, so that the foreground and background are more appropriate. This process requires a certain image editing skill of the person, and it takes a lot of time to process a single picture.

[0048] Traditional methods transfer the artificially designed statistical knowledge from existing images to the adjusted foreground image, including color features, texture features, or using the Poisson fusion method. Although some results have been achieved, these methods can only be effective on simple examples, usually under the condition that the foreground and background of these examples are already relatively harmonious to some extent. Other existing deep learning-based image synthesis techniques cannot coordinate the lighting and color style features of the synthesized image foreground and background from a global perspective, which reduces the quality of the synthesized image. Therefore, the present application provides a synthesis image harmonization model training method, harmonization method and device for improving the realism of the synthesized image.

[0049] The human visual and cognitive system recovers the original color of an object under various lighting conditions. According to the color characteristics of a known object, the system can infer the lighting conditions of an unfamiliar scene. Therefore, humans are quite sensitive to the mismatch of ambient light between foreground and background, and can easily distinguish synthetic images with different foreground and background lighting conditions. One aspect of the present application provides a synthetic image harmonization model training method, comprising steps S101-S103:

[0050] S101: Obtain a training sample set, the training sample set comprising a plurality of samples, each sample comprising a synthetic image and a real image, the synthetic image distinguishing foreground and background using a mask, the foreground and background of the synthetic image not being harmonized, and the real image being obtained by harmonizing the foreground of the synthetic image.

[0051] S102: Obtain a generator and a discriminator to form a generative adversarial network, as shown in Figure 1 The generator comprises an encoder, a light fusion module and a decoder connected in sequence; the encoder comprises a style feature encoder and a light estimation module connected in parallel; in the generator, the synthetic image in a single sample is input into the style feature encoder to collect the color style features of the synthetic image; the foreground and background of the synthetic image are distinguished based on the mask and input into the light estimation module to calculate a first global light estimation of the foreground of the synthetic image and a second global light estimation of the background of the synthetic image; the color style features, the first global light estimation and the second global light estimation are input into the light fusion module to obtain fusion features, and the color style features and the fusion features are superimposed in channels and then input into the decoder to obtain a harmonized image; wherein the first global light estimation is calculated according to the light features of the corresponding synthetic image foreground in three primary color channels, and the second global light estimation is calculated according to the light features of the corresponding synthetic image background in three primary color channels; the discriminator comprises a plurality of convolution layers connected in series, and an activation function layer and a batch normalization layer are connected after each convolution layer.

[0052] S103: Train the generator using the training sample set, calculate the relative loss according to the real image and the corresponding harmonized image in each sample, input the real image and the corresponding harmonized image in each sample into the discriminator to calculate the discrimination loss, calculate the joint loss according to the relative loss and the discrimination loss, and iteratively update the parameters of the generator according to the joint loss, and use the trained generator as a synthetic image harmonization model.

[0053] In step S101, the synthetic image is a combined image formed by obtaining the foreground of an image and pasting it onto the background of another image. Due to the differences in the fibers and scenes of the two images, the foreground and background parts of the synthetic image are significantly different and inharmonious, and are easily distinguished. The real image is obtained after the synthetic image is harmonized. The real image in the training sample set is obtained by manually adjusting the illumination and color of the foreground of the corresponding synthetic image using a traditional method. Therefore, the foreground parts of the synthetic image and the real image are different, and the background parts are consistent. The mask used in this embodiment is a black and white bitmap used to distinguish the foreground and background. The black part is the part to be displayed, and the white part is the part to be transparent. After the mask operation, the background color of a bitmap can be removed and only the content part is retained.

[0054] In step S102, in this embodiment, a generative adversarial network is constructed for learning. The generator is used for harmonization processing of the synthetic image, and the discriminator is used for identifying the authenticity of the coordinated image obtained by the generator and the real image. The generator and the discriminator are alternately trained through adversarial learning.

[0055] For the generator, the encoder is used for feature extraction. The encoder is composed of two parts. The style feature encoder is used to extract color style features. The illumination estimation module sets two branches to extract the global illumination estimation of the foreground and background of the synthetic image. The color style features, the global illumination estimation of the foreground and the background are fused through the illumination fusion module to obtain the fused features. After the fused features and the original color style features are superimposed in the channel, the decoder is used for image reconstruction. The foreground part is harmonized to obtain the coordinated image.

[0056] The style feature encoder includes three layers of convolution layers, four layers of residual modules, and two layers of convolution layers connected in sequence. Specifically, the style feature encoder module performs down-sampling on the input synthetic image through the convolution layer with a 3x3 convolution kernel, and adds a residual block to refine the rough color style features.

[0057] In this embodiment, the illumination estimation module includes two first branches and second branches with the same structure. The first branch and the second branch are both composed of five convolution layers. The first branch is used to extract the first global illumination estimation of the foreground of the synthetic image, and the second branch is used to extract the second global illumination estimation of the background of the synthetic image.

[0058] In some embodiments, the foreground and background of the synthetic image are distinguished based on the mask and input into the illumination estimation module. The first global illumination estimation of the foreground of the synthetic image and the second global illumination estimation of the background of the synthetic image are calculated, including steps S201-S203:

[0059] Step S201: distinguish the foreground and background of the composite image using a mask.

[0060] Step S202: input the foreground of the composite image into the first branch, extract the foreground into a plurality of foreground patches according to a set size, extract a four-channel feature map for each foreground patch, the first three channels respectively extract three groups of illumination features of each foreground patch in three primary color channels, and normalize to obtain the illumination color of the corresponding foreground patch, and the fourth channel is used to calculate the contribution weight of the corresponding foreground patch; multiply the illumination color corresponding to each foreground patch and the contribution weight to obtain the first global illumination estimation.

[0061] Step S203: input the background of the composite image into the second branch, extract the background into a plurality of background patches according to a set size, extract a four-channel feature map for each background patch, the first three channels respectively extract three groups of illumination features of each background patch in three primary color channels, and normalize to obtain the illumination color of the corresponding background patch, and the fourth channel is used to calculate the contribution weight of the corresponding background patch; multiply the illumination color corresponding to each background patch and the contribution weight to obtain the second global illumination estimation.

[0062] In steps S201-S203, after distinguishing the foreground and background of the composite image using a mask, a set size of convolution kernel is used to slide on the foreground and background to extract a plurality of background patches, and four-channel feature maps are obtained by feature extraction on each extracted patch, the first three channels are three groups of illumination features in three primary color channels, respectively corresponding to hue, chroma and brightness features. The fourth channel is used to estimate the contribution of the corresponding patch to the global illumination estimation for weighting.

[0063] Specifically, the illumination estimation module uses a 16x16 convolution kernel to extract a plurality of foreground patches and background patches on the foreground and background of the composite image respectively, and normalizes three groups of illumination features to obtain the illumination color of the corresponding patch. Specifically, the calculation formula of the illumination color of each patch is:

[0064]

[0065] wherein ψ γ (p represents the illumination color of patch p, normalize(·) represents normalization processing, φ(p) represents the illumination feature map extracted by a group of convolution layers in the illumination estimation module for patch p, h represents the height of the illumination feature map, w represents the width of the illumination feature map, and c represents the channel of the illumination feature map.

[0066] The fourth channel is used to calculate the contribution weight of each patch, and the contribution weight calculation formula is:

[0067] ψ ω (p=normalize(∑h,w φ(p) (h,w,4) );

[0068] wherein ψ ω (p) represents the contribution weight of the patch, normalize(·) represents the normalization processing, φ(p) represents the illumination feature map extracted by a group of convolution layers in the illumination estimation module for the patch p, h represents the height of the illumination feature map, and w represents the width of the illumination feature map.

[0069] The global illumination estimation is obtained by multiplying and accumulating the illumination color corresponding to each patch and the contribution weight. The global illumination estimation is calculated for the foreground and the background of the synthesized image respectively. The calculation formula of the global illumination estimation is as follows:

[0070]

[0071] wherein F LEM represents the global illumination estimation, ψ ω (p i ) represents the contribution weight of the patch, ψ γ (p i ) represents the illumination color of the patch, represents the synthesized image, and p i refers to the ith patch.

[0072] In the embodiment, the illumination fusion module is composed of three multi-layer perceptrons, which are used to fuse the color style feature, the first global illumination estimation and the second global illumination estimation to combine the features of color style and illumination. Specifically, the color style feature, the first global illumination estimation and the second global illumination estimation are input into the illumination fusion module to calculate the fusion feature, and the calculation formula is as follows:

[0073]

[0074] wherein F represents the fusion feature, represents the foreground of the synthesized image, represents the background of the synthesized image, represents the first global illumination estimation, represents the second global illumination estimation, and e n represents the color style feature.

[0075] The illumination fusion module takes the ratio of the first global illumination estimation to the second global illumination estimation as a coefficient, and multiplies the coefficient with the color style feature to adjust the color style feature of the synthesized image to adapt to the illumination of the synthesized image, so as to realize the feature fusion.

[0076] Further, the fusion feature also needs to be superimposed with the color style feature in the channel to retain the information in the original color style, and the image is processed by the decoder for coordination.

[0077] Specifically, the decoder is composed of three up-sampling layers and three convolutional layers alternately. The decoder part utilizes bilinear up-sampling and convolutional layers to up-sample the feature map. The up-sampling layer adopts a bilinear interpolation algorithm with an expansion factor of 2, and the convolutional layer adopts a 1x1 convolution kernel with a convolution kernel step of 1. Through up-sampling, the coordination image can have a higher resolution, increasing the harmonization degree of the coordination image.

[0078] In step S103, the relative loss is calculated according to the real image in each sample and the corresponding coordination image, and the calculation formula is:

[0079]

[0080] wherein, represents the relative loss, represents the coordination image, I represents the real image, and ||·||1 represents the L1 norm.

[0081] Further, the real image in each sample and the corresponding coordination image are input into the discriminator to calculate the discrimination loss, and the calculation formula is:

[0082]

[0083] wherein, represents the discrimination loss, D(·) represents the processing flow of the discriminator, represents the coordination image calculated by the generator based on the synthesized image and the mask M, represents the average value, and I represents the real image.

[0084] Further, the joint loss is calculated according to the relative loss and the discrimination loss, and the calculation formula is:

[0085]

[0086] wherein, represents the joint loss, represents the relative loss, represents the discrimination loss, and λ is a hyperparameter.

[0087] Wherein, the joint loss is calculated to perform parameter iteration on the generator, so that the generated coordination image has higher authenticity.

[0088] Further, the original loss function of the generative adversarial network is used to perform parameter iteration on the discriminator, and the expression of the original loss function is:

[0089]

[0090] wherein, LD represents the original loss function of the discriminator, D(·) represents the processing flow of the discriminator, represents the average value, I represents the real image, represents the harmonious image, and M represents the mask.

[0091] On the other hand, the present application also provides a synthetic image harmonization method, comprising steps S301-S302:

[0092] S301: obtaining a synthetic image to be harmonized, and the synthetic image distinguishes foreground and background by using a mask.

[0093] S302: inputting the synthetic image into the synthetic image harmonization model in the synthetic image harmonization model training method in steps S101-S103, and obtaining a harmonious image of the synthetic image.

[0094] On the other hand, the present application also provides an electronic device comprising a processor and a memory, wherein the memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the above method.

[0095] On the other hand, the present application also provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented. When the computer program is executed by the processor, the steps of the above edge computing server deployment method are implemented. The computer readable storage medium can be a tangible storage medium, such as random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0096] The present application will be described in detail below in conjunction with specific embodiments:

[0097] The present application provides a synthetic image harmonization model capable of perceiving illumination differences, and a training method thereof, comprising the following steps:

[0098] 1) Construct a generator and a discriminator based on a generative adversarial network. The generator includes an encoder, an illumination fusion module, and a decoder connected in sequence. The encoder includes a style feature encoder and an illumination estimation module connected in parallel. In the generator, the synthesized image from a single sample is input into the style feature encoder to collect the color style features of the synthesized image. The foreground and background of the synthesized image are distinguished based on a mask and input into the illumination estimation module to calculate the first global illumination estimate of the foreground and the second global illumination estimate of the background of the synthesized image, respectively. The color style features, the first global illumination estimate, and the second global illumination estimate are input into the illumination fusion module to calculate the fusion features. The color style features and the fusion features are channel-superimposed and then input into the decoder to obtain a coordinated image.

[0099] The training data is a publicly available image dataset containing pairs of real and synthetic images.

[0100] 2) In the generator, the synthesized image to be adjusted is used as the original input image and input into the style feature encoder and the illumination estimation module respectively. The output is the style features of the synthesized image, as well as the global illumination estimation of the foreground and background.

[0101] Specifically, the foreground and background of the real image are labeled as I. f and I b And set the mask that distinguishes the foreground and background to M. Composite image. The foreground and background are respectively used and This indicates that, since the real image and the synthetic image differ only in the foreground area, The goal of this embodiment is to synthesize images. and As input, a reconciled image that closely approximates the real image is obtained. If we denote the coordination model as Φ, then the coordination task can be expressed as:

[0102] The Illumination Estimation Module (LEM) learns the illumination features of the background and foreground. The LEM has two branches, which extract the local illumination features of the foreground and background respectively and generate a global illumination estimate.

[0103] Specifically, LEM first uses a corresponding mask M to synthesize the foreground of the image. and background The images are separated, and then convolutional layers are used to extract local lighting estimates from the image patches. The local lighting estimates are weighted by their contribution to the aggregated global lighting estimate to obtain the foreground. The corresponding first global illumination estimation and background The corresponding second global illumination estimate.

[0104] Specifically, the light estimation module processes the foreground or background , first slides to extract a patch of size 16x16, extracts local light features from the patch using a convolutional layer, since each patch has different effects on the image, we perform contribution evaluation on each patch and use it as a weight to aggregate the local light features corresponding to each patch to obtain the global light estimation.

[0105] In order to obtain fine light, LEM obtains a four-channel feature map through a convolutional layer, of which the first three are local light features based on a three-channel feature map of three primary colors, and the fourth is the contribution weight of the evaluated patch.

[0106] The global light estimation is calculated for the foreground or background of the synthetic image respectively.

[0107] Specifically, assume p is a patch of the foreground or background of the synthetic image, and the first three channels are specified to represent the illumination color ψ γ (p) estimated from each corresponding patch. The illumination color ψ γ (p) of each patch is calculated as follows:

[0108]

[0109] Where ψ γ (p) represents the illumination color of patch p, normalize(·) represents normalization processing, φ(p) represents the light feature map extracted by a group of convolutional layers in the light estimation module for patch p, h represents the height of the light feature map, w represents the width of the light feature map, and c represents the channel of the light feature map.

[0110] The contribution weight ψ ω (p) of the patch is calculated from the last channel of the feature map as follows:

[0111] ψ ω (p) = normalize(∑ h，w φ(p) (h，w，4) );

[0112] Where ψ ω (p) represents the contribution weight of the patch, normalize(·) represents normalization processing, φ(p) represents the light feature map extracted by a group of convolutional layers in the light estimation module for patch p, h represents the height of the light feature map, and w represents the width of the light feature map.

[0113] The global light estimation of the foreground or background extracted by LEM is defined as:

[0114]

[0115] where F LEM represents the global illumination estimation, ψ ω (p i ) represents the contribution weight of the patch, ψ γ (p i ) represents the lighting color of the patch, represents the synthesized image, p i refers to the i-th patch.

[0116] 3) The Light Fusion Module (LFM) connects the encoder and the decoder, fuses the style features and the illumination estimation, and uses the fused features to adjust the foreground lighting.

[0117] The global illumination estimation of the synthesized image foreground and background obtained by the illumination estimation module is input into the light fusion module together with the color style adjustment obtained by the style feature encoder, to obtain the fused features with lighting features, which can be expressed as follows:

[0118]

[0119] where, represents the fused features, represents the foreground of the synthesized image, represents the background of the synthesized image, represents the first global illumination estimation, represents the second global illumination estimation, e n represents the color style features.

[0120] The LFM learns the difference between the foreground and the background in the global illumination estimation, and adjusts the foreground lighting features to adapt to the background.

[0121] 4) The fused features are superimposed with the original color style features in the channel, and input into the decoder to generate a coordinated image with consistent lighting.

[0122] In this work, the relative L1 loss between the synthesized image and the coordinated image is calculated using the mask M. In addition, an adversarial loss is introduced into the architecture to improve the visual quality. Considering the proposed model Φ as a generator, an additional discriminator D is also used to identify the authenticity of the image. Specifically, the discriminator consists of seven convolutional layers, followed by a ReLU activation function layer and a batch normalization layer. The adversarial loss can be expressed as:

[0123]

[0124] wherein D(·) represents a processing procedure of the discriminator, represents the generated image based on the synthesized image and the mask M, represents calculating the average value, and I represents the real image.

[0125] Accordingly, the total loss function is represented as:

[0126]

[0127] wherein, represents the joint loss, represents the relative loss, represents the discriminant loss, and λ is a hyperparameter for balancing the joint loss and the relative loss.

[0128] wherein, is the relative loss, and the calculation formula is:

[0129]

[0130] wherein, represents the generated image, I represents the real image, and ||·||1 represents calculating the L1 norm; λ is a hyperparameter for balancing the two losses.

[0131] The synthesized image and the harmonization model in the embodiment are compared quantitatively and qualitatively with other state-of-the-art image harmonization algorithms, and the comparison objects include DoveNet, S 2 AM, DIH, BargainNet and RainNet.

[0132] Regarding the quantitative evaluation, the synthesized image and the harmonization model (Ours) in the embodiment and the existing DoveNet, S 2 AM and DIH are compared by using the dataset iHarmony4 and multiple sub-datasets thereof, and the parameters for comparison are mean square error (MSE) and peak signal-to-noise ratio (PSNR), and the comparison results are shown in Table 1.

[0133] Table 1

[0134]

[0135] It can be observed that the method in the embodiment not only significantly exceeds the traditional method on all sub-datasets, but also exceeds the deep learning-based method on the entire test set.

[0136] Further comparative experiments are carried out on the self-built dataset, and the same indicators are used to evaluate the synthesized image harmonization model (Ours) described in the embodiment, S 2 AM, BargainNet and RainNet. The experimental results are shown in Table 2:

[0137] Table 2

[0138]

[0139] It can be seen that the synthesized image harmonization model described in the embodiment is obviously superior to other methods under inconsistent lighting conditions. In addition, the image structural similarity (fSSIM) of the foreground region is also calculated as an additional indicator, and the experimental results are shown in Table 3:

[0140] Table 3

[0141]

[0142] It can be seen that the foreground image generated by the synthesized image harmonization model described in the embodiment is more similar to the real image.

[0143] Regarding the qualitative evaluation, as shown in Figure 2 , the real image (Ground Truth), the synthesized image (Input) and the coordinated image obtained by respectively processing DIH, S2AM, DoveNet and the synthesized image harmonization model (Ours) described in the embodiment are compared. It can be seen that the coordinated image obtained by the method of the embodiment is more similar to the real image in visual effect, especially when the lighting conditions of the foreground and background of the synthesized image are obviously different, as shown in the 4th row of Figure 2 , the coordinated image obtained by the synthesized image harmonization model (Ours) described in the embodiment is closer to the real image in visual effect.

[0144] In summary, the synthesized image harmonization model training method, harmonization method and device described in the embodiment, the color style features and global illumination estimation of the synthesized image are respectively extracted by the parallel style feature encoder and illumination estimation module of the encoder, and the synthesized image is harmonized from the aspects of color style and global illumination by fusing the color style features and global illumination estimation of the foreground and background of the synthesized image, thereby improving the harmonization degree of the foreground and background of the coordinated image.

[0145] Further, the foreground of the synthesized image is extracted as a plurality of foreground patches, the background of the synthesized image is extracted as a plurality of background patches, and on the basis of obtaining the local illumination features of the foreground and the background, the global illumination estimation is formed, which can mine more detailed illumination differences between the foreground and the background, so as to adapt the illumination of the foreground and the background of the synthesized image from a global perspective, thereby improving the harmonization degree of the coordinated image.

[0146] Further, in the synthetic image and harmonization model training, the generator and the discriminator are alternately trained to continuously carry out the confrontation, and a joint loss constructed by introducing a relative loss and an adversarial loss is used to adjust the parameters of the generator, so that the generator has better harmonization capability, and the generated coordinated image has better authenticity.

[0147] Corresponding to the above method, the application also provides a device / system, which comprises a computer device including a processor and a memory, and the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device / system implements the steps of the method as described above.

[0148] Those of ordinary skill in the art should understand that the exemplary components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software or a combination thereof. Whether the implementation is in hardware or software depends on the specific application and design constraints imposed on the overall system. Those of skill can use various approaches to implement the described functionality depending on the specific application and design constraints. Such implementation does not affect the scope of the application. When implemented in hardware, the hardware can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the application are the program or code segments to perform the necessary tasks. The program or code segments can be stored in a machine-readable medium, or transmitted by a data signal carried in a carrier wave over a transmission medium or communication link.

[0149] It should be noted that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of known methods are omitted. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0150] In the present application, the features described and / or exemplified for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or in combination with or instead of the features of other embodiments.

[0151] The above description is only preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A synthetic image and harmonization model training method, characterized by, The method comprises the following steps: An training sample set is obtained, the training sample set comprising a plurality of samples, each sample comprising a synthetic image and a real image, the synthetic image distinguishing foreground and background by a mask, the foreground and the background of the synthetic image not being harmonized, and the real image being obtained by harmonizing the foreground of the synthetic image; A generator and a discriminator constitute a generative adversarial network, the generator comprising an encoder, an illumination fusion module and a decoder connected in sequence; the encoder comprising a style feature encoder and an illumination estimation module connected in parallel; in the generator, the synthetic image in a single sample is input into the style feature encoder to collect color style features of the synthetic image; based on the mask distinguishing the foreground and the background of the synthetic image and input into the illumination estimation module, a first global illumination estimation of the foreground of the synthetic image and a second global illumination estimation of the background of the synthetic image are calculated respectively; the color style features, the first global illumination estimation and the second global illumination estimation are input into the illumination fusion module to obtain fusion features, and the color style features and the fusion features are superimposed in channels and then input into the decoder to obtain a harmonized image; wherein the first global illumination estimation is calculated according to illumination features of the corresponding synthetic image foreground in three primary color channels, and the second global illumination estimation is calculated according to illumination features of the corresponding synthetic image background in three primary color channels; the discriminator comprises a plurality of convolution layers connected in series, and an activation function layer and a batch normalization layer are connected after each convolution layer; The training sample set is used to train the generator, a relative loss is calculated according to the real image and the corresponding harmonized image in each sample, the real image and the corresponding harmonized image in each sample are input into the discriminator to calculate a discrimination loss, a joint loss is calculated according to the relative loss and the discrimination loss, and the generator is iterated in parameters according to the joint loss, and the trained generator is used as a synthetic image harmonization model. 2.The synthetic image and harmonization model training method of claim 1, wherein, The style feature encoder comprises three convolution layers, four residual modules and two convolution layers connected in sequence. 3.The synthetic image and harmonization model training method of claim 1, wherein, The illumination estimation module comprises two first branches and second branches with the same structure, each of the first branches and the second branches being composed of five convolution layers, the first branch being used to extract the first global illumination estimation, and the second branch being used to extract the second global illumination estimation. 4.The synthetic image and harmonization model training method of claim 3, wherein, Based on the mask distinguishing the foreground and the background of the synthetic image and input into the illumination estimation module, the first global illumination estimation of the foreground of the synthetic image and the second global illumination estimation of the background of the synthetic image are calculated respectively, comprising: The mask is used to distinguish the foreground and the background of the synthetic image. input the foreground of the synthetic image into the first branch, extract the foreground into a plurality of foreground patches according to a set size, extract a four-channel feature map for each foreground patch, the first three channels respectively extract three groups of illumination features of each foreground patch in three primary color channels, and normalize to obtain the illumination color of the corresponding foreground patch, and the fourth channel is used to calculate the contribution weight of the corresponding foreground patch; multiply the illumination color and the contribution weight corresponding to each foreground patch and accumulate to obtain the first global illumination estimation; input the background of the synthetic image into the second branch, extract the background into a plurality of background patches according to a set size, extract a four-channel feature map for each background patch, the first three channels respectively extract three groups of illumination features of each background patch in three primary color channels, and normalize to obtain the illumination color of the corresponding background patch, and the fourth channel is used to calculate the contribution weight of the corresponding background patch; multiply the illumination color and the contribution weight corresponding to each background patch and accumulate to obtain the second global illumination estimation. 5.The synthetic image and harmonization model training method of claim 1, wherein, The illumination fusion module is composed of a three-layer multilayer perception machine; the decoder is composed of three up-sampling layers and three convolutional layers alternately. 6.The synthetic image and harmonium model training method of claim 1, wherein, input the color style feature, the first global illumination estimation and the second global illumination estimation into the illumination fusion module to calculate a fusion feature, and the calculation formula is as follows: wherein, denotes the fusion feature, denotes the foreground of the composite image, denotes the background of the composite image, denotes the first global illumination estimate, denotes the second global illumination estimate, e n denotes the color style feature. 7.The synthetic image and harmonium model training method of claim 1, wherein, calculate a relative loss according to the real image and the corresponding coordinated image in each sample, and the calculation formula is as follows: wherein, denotes the relative loss, denotes the coordinated image, I denotes the real image, and ||·||1denotes the L1 norm; input the real image and the corresponding coordinated image in each sample into the discriminator and calculate a discrimination loss, and the calculation formula is as follows: wherein, denotes the discriminative loss, D(·) denotes the processing flow of the discriminator, denotes the generator calculates the harmonized image based on the synthesized image and the mask M, denotes the average, I denotes the real image; calculate a joint loss according to the relative loss and the discrimination loss, and the calculation formula is as follows: wherein, denotes the joint loss, denotes the relative loss, denotes the discriminative loss, and λ is a hyperparameter.

8. A synthetic image and harmonization method, characterized by, comprising: obtain a synthetic image to be harmonized, and the synthetic image distinguishes foreground and background by using a mask; input the synthetic image into the synthetic image harmonization model in the synthetic image harmonization model training method according to any one of claims 1 to 7, and obtain a coordinated image of the synthetic image.

9. A synthetic image and harmonization apparatus comprising a processor and a memory, wherein, The memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method according to any one of claims 1 to 8.

Images