Image harmonization processing method and device

Abstract

The present invention discloses a method and device for image harmonization processing, which can improve the expressiveness of a 3D lookup table while occupying fewer memory and computer resources. The method comprises: extracting features from a composite image to be processed to obtain composite features; utilizing the composite features to determine the weights of each subnetwork in a trained three-dimensional lookup table network, wherein the three-dimensional lookup table network is obtained by training an initial three-dimensional lookup table network using a training composite image as input and a training original image corresponding to the training composite image as output; determining a prediction network in the three-dimensional lookup table network based on the subnetworks and their weights; and harmonizing the composite image based on the prediction network to obtain a harmonized image of the composite image.

Description

Technical Field

[0001] The present invention relates to the technical field of image synthesis, and in particular to a method and device for image harmonization processing. Background Art

[0002] Image composition is a common operation in image processing. It combines the foreground (the object of interest) of one image with the background (the scenery behind the subject, representing the subject's spatial and temporal environment) of another image to create a composite image. However, because the foreground and background are captured under different conditions (such as lighting and weather), there is a significant mismatch in brightness and color, resulting in an unharmonious composite image.

[0003] Image harmonization aims to adjust the foreground of a composite image to make it appear harmonious with the background. While 3D lookup tables (LUTs) are a commonly used image enhancement method, current 3D LUTs are generally manually designed and applied to individual tasks. For batch image harmonization tasks, relying solely on a few manually designed 3D LUTs is far from sufficient, and requires expert design, which consumes significant human resources. Summary of the Invention

[0004] The present invention provides a method and device for image harmonization processing, which uses a trained 3D lookup table network to harmonize a synthetic image, can improve the expression ability of the 3D lookup table, and occupy less memory resources and computer resources.

[0005] In a first aspect, an embodiment of the present invention provides a method for image harmonization processing, comprising:

[0006] Perform feature extraction on the synthetic image to be processed to obtain synthetic features;

[0007] Determining weights of each sub-network in a trained three-dimensional lookup table network using the synthesized features, wherein the three-dimensional lookup table network is obtained by training an initial three-dimensional lookup table network using a training synthesized image as input and an original training image corresponding to the training synthesized image as output;

[0008] Determining a prediction network in the three-dimensional lookup table network according to the sub-networks and the weights of the sub-networks;

[0009] The synthetic image is harmonized according to the prediction network to obtain a harmonic graph of the synthetic image.

[0010] The embodiment of the present invention utilizes a combination of deep learning and three-dimensional lookup tables to train a dynamic three-dimensional lookup table network. The synthetic features extracted from the synthetic image are used to determine the weights of each sub-network in the three-dimensional lookup table network, that is, the weights of each three-dimensional lookup table. The weights and corresponding three-dimensional lookup tables are then combined into a new prediction network (i.e., a new three-dimensional lookup table). The new three-dimensional lookup table is then used to perform harmonization processing on the synthetic image, thereby improving the expressive power of the 3D lookup table and occupying fewer memory and computer resources.

[0011] In some embodiments, extracting features from the composite image to be processed to obtain composite features includes:

[0012] performing binarization processing on the synthesized image to obtain a mask image of the foreground;

[0013] Feature extraction is performed on the composite image and the mask image to obtain the composite feature.

[0014] In some embodiments, performing harmonization processing on the composite image according to the prediction network to obtain a harmonization graph of the composite image includes:

[0015] performing harmonization processing on the synthetic image according to the prediction network to obtain an initial harmonized image;

[0016] A harmony map of the composite image is generated according to the foreground in the initial harmony map and the background of the composite image.

[0017] In some embodiments, performing harmonization on the synthetic image according to the prediction network to obtain an initial harmonized image includes:

[0018] Determining, from the prediction network, predicted colors corresponding to each of the three primary colors in the composite image;

[0019] Each of the three primary colors in the composite image is processed into a corresponding predicted color to obtain the initial harmony image.

[0020] In some embodiments, determining, from the prediction network, predicted colors corresponding to the three primary colors in the composite image includes:

[0021] If the prediction network does not include the three primary colors in the synthesized image, predicted colors corresponding to the three primary colors are determined from the prediction network according to a trilinear interpolation method.

[0022] In some embodiments, generating a harmony map of the composite image based on the foreground in the initial harmony map and the background of the composite image includes:

[0023] The foreground of the initial harmony image is pasted into the background of the composite image to generate a harmony image of the composite image.

[0024] In some embodiments, before extracting features from the composite image to be processed, the method further includes:

[0025] If the resolution of the composite image is higher than a threshold, downsampling the composite image;

[0026] The down-sampled image is determined as the composite image to be processed.

[0027] In some embodiments, the trained three-dimensional lookup table network is determined as follows:

[0028] Determining a loss value based on a training original image corresponding to the training composite image and a training harmonic image obtained by harmonizing the training composite image through an initial three-dimensional lookup table network;

[0029] Determining a loss weight of a loss value corresponding to the training synthetic image according to a comparison result of a pixel value of a foreground in the training synthetic image and a threshold, wherein the loss weight is inversely proportional to the size of the pixel value;

[0030] Determine the loss function based on the loss value and loss weight corresponding to each training synthetic graph;

[0031] If the function value of the loss function is less than a preset value, the initial three-dimensional lookup table network is determined to be the trained three-dimensional lookup table network.

[0032] In a second aspect, an embodiment of the present invention provides an image harmonization device, including a processor and a memory, wherein the memory is configured to store a program executable by the processor, and the processor is configured to read the program in the memory and perform the following steps:

[0033] Perform feature extraction on the synthetic image to be processed to obtain synthetic features;

[0034] Determining weights of each sub-network in a trained three-dimensional lookup table network using the synthesized features, wherein the three-dimensional lookup table network is obtained by training an initial three-dimensional lookup table network using a training synthesized image as input and an original training image corresponding to the training synthesized image as output;

[0035] Determining a prediction network in the three-dimensional lookup table network according to the sub-networks and the weights of the sub-networks;

[0036] The synthetic image is harmonized according to the prediction network to obtain a harmonic graph of the synthetic image.

[0037] In some embodiments, the processor is specifically configured to perform:

[0038] performing binarization processing on the synthesized image to obtain a mask image of the foreground;

[0039] Feature extraction is performed on the composite image and the mask image to obtain the composite feature.

[0040] In some embodiments, the processor is specifically configured to perform:

[0041] performing harmonization processing on the synthetic image according to the prediction network to obtain an initial harmonized image;

[0042] A harmony map of the composite image is generated according to the foreground in the initial harmony map and the background of the composite image.

[0043] In some embodiments, the processor is specifically configured to perform:

[0044] Determining, from the prediction network, predicted colors corresponding to each of the three primary colors in the composite image;

[0045] Each of the three primary colors in the composite image is processed into a corresponding predicted color to obtain the initial harmony image.

[0046] In some embodiments, the processor is specifically configured to perform:

[0047] If the prediction network does not include the three primary colors in the synthesized image, predicted colors corresponding to the three primary colors are determined from the prediction network according to a trilinear interpolation method.

[0048] In some embodiments, the processor is specifically configured to perform:

[0049] The foreground of the initial harmony image is pasted into the background of the composite image to generate a harmony image of the composite image.

[0050] In some embodiments, before extracting features from the composite image to be processed, the processor is further configured to execute:

[0051] If the resolution of the composite image is higher than a threshold, downsampling the composite image;

[0052] The down-sampled image is determined as the composite image to be processed.

[0053] In some embodiments, the processor is specifically configured to determine the trained three-dimensional lookup table network in the following manner:

[0054] Determining a loss value based on a training original image corresponding to the training composite image and a training harmonic image obtained by harmonizing the training composite image through an initial three-dimensional lookup table network;

[0055] Determining a loss weight of a loss value corresponding to the training synthetic image according to a comparison result of a pixel value of a foreground in the training synthetic image and a threshold, wherein the loss weight is inversely proportional to the size of the pixel value;

[0056] Determine the loss function based on the loss value and loss weight corresponding to each training synthetic graph;

[0057] If the function value of the loss function is less than a preset value, the initial three-dimensional lookup table network is determined to be the trained three-dimensional lookup table network.

[0058] In a third aspect, an embodiment of the present invention further provides a device for image harmonization processing, the device comprising:

[0059] A feature extraction unit, configured to extract features from the composite image to be processed to obtain composite features;

[0060] a weight determination unit, configured to determine, using the synthesized features, the weights of each sub-network in a trained three-dimensional lookup table network, wherein the three-dimensional lookup table network is obtained by training an initial three-dimensional lookup table network using a training synthesized image as input and an original training image corresponding to the training synthesized image as output;

[0061] a harmonic processing unit, configured to determine a prediction network in the three-dimensional lookup table network according to the sub-networks and the weights of the sub-networks, and perform harmonic processing on the composite image according to the prediction network to obtain an initial harmonic image;

[0062] The synthesis generating unit is configured to generate a harmony map of a synthesized image according to the foreground in the initial harmony map and the background of the synthesized image.

[0063] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement the steps of the method described in the first aspect above.

[0064] These and other aspects of the present application will be more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 A schematic diagram of a 3D lookup table provided in an embodiment of the present invention;

[0067] Figure 2A A schematic diagram of a network architecture for a first image harmonization process provided by an embodiment of the present invention;

[0068] Figure 2B A schematic diagram of a network architecture for a first image harmonization process provided by an embodiment of the present invention;

[0069] Figure 2C A schematic diagram of a network architecture for a first image harmonization process provided by an embodiment of the present invention;

[0070] Figure 3 A flowchart of an image harmonization method according to an embodiment of the present invention;

[0071] Figure 4 A method for calculating a three-dimensional lookup table provided in an embodiment of the present invention;

[0072] Figure 5 A detailed implementation flow chart of an image harmonization process provided by an embodiment of the present invention;

[0073] Figure 6 A schematic diagram of an image harmonization device provided by an embodiment of the present invention;

[0074] Figure 7 A schematic diagram of an image harmonization processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0076] In embodiments of the present invention, the term "and / or" describes the association relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0077] The application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Persons skilled in the art will appreciate that as new application scenarios emerge, the technical solutions provided by the embodiments of the present invention will also be applicable to similar technical problems. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0078] Image composition is a common operation in image processing. It combines the foreground (the object of interest) of one image with the background (the scenery behind the subject, representing the subject's spatial and temporal environment) of another image to create a composite image. However, because the foreground and background are captured under different conditions (such as lighting and weather), there is a significant mismatch in brightness and color, resulting in an unharmonious composite image.

[0079] Image harmonization aims to adjust the foreground in a composite image so that it looks harmonious with the background. Currently, the mainstream method for image harmonization is manual photo editing by people using software such as PS. Although the effect is good, the labor cost is high and it cannot be used for batch image harmonization. The image harmonization methods based on deep learning all have fixed image input and output sizes, and the resolution of images that can be harmonized is low, making it difficult to handle the harmonization of higher-resolution images in life. Although the 3D lookup table is a relatively common image enhancement method, it is simple to use and can be understood as a function. The input is the value of the three primary colors and the output is a new color. The 3D lookup table itself is a three-dimensional structure. The three-dimensional coordinates represent the values ​​of the three primary colors (RGB) of the input, and the values ​​on the corresponding coordinates are the values ​​to be output, such as Figure 1 As shown in the figure, for the sake of intuitiveness, the values ​​of corresponding points are directly marked with the corresponding output colors. Therefore, the 3D lookup table structure is equivalent to a direct mapping from color to color, which is independent of the spatial position in the image and only related to the input color.

[0080] However, current 3D lookup tables are generally manually designed and applied to individual tasks. They are usually designed by experts based on specific problems. For example, if the problem is to reduce contrast, then the experts will design a 3D lookup table mapping that can reduce contrast. For batch image harmonization tasks, relying on only a few manually designed 3D lookup tables is far from sufficient in terms of expressiveness. Moreover, the need for experts to design 3D lookup tables also results in a large demand for human resources.

[0081] An embodiment of the present invention provides a method that combines deep learning and a 3D lookup table, and uses a training synthetic image-training original image to train a 3D lookup table network, thereby improving the expressive power of the 3D lookup table. After feature extraction of the training synthetic image, the extracted features are input into each sub-network, and a prediction network is determined based on the weights of each sub-network. The weights of each sub-network are trained using the corresponding training original image. Different synthetic images are input into the three-dimensional lookup table network, and the weights of each sub-network obtained are different. The prediction network is determined using the weights of the sub-networks and the sub-networks, and the synthetic image is harmonized using the prediction network to finally obtain a harmonious image.

[0082] In some embodiments, as Figure 2A As shown, the network architecture for image harmonization processing provided by this embodiment includes a feature extraction network 200 and a three-dimensional lookup table network 201, wherein the three-dimensional lookup table network 201 includes: multiple sub-networks and a prediction network, wherein each sub-network is used to represent each three-dimensional lookup table, and the prediction network is used to represent a new three-dimensional lookup table determined based on each three-dimensional lookup table and the corresponding weights.

[0083] In some embodiments, as Figure 2B As shown, a network structure for low resolution can also be designed, that is, after the feature extraction network 200 (equivalent to the encoder), a decoder 202 is added to restore the extracted features using the decoder, thereby obtaining a harmonized image. This simple network structure is used to process low-resolution synthetic images and provide feature extraction for high-resolution synthetic images. In addition, in order to reduce the amount of calculation, as shown in FIG. Figure 2C As shown, the high-resolution synthetic image can also be downsampled to a low-resolution image of a fixed size before feature extraction.

[0084] During implementation, the synthetic image is input into the feature extraction network 200 for feature extraction to obtain synthetic features; the synthetic features are input into multiple sub-networks to determine the weights of each sub-network, and each sub-network and the corresponding weight are weighted and summed to obtain a prediction network, which can be understood as a predicted 3D lookup table. The prediction network is used to harmonize the synthetic image, and the foreground in the initial harmonious image after harmonization is synthesized with the background of the synthetic image to obtain a harmonious image of the synthetic image.

[0085] It should be noted that the synthetic features extracted from the synthetic image in this embodiment are used to determine the weights of each sub-network, that is, to determine the weights of each 3D lookup table. In the process of harmonizing the synthetic image using the prediction network, since the prediction network is also a 3D lookup table, the synthetic features of the synthetic image are not used for harmonization. Instead, the three primary color information in the synthetic image is used to find the predicted color.

[0086] like Figure 3 As shown, the implementation process of the image harmonization method provided by this embodiment is as follows:

[0087] Step 300: extract features from the composite image to be processed to obtain composite features;

[0088] In some embodiments, the feature extraction network used in this embodiment is a lightweight neural network based on a U-net structure. The feature extraction network in this embodiment is primarily a lightweight network used to solve the task of harmonizing low-resolution composite images. The loss function of the feature extraction network is determined based on the result obtained after inputting the composite image into the feature extraction network and the original image corresponding to the composite image.

[0089] During implementation, since it is necessary to make the three-dimensional lookup table network perform harmonization processing and clarify the foreground that needs to be harmonized and the background that does not need to be harmonized in the synthesized image, it is necessary to extract features from the mask image of the foreground. The obtained synthetic features include the features of the foreground and background, and by using the features extracted from the mask image of the foreground, it is possible to know which synthetic features are foreground features and which synthetic features are background features.

[0090] In some embodiments, the synthetic features may be obtained by the following steps:

[0091] Step 1-1, performing binarization processing on the composite image to obtain a mask image of the foreground;

[0092] In implementation, each pixel of the composite image is binarized according to whether it belongs to the foreground or the background, thereby obtaining a mask image of the foreground of the composite image.

[0093] Step 1-2: extract features from the composite image and the mask image to obtain the composite features.

[0094] Generally speaking, the overall features of an image are not closely related to its details. For example, if a portrait is harmonized, no matter how fine the facial details are, the extracted features are similar, and the resulting image transformations are also similar. Therefore, if the input of this embodiment is a high-definition composite image (a high-resolution composite image), then in order to increase the operating speed and reduce the amount of computation, the high-definition image to be harmonized and the high-definition mask image of the foreground of the high-definition image can be downsampled to a fixed-size low resolution, thereby obtaining a low-resolution image to be harmonized and a corresponding low-resolution mask image. This reduces the size of the image to be processed, thereby increasing the operating speed and reducing the amount of computation.

[0095] In some embodiments, before extracting features from the composite image to be processed, the method further includes:

[0096] If the resolution of the composite image is higher than a threshold, downsampling the composite image;

[0097] The down-sampled image is determined as the composite image to be processed.

[0098] Step 301: Determine the weights of each sub-network in a trained three-dimensional lookup table network using the synthesized features, wherein the three-dimensional lookup table network is obtained by training an initial three-dimensional lookup table network using a training synthesized image as input and an original training image corresponding to the training synthesized image as output;

[0099] During implementation, the three-dimensional lookup table network includes various sub-networks and a prediction network, wherein each sub-network is used to represent each basic 3D lookup table, such as four 3D lookup tables running in parallel, and the prediction network is used to represent a new 3D lookup table generated based on each 3D lookup table and the corresponding weights. At the same time, the prediction network is dynamic, and the prediction network is determined based on the weights of each sub-network and each sub-network. Different synthetic images may have different corresponding weights of each sub-network, and the resulting prediction network is also different. Therefore, the three-dimensional lookup table network in this embodiment is dynamically changing and varies with different synthetic images.

[0100] In some embodiments, to determine the weights of each sub-network based on the synthesized features, a very lightweight network can be used for learning, where the number of output weights is used to represent the number of sub-networks (basic 3D lookup tables). The structure of the network can be an average pooling layer added between two convolutional layers. This attention-like network structure greatly reduces the amount of computation while maintaining the accuracy of feature extraction.

[0101] It should be noted that in the process of harmonizing the composite image, it is hoped that the background of the composite image remains unchanged. For a composite image with a large foreground and a small foreground, if weight information is not added to the composite image, the composite image with a small foreground will obviously have a smaller loss value than the composite image with a large foreground, and the harmonization of the corresponding foreground will be more difficult to train. To overcome this problem, the loss function in this embodiment uses a weighted mean square error loss, that is, the weight of the image is inversely proportional to the size of the foreground (which can be understood as the pixel value of the foreground). The larger the foreground, the smaller the weight of the composite image. Based on this method, the loss weight of the loss value corresponding to each training composite image in the training sample is set, which can avoid the risk of the corresponding image weight being too large due to the foreground being too small, thereby making the model more sensitive.

[0102] In implementation, the trained three-dimensional lookup table network is determined by the following steps:

[0103] Step 2-1, determining a loss value based on a training original image corresponding to the training composite image and a training harmonic image obtained by harmonizing the training composite image through an initial three-dimensional lookup table network;

[0104] Step 2-2: determining a loss weight of a loss value corresponding to the training synthetic image based on a comparison result of a pixel value of a foreground in the training synthetic image and a threshold, wherein the loss weight is inversely proportional to the size of the pixel value;

[0105] The threshold can be freely selected and can generally be 1000.

[0106] Step 2-3: Determine the loss function based on the loss value and loss weight corresponding to each training synthetic graph;

[0107] Step 2-4: If the function value of the loss function is less than a preset value, determining that the initial three-dimensional lookup table network is the trained three-dimensional lookup table network.

[0108] Since the background in this embodiment is obtained by pasting, only the foreground of the composite image is considered when calculating the loss. In order to solve the problem of different weights of different images caused by different foreground sizes, this embodiment uses weighted mean square error loss as the loss function.

[0109] Step 302: Determine a prediction network in the three-dimensional lookup table network according to the sub-networks and the weights of the sub-networks;

[0110] Step 303: Harmonize the composite image according to the prediction network to obtain a harmony graph of the composite image.

[0111] The core of the embodiment of the present invention is to use each sub-network and the corresponding weights to merge into a new 3D lookup table (prediction network), and use the corresponding lookup table algorithm to directly obtain the harmonized result.

[0112] In some embodiments, a weighted sum is performed based on the sub-networks and the weights of the sub-networks to obtain a prediction network.

[0113] In some examples, the composite image is harmonized using the prediction network to obtain an initial harmonic image; and a harmonic image of the composite image is generated based on the foreground in the initial harmonic image and the background of the composite image. In implementation, the prediction network first determines predicted colors corresponding to each of the three primary colors in the composite image; each of the three primary colors in the composite image is processed into the corresponding predicted colors to obtain the initial harmonic image; and then, the foreground in the initial harmonic image is composited with the background of the composite image to generate a final harmonic image.

[0114] In general computer images, the value of each primary color is an integer between 0 and 255. Therefore, in theory, a 3D lookup table can have 256 scales for each dimension. In this way, for any input composite image, the results corresponding to each of the three primary colors in the image can be directly looked up. However, if each dimension has 256 values, this will lead to the following problems: First, the storage capacity is huge. If there are several such 3D lookup tables, the memory occupied will be in GB, which is not suitable for practical use. Second, if the design adheres to the principle that two inputs that are similar before mapping should also be similar after mapping, and the mapping value is learned for each point without other constraints, overfitting will easily occur, and the output values ​​between two adjacent points may differ greatly. Third, such direct mapping is not conducive to the propagation of gradients, making it difficult to combine with deep learning methods for learning.

[0115] Therefore, this embodiment does not adopt a direct mapping approach. Instead, each dimension is limited to a set number of scales (less than 256), for example, 33 scales per dimension. At this scale, the storage capacity is only in MB. Due to the coprimacy between 33 and 255, if the scale is expanded into a grid and the original space is divided into cubes, the three primary color inputs of each composite image will fall within a small cube.

[0116] For the three primary colors in the same small cube, it is hoped that the predicted colors found out are relatively close. Therefore, this embodiment uses the following method to obtain the corresponding predicted colors:

[0117] If the prediction network does not include the three primary colors in the synthesized image, predicted colors corresponding to the three primary colors are determined from the prediction network according to a trilinear interpolation method.

[0118] In some embodiments, as Figure 4 As shown in the figure, the three-dimensional space of the prediction network (new 3D lookup table) is divided into cubes. If the three primary colors in the synthetic image do not belong to the vertices of the cubes in the three-dimensional space, but belong to the interior of a cube, the predicted colors corresponding to the three primary colors of the output are calculated using the colors of the eight vertices of the cube and the three primary colors of the input synthetic image.

[0119] Among them, let the side length of the cube be s, for the three primary colors of the composite image, that is, the point inside the cube (x d ,y d ,z d ),set up

[0120] The output value c(x,y,z) of the internal point is calculated by the following formula, that is, the predicted color:

[0121] c(x,y,z)=(1-x d )(1-y d )(1-z d )c(i,j,k)+x d (1-y d )(1-z d )c(i+s,j,k)+(1-x d )y d (1-z d )c(i,j+s,k)+(1-x d )(1-y d )z d c(i,j,k+s)+x d y d (1-z d )c(i+s,j+s,k)+x d (1-y d )z d c(i+s,j,k+s)+(1-x d )y d z d c(i,j+s,k+s)+x d y d z d c(i+s)(j+s)(k+s);

[0122] Where c() represents the output value obtained by searching the prediction network for the value in the brackets, that is, the predicted color.

[0123] The interpolation method in this embodiment ensures that the outputs of adjacent points in the same space are similar, and can also perform parallel calculations on the image on the Graphics Processing Unit (GPU), thereby greatly improving the running speed.

[0124] It's important to note that the size of each cube in the prediction network reflects the number of parameters and the complexity of the 3D lookup table. Larger cubes mean fewer parameters, less learnable content, weaker expressiveness, and a greater tendency toward underfitting. Smaller cubes, on the other hand, mean more parameters, more learnable content, and stronger expressiveness, but also a greater tendency toward overfitting. Therefore, this solution takes both into account and adopts a neutral strategy: setting fewer scales (33) per dimension in the prediction network to segment its three-dimensional space.

[0125] Since both the foreground and background in the initial harmonious image are harmonized, in order to ensure that the background of the harmonized image is consistent with the background in the synthesized image, only the foreground in the initial harmonious image and the background of the synthesized image are captured, and finally a harmonious image of the synthesized image is obtained.

[0126] In some embodiments, the foreground in the initial harmony map is pasted into the background of the composite image to generate the harmony map of the composite image.

[0127] This embodiment of the present invention utilizes multiple basic 3D lookup tables (sub-networks). The weights of each basic 3D lookup table are determined by the features of the composite image, and a weighted sum of these basic 3D lookup tables is performed to obtain a new 3D lookup table (prediction network) corresponding to the composite image. The input composite image is then calculated using the aforementioned trilinear interpolation algorithm. Finally, only the foreground portion of the harmonized initial harmonized image is retained and pasted onto the background of the pre-harmonized composite image to obtain the final harmonized result. This embodiment can utilize a combination of deep learning and 3D lookup tables, using the features of the composite image to train the weights of each 3D lookup table, so that the 3D lookup table learns the parameters corresponding to the harmonization process, thereby dynamically designing the 3D lookup table.

[0128] The harmonization method in this embodiment does not consider the spatial information of the image. Therefore, when harmonizing the foreground, points in the background that share the same pixels as the foreground will also be harmonized, causing the background color to change. This runs counter to the goal of minimizing background change. If the 3D lookup table network is trained to minimize background changes, the constraints on foreground changes will be significant. To address this difficulty, this embodiment incorporates a background-pasting operation at the end. This operation takes the harmonized foreground result and pastes it onto the unharmonized background image, allowing the 3D lookup table to focus on learning the foreground mapping solution.

[0129] The highlight of this embodiment is that for each synthetic image, there is a corresponding set of weights. Deep learning technology is integrated with traditional technology to realize the dynamic design of 3D lookup tables, which greatly improves the expression ability while giving full play to the speed and space advantages of the lookup table. Therefore, the dynamic method of predicting weights is irreplaceable in this embodiment, and the network for predicting weights is a streamlined and efficient network structure that cannot be replaced by a simple convolutional neural network or a simple autoencoder.

[0130] In some embodiments, as Figure 5 As shown, the embodiment of the present invention also provides a detailed implementation process of image harmonization processing, as shown below:

[0131] Step 500: down-sampling the composite image with a resolution higher than a threshold to obtain a composite image to be processed;

[0132] Step 501: Binarize the composite image to obtain a foreground mask image; perform feature extraction on the composite image and the mask image to obtain composite features.

[0133] Step 502: Using the synthesized features, determine the weights of each sub-network in the trained three-dimensional lookup table network;

[0134] The three-dimensional lookup table network is obtained by training an initial three-dimensional lookup table network using a training synthetic image as input and a training original image corresponding to the training synthetic image as output;

[0135] The trained three-dimensional lookup table network is determined as follows:

[0136] Determining a loss value based on a training original image corresponding to the training composite image and a training harmonic image obtained by harmonizing the training composite image through an initial three-dimensional lookup table network;

[0137] Determining a loss weight of a loss value corresponding to the training synthetic image according to a comparison result of a pixel value of a foreground in the training synthetic image and a threshold, wherein the loss weight is inversely proportional to the size of the pixel value;

[0138] Determine the loss function based on the loss value and loss weight corresponding to each training synthetic graph;

[0139] If the function value of the loss function is less than a preset value, the initial three-dimensional lookup table network is determined to be the trained three-dimensional lookup table network.

[0140] Step 503: Determine a prediction network in the three-dimensional lookup table network based on each sub-network and its weight;

[0141] Step 504: Determine the predicted colors corresponding to the three primary colors in the composite image from the prediction network; process the three primary colors in the composite image into the corresponding predicted colors to obtain an initial harmony image.

[0142] In implementation, if the prediction network does not include the three primary colors in the synthesized image, predicted colors corresponding to the three primary colors are determined from the prediction network according to trilinear interpolation.

[0143] Step 505: Paste the foreground of the initial harmony image into the background of the composite image to generate a harmony image.

[0144] Based on the same inventive concept, an embodiment of the present invention also provides a device for image harmonization processing. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0145] like Figure 6 As shown, the device includes a processor 600 and a memory 601, wherein the memory 601 is used to store a program executable by the processor 600, and the processor 600 is used to read the program in the memory 601 and perform the following steps:

[0146] Perform feature extraction on the synthetic image to be processed to obtain synthetic features;

[0147] Determining weights of each sub-network in a trained three-dimensional lookup table network using the synthesized features, wherein the three-dimensional lookup table network is obtained by training an initial three-dimensional lookup table network using a training synthesized image as input and an original training image corresponding to the training synthesized image as output;

[0148] Determining a prediction network in the three-dimensional lookup table network according to the sub-networks and the weights of the sub-networks;

[0149] The synthetic image is harmonized according to the prediction network to obtain a harmonic graph of the synthetic image.

[0150] In some embodiments, the processor 600 is specifically configured to execute:

[0151] performing binarization processing on the synthesized image to obtain a mask image of the foreground;

[0152] Feature extraction is performed on the composite image and the mask image to obtain the composite feature.

[0153] In some embodiments, the processor 600 is specifically configured to execute:

[0154] performing harmonization processing on the synthetic image according to the prediction network to obtain an initial harmonized image;

[0155] A harmony map of the composite image is generated according to the foreground in the initial harmony map and the background of the composite image.

[0156] In some embodiments, the processor 600 is specifically configured to execute:

[0157] Determining, from the prediction network, predicted colors corresponding to each of the three primary colors in the composite image;

[0158] Each of the three primary colors in the composite image is processed into a corresponding predicted color to obtain the initial harmony image.

[0159] In some embodiments, the processor 600 is specifically configured to execute:

[0160] If the prediction network does not include the three primary colors in the synthesized image, predicted colors corresponding to the three primary colors are determined from the prediction network according to a trilinear interpolation method.

[0161] In some embodiments, the processor 600 is specifically configured to execute:

[0162] The foreground of the initial harmony image is pasted into the background of the composite image to generate a harmony image of the composite image.

[0163] In some embodiments, before extracting features from the composite image to be processed, the processor 600 is further configured to execute:

[0164] If the resolution of the composite image is higher than a threshold, downsampling the composite image;

[0165] The down-sampled image is determined as the composite image to be processed.

[0166] In some embodiments, the processor 600 is specifically configured to determine the trained three-dimensional lookup table network in the following manner:

[0167] Determining a loss value based on a training original image corresponding to the training composite image and a training harmonic image obtained by harmonizing the training composite image through an initial three-dimensional lookup table network;

[0168] Determining a loss weight of a loss value corresponding to the training synthetic image according to a comparison result of a pixel value of a foreground in the training synthetic image and a threshold, wherein the loss weight is inversely proportional to the size of the pixel value;

[0169] Determine the loss function based on the loss value and loss weight corresponding to each training synthetic graph;

[0170] If the function value of the loss function is less than a preset value, the initial three-dimensional lookup table network is determined to be the trained three-dimensional lookup table network.

[0171] Based on the same inventive concept, an embodiment of the present invention further provides a device for image harmonization processing. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0172] like Figure 7 As shown, the device includes:

[0173] A feature extraction unit 700 is used to extract features from the composite image to be processed to obtain composite features;

[0174] a weight determination unit 701 for determining weights of each sub-network in a trained three-dimensional lookup table network using the synthesized features, wherein the three-dimensional lookup table network is obtained by training an initial three-dimensional lookup table network using a training synthesized image as input and an original training image corresponding to the training synthesized image as output;

[0175] A harmony processing unit 702 is configured to determine a prediction network in the three-dimensional lookup table network according to the sub-networks and the weights of the sub-networks;

[0176] The synthesis generation unit 703 is configured to perform harmonization processing on the synthesized image according to the prediction network to obtain a harmonization graph of the synthesized image.

[0177] Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0178] Perform feature extraction on the synthetic image to be processed to obtain synthetic features;

[0179] Determining weights of each sub-network in a trained three-dimensional lookup table network using the synthesized features, wherein the three-dimensional lookup table network is obtained by training an initial three-dimensional lookup table network using a training synthesized image as input and an original training image corresponding to the training synthesized image as output;

[0180] Determining a prediction network in the three-dimensional lookup table network according to the sub-networks and the weights of the sub-networks;

[0181] The synthetic image is harmonized according to the prediction network to obtain a harmonic graph of the synthetic image.

[0182] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0183] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that performs the functions specified in one or more boxes.

[0184] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0186] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for image harmonization processing, characterized in that: The method includes: Binarizing the composite image to be processed to obtain a mask image of the foreground of the composite image; extracting features from the composite image and the mask image to obtain composite features, wherein the composite features include features of the foreground and background; Determining weights of each sub-network in a trained three-dimensional lookup table network using the synthesized features, wherein the three-dimensional lookup table network is obtained by training an initial three-dimensional lookup table network using a training synthesized image as input and an original training image corresponding to the training synthesized image as output; Determining a prediction network in the three-dimensional lookup table network according to the sub-networks and the weights of the sub-networks; performing harmonization processing on the synthesized image according to the prediction network to obtain an initial harmonic image; synthesizing the foreground of the initial harmonic image with the background of the synthesized image to obtain a harmonic image of the synthesized image; The trained three-dimensional lookup table network is determined by: Determining a loss value based on a training original image corresponding to the training composite image and a training harmonic image obtained by harmonizing the training composite image through an initial three-dimensional lookup table network; Determining a loss weight of a loss value corresponding to the training synthetic image according to a comparison result of a pixel value of a foreground in the training synthetic image and a threshold, wherein the loss weight is inversely proportional to the size of the pixel value; Determine the loss function based on the loss value and loss weight corresponding to each training synthetic graph; If the function value of the loss function is less than a preset value, the initial three-dimensional lookup table network is determined to be the trained three-dimensional lookup table network.

2. The method according to claim 1, characterized in that The step of performing harmonization processing on the synthetic image according to the prediction network to obtain an initial harmonized image includes: Determining, from the prediction network, predicted colors corresponding to each of the three primary colors in the composite image; Each of the three primary colors in the composite image is processed into a corresponding predicted color to obtain the initial harmony image.

3. The method according to claim 2, characterized in that Determining, from the prediction network, predicted colors corresponding to the three primary colors in the synthesized image includes: If the prediction network does not include the three primary colors in the synthesized image, predicted colors corresponding to the three primary colors are determined from the prediction network according to a trilinear interpolation method.

4. The method according to claim 1, wherein Generating a harmony image of the synthesized image according to the foreground of the initial harmony image and the background of the synthesized image includes: The foreground of the initial harmony image is pasted into the background of the composite image to generate a harmony image of the composite image.

5. The method according to any one of claims 1 to 4, characterized in that: Before extracting features from the composite image to be processed, the method further includes: If the resolution of the composite image is higher than a threshold, downsampling the composite image; The down-sampled image is determined as the composite image to be processed.

6. A device for image harmonization processing, characterized in that: The device includes a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the steps of any one of the methods according to claims 1 to 5.

7. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images