Harmonization method of high-definition image, neural network training method, and electronic device

Abstract

The application relates to the technical field of image processing, and provides a harmonization method of a high-definition image, a neural network training method and electronic equipment, to solve the problem of how to harmonize a high-definition image. First, a high-definition image to be harmonized and a high-definition mask image of the high-definition image to be harmonized are acquired, then, the high-definition image to be harmonized and the high-definition mask image are respectively subjected to downsampling processing, to obtain a first image and a first mask. The first mask is used for harmonization processing of the first image, to obtain a second image. Then, the second image is upsampled to the resolution of the high-definition image to be harmonized, to obtain a third image. Finally, the third image is subjected to a deblurring operation, to obtain a high-definition image after harmonization. In summary, the application can save time and resources when harmonizing a high-definition image, improve the speed and efficiency of image processing, achieve a better image display effect, and improve user experience.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and particularly relates to a high-definition image harmonization method, a neural network training method and an electronic device. BACKGROUND

[0002] With the development of image processing technology and the increasing maturity of imaging device technology, high-definition pictures are increasingly easy to obtain, thus giving rise to the demand scenario of high-definition picture harmonization.

[0003] In image harmonization, the cut foreground needs to be pasted to the background. Due to color distribution and lighting conditions, the problem of inharmony between the foreground and the background is caused, which greatly affects the display effect of the current image.

[0004] In the related art, with the rise of neural networks, researchers have begun to try to use semantic segmentation technology to harmonize images. Some researchers regard backgrounds of different light and color as different domains, and use domain information to guide the harmonization of the foreground. However, these methods are designed for low-resolution images and are not suitable for harmonizing high-definition images. SUMMARY

[0005] The purpose of the present application is to provide a high-definition image harmonization method, a neural network training method and an electronic device to fill the gap in the field of high-definition image harmonization.

[0006] In a first aspect, the present application provides a high-definition image harmonization method, which comprises:

[0007] obtaining a high-definition image to be harmonized and a high-definition mask image of the high-definition image to be harmonized, the high-definition mask image being used to indicate a region to be harmonized in the high-definition image to be harmonized;

[0008] performing down-sampling processing on the high-definition image to be harmonized and the high-definition mask image respectively to obtain a first image and a first mask;

[0009] performing harmonization processing on the first image using the first mask to obtain a second image;

[0010] up-sampling the second image to the resolution of the high-definition image to be harmonized to obtain a third image;

[0011] performing deblurring operation on the third image to obtain a high-definition image after harmonization.

[0012] In some embodiments, performing harmonization processing on the first image using the first mask to obtain a second image, the method comprises:

[0013] An encoder of the harmonization network encodes the first image and the first mask to obtain encoded features;

[0014] A decoder of the harmonization network decodes the encoded features to obtain decoded features;

[0015] The second image is generated based on the decoded features.

[0016] In some embodiments, the third image is deblurred to obtain a harmonized high-definition image, and the method further includes:

[0017] The decoded features are up-sampled to obtain up-sampled decoded features;

[0018] The to-be-harmonized high-definition image, the high-definition mask image, the third image, and the up-sampled decoded features are fused to obtain fused features;

[0019] The fused features are down-sampled and error-reconstructed to obtain down-sampled features;

[0020] The down-sampled features are up-sampled and error-reconstructed to obtain the harmonized high-definition image.

[0021] In some embodiments, the down-sampled error reconstruction of the fused features includes:

[0022] The fused features are down-sampled by a convolution layer with a first specified step size to obtain a first down-sampled tensor;

[0023] The first down-sampled tensor is up-sampled and processed by a convolution layer with a second specified step size to obtain a first up-sampled tensor; the first up-sampled tensor and the fused features have the same resolution;

[0024] A first error tensor between the first up-sampled tensor and the fused features is determined;

[0025] The first error tensor is down-sampled by a convolution layer with a first specified step size to obtain a down-sampled error feature;

[0026] The down-sampled error feature and the first down-sampled tensor are added to obtain the down-sampled features.

[0027] In some embodiments, the up-sampled error reconstruction of the down-sampled features includes:

[0028] The down-sampled features are up-sampled and processed by a convolution layer with a second specified step size to obtain a second up-sampled tensor;

[0029] performing convolution layer down-sampling on the second up-sampling tensor with a first specified step size to obtain a second down-sampling tensor, wherein the second down-sampling tensor has the same resolution as the down-sampled feature;

[0030] determining a second error tensor between the down-sampled feature and the second down-sampling tensor;

[0031] performing up-sampling on the second error tensor to obtain an up-sampled error feature;

[0032] adding the up-sampled error feature and the second up-sampling tensor, and performing convolution layer processing with a second specified step size to obtain an up-sampled feature, and obtaining the harmonized high-definition image based on the up-sampled feature.

[0033] In some embodiments, the harmonization network comprises an encoder and a decoder; the encoder comprises a plurality of convolutional neural network layers connected in sequence, and the convolutional neural network layers are used to extract feature information of the first image and the first mask.

[0034] In a second aspect, the present application provides a neural network training method, wherein the neural network comprises a harmonization network and a deblurring network, and the method comprises:

[0035] obtaining a high-definition sample image and a sample mask image of the high-definition sample image, wherein the sample mask image is used to indicate a to-be-harmonized region in the high-definition sample image;

[0036] performing down-sampling processing on the high-definition sample image and the sample mask image respectively to obtain a training image and a mask image of the training image;

[0037] training the harmonization network using the training image and the mask image of the training image until training convergence is achieved, to obtain a trained harmonization network;

[0038] performing up-sampling on a harmonized image of the high-definition sample image output by the trained harmonization network to the resolution of the high-definition sample image to obtain an up-sampled image of the high-definition sample image; and inputting the up-sampled image to the deblurring network to obtain a harmonized high-definition image of the harmonized image;

[0039] comparing the harmonized high-definition image with a standard harmonized image of the high-definition sample image to determine a loss of the deblurring network;

[0040] updating network parameters of the deblurring network based on the loss of the deblurring network.

[0041] In some embodiments, a loss function used for training the harmonization network is:

[0042]

[0043] wherein, L1 represents a loss value, M represents a mask image of the training image, I gt represents a standard harmonization image of the training image, G represents the harmonization network, G(M, I) represents a decoded picture output by the harmonization network, and I represents the training image.

[0044] The loss function used in training the deblurring network is:

[0045]

[0046] wherein, L2 represents a loss value, M2 represents a sample mask image of the high-definition sample image, I gt represents a high-definition sample image used for supervised training, G2 represents the deblurring network, G2(M2, I2) represents a decoded picture output by the deblurring network, and I2 represents the harmonized high-definition image.

[0047] In some embodiments, before the inputting the up-sampled image into the deblurring network to obtain the harmonized high-definition image of the harmonized image, the method further comprises:

[0048] obtaining decoded features of the training image and the training mask extracted in a decoder in the harmonization network;

[0049] performing feature fusion on the high-definition sample image, the high-definition sample mask image, the up-sampled image, and the decoded features to obtain fusion features as the up-sampled image input into the deblurring network.

[0050] In some embodiments, the deblurring network comprises a down-sampling error reconstruction module and an up-sampling error reconstruction module, wherein:

[0051] the down-sampling error reconstruction module is configured to perform down-sampling error reconstruction on the fusion features to obtain down-sampled features;

[0052] the up-sampling error reconstruction module is configured to perform up-sampling error reconstruction on the down-sampled features to obtain up-sampled features, and obtain the harmonized high-definition image of the harmonized image based on the up-sampled features.

[0053] In a third aspect, the present application further provides an electronic device comprising:

[0054] a display configured to display a user interface of an image;

[0055] a memory configured to store executable instructions of the processor;

[0056] A processor for executing the executable instructions to implement any of the methods provided in the first and second aspects of this application.

[0057] Fourthly, one embodiment of this application also provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the methods provided in the first and second aspects of this application.

[0058] Fifthly, one embodiment of this application provides a computer program product including a computer program that, when executed by a processor, implements any of the methods provided in the first and second aspects of this application.

[0059] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0060] This invention provides a high-quality solution for harmonizing foreground and background in high-definition images. It employs a progressive neural network to process high-resolution images, achieving high-quality and efficient image harmonization. Based on the progressive neural network, high-resolution images are first harmonized at a lower resolution, significantly improving algorithm speed and effectively solving the problem of high hardware resource consumption in high-definition image processing. Furthermore, the progressive neural network upsamples the harmonized low-resolution image to high resolution and further improves the high-resolution image, greatly enhancing the quality of high-resolution image harmonization. In summary, when using the high-definition image harmonization method of this invention to process high-definition images, users save time and resources, improve efficiency, and achieve better image display results.

[0061] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. Based on common knowledge in the art, the above preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this application. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 A flowchart illustrating a method for harmonizing high-definition images provided in an embodiment of this application;

[0064] Figure 2 A flowchart illustrating the portrait segmentation algorithm provided in this application embodiment;

[0065] Figure 3 A flowchart illustrating a neural network module for low-resolution image harmonization provided in an embodiment of this application;

[0066] Figure 4 A schematic flowchart of a neural network for low-resolution image harmonization provided in an embodiment of this application;

[0067] Figure 5 A flowchart illustrating the depth projection error reconstruction module provided in an embodiment of this application;

[0068] Figure 6 A schematic flowchart of a neural network for high-resolution image deblurring provided in an embodiment of this application;

[0069] Figure 7 A flowchart illustrating the feature fusion module provided in an embodiment of this application;

[0070] Figure 8 A flowchart illustrating the downsampling space reconstruction module provided in an embodiment of this application;

[0071] Figure 9 A flowchart illustrating the upsampling space reconstruction module provided in an embodiment of this application;

[0072] Figure 10 A flowchart illustrating a neural network training method provided in an embodiment of this application;

[0073] Figure 11 A schematic diagram illustrating a training dataset example provided in an embodiment of this application;

[0074] Figure 12 A flowchart illustrating the overall framework of the high-definition image harmonization method provided in the embodiments of this application;

[0075] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0077] Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0078] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0079] Image harmonization aims to address the problem of foreground and background disharmony when a cropped foreground is pasted onto a background due to factors such as color distribution and lighting conditions. Traditional image harmonization methods employ gradient fields, color distribution matching, and multi-scale statistics to achieve foreground-background harmony. However, these methods are based on statistical information or predefined features, resulting in poor generalization performance. With the rise of neural networks, researchers have begun to explore using semantic maps obtained from semantic segmentation to guide image harmonization. Some researchers treat backgrounds with different lighting and colors as different domains, utilizing domain information to guide foreground harmonization. However, these methods are not designed for high-resolution image harmonization; they address the problem of harmonizing low-resolution images. Therefore, how to harmonize high-resolution images is a significant concern in the field.

[0080] Current image harmonization methods primarily address low-resolution images, or create disharmony between foreground and background, significantly impacting the image's display quality. Therefore, this application provides a method for harmonizing high-resolution images, a neural network training method, and an electronic device to solve the problem of the inability to harmonize high-resolution images.

[0081] The method for harmonizing high-definition images provided in this application will be described below with reference to embodiments.

[0082] The inventive concept of this application can be summarized as follows: First, a high-definition image to be harmonized and a high-definition mask image of the high-definition image to be harmonized are obtained, wherein the high-definition mask image is used to indicate the region to be harmonized in the high-definition image to be harmonized, such as a human figure in the high-definition image to be harmonized. Then, the high-definition image to be harmonized and the high-definition mask image are downsampled respectively to obtain a first image and a first mask. Then, the first mask is used to harmonize the first image to obtain a second image. After that, the second image is upsampled to the resolution of the high-definition image to be harmonized to obtain a third image. Finally, the third image is deblurred to obtain the harmonized high-definition image. In summary, the embodiments of this application, under the current technical conditions, allow users to process high-definition images using the high-definition image harmonization method of this invention. By downsampling the high-definition image to be harmonized and the high-definition mask image to obtain a first image and a first mask, and then using the first mask to harmonize the first image to obtain a second image, this process reduces the resolution of the high-definition image, saves time and resources, and improves the speed and efficiency of image processing. Finally, the upsampling and deblurring operations on the third image can improve the resolution and quality of the image.

[0083] After introducing the main inventive concepts of the embodiments of this application, the following is a brief description of the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0084] To fill the gap in the current field of image processing for high-definition image harmonization, this application provides a method for harmonizing high-definition images. Based on the above description, a flowchart of a high-definition image harmonization method provided in this application embodiment is shown below. Figure 1 As shown, it includes the following:

[0085] In step 101, a high-definition image to be censored and a high-definition mask image of the high-definition image to be censored are obtained. The high-definition mask image is used to indicate the area to be censored in the high-definition image to be censored.

[0086] To obtain a high-resolution mask image for the high-resolution image to be harmonized, in some embodiments, for scenes where foreground and background harmonization of people is required, a portrait segmentation algorithm can be used to obtain the mask of the region to be harmonized. For example... Figure 2As shown. There are three common methods for obtaining masks. One is through object segmentation or matting algorithms, such as obtaining a mask image of the portrait when compositing a human image. Common methods such as Mobile Real-time Video Segmentation and Deep Image Matting can be used to obtain the mask image. Another method is that when a user edits an image, the software records the foreground position of the cut image to obtain the mask image. The most common method is manual mask annotation, but this method is expensive. Therefore, this embodiment of the application uses a human image segmentation algorithm. In the process of obtaining a high-definition mask image, the harmonized foreground region is determined based on the background, such as... Figure 2 In the background image, the foreground area is usually a human figure. The mask is generally represented by two values, 0 and 1, where 0 represents the background and 1 represents the foreground. After determining the foreground area that needs to be harmonized, the foreground area will appear white in the visualization, as shown below. Figure 2 The mask image in the image.

[0087] In step 102, the high-definition image to be harmonized and the high-definition mask image are downsampled respectively to obtain the first image and the first mask.

[0088] In some embodiments, downsampling is used to reduce the resolution of high-definition images, thereby reducing the high hardware resource consumption in processing high-definition images and improving processing speed. For example, if the high-definition image and the high-definition mask image obtained in step 101 both have a resolution of 1080p (1920*1080), downsampling reduces the resolution of the first image and the first mask to 256*256, thus improving image processing speed.

[0089] In step 103, the first image is harmonized using a first mask to obtain the second image.

[0090] In some embodiments, a first mask is used to harmonize the first image to obtain a second image. A neural network for harmonizing low-resolution images (hereinafter also referred to as the harmonization network) is introduced here, and its overall framework is as follows: Figure 3 As shown:

[0091] The harmonization network has a depth of 7 layers, mainly composed of a U-shaped convolutional network; two layers are omitted in the figure. The encoder consists of seven layers arranged downwards on the left, with five layers shown in the figure. The first three encoder layers capture external feature distributions, such as foreground and background colors and illumination. Notably, this application omits normalization layers in the first two encoder layers. This allows the network to retain appearance feature distributions to a greater extent during learning, making the network more adaptable to harmonization tasks. The last four layers, due to the continuously expanding receptive field of the network, can capture global contextual information. The decoder consists of seven layers arranged upwards on the right. Corresponding to the encoder, from top to bottom on the right, the first three decoder layers capture external feature distributions, such as foreground and background colors and illumination. Accordingly, this application also omits normalization layers in the first two decoder layers. The last four layers of both the decoder and encoder, due to the continuously expanding receptive field of the network, can capture global contextual information. In the figure, M represents the mask, and I represents the input image. The image represents a harmonized image, with the image resolution indicated in the upper right footnote. The steps involved in the harmonization network are as follows: Figure 4 It can be implemented as follows:

[0092] In step 401, the encoder of the harmonicization network is used to encode the first image and the first mask to obtain coded features. Figure 3 M 256 As the first mask, I 256 The first image is represented by 256, which indicates the image resolution. A suitable resolution can be set as needed during implementation. The encoded features in this embodiment are the features output by the last layer of the encoder in the figure.

[0093] In step 402, the decoder of the harmonicization network is used to decode the encoded features to obtain the decoded features. The decoded features are... Figure 3 Features of the output of the last layer of the decoder.

[0094] In step 403, a second image is generated based on the decoded features. That is, based on... Figure 3 Decoding features in the middle, generating This is the second image.

[0095] It should be noted that during the process of harmonizing the first image using the first mask, the resolution of the resulting second image remains unchanged at 256*256.

[0096] After obtaining the second image, in step 104, the second image is upsampled to the resolution of the high-definition image to be harmonized, thus obtaining the third image.

[0097] In some embodiments, after obtaining the harmonized low-resolution image, a second image is obtained. The generated second image is then upsampled using bilinear interpolation. For example, if both the original high-resolution image and the original high-resolution mask image have a resolution of 1080p, after downsampling, the resolution of the first image and the first mask is reduced to 256*256. The resolution of the second image obtained after harmonization remains unchanged. After upsampling, a third image is obtained, and the resolution of the third image is restored to 1080p. However, the obtained third image is relatively blurry. To weaken or even eliminate the mosaic effect that occurs during the upsampling process, the embodiments of this application may perform the following step 105 to further improve image quality.

[0098] In step 105, the third image is deblurred to obtain a harmonized high-definition image.

[0099] In some embodiments, a high-resolution image deblurring neural network (hereinafter also referred to as a deblurring network), also known as a depth projection error reconstruction module, is introduced, such as... Figure 5 As shown, the third image, which has undergone simple upsampling, is further processed. This network module mainly consists of two error reconstruction sub-modules: downsampling error reconstruction and upsampling error reconstruction. The specific module flow is as follows... Figure 5 As shown: the input feature is the first module on the left, and the output feature is the first module on the right. The input feature is first reconstructed using downsampling error, and then reconstructed using upsampling error to obtain the output feature. The steps of this process are as follows: Figure 6 The following can be implemented as follows:

[0100] To improve image quality, in step 601 of this embodiment, feature fusion is performed on the third image, the high-definition image to be harmonized, the high-definition mask image, the third image, and the upsampled decoding features to obtain fused features.

[0101] To obtain the aforementioned fused features, a feature fusion module is introduced, such as... Figure 7 M represents the mask, I represents the image to be harmonized, Up represents upsampling, G represents the harmonization network, and F represents the last feature map in the decoder of the harmonization network. The superscript of the letters indicates the image resolution. The fused features include: obtaining the last feature map F from the decoder of the harmonization network. 256 ,like Figure 3 The topmost decoder features are then upsampled to obtain Up(F). 256 This refers to upsampling decoding features. The high-definition image to be harmonized (I) 1080p High-definition mask image M 1080p and the third image Up(G(I) 256 M 256The images are superimposed and merged using the torch.cat function in PyTorch software to obtain a seven-channel image. Then, the image is processed by a convolutional layer to obtain the first deblurred feature. The first deblurred feature and the upsampled decoded feature are added together and processed by a convolutional layer with a stride of 1 to obtain the fused second deblurred feature.

[0102] In step 602, the fused second deblurred feature is reconstructed by downsampling error to obtain the downsampled feature.

[0103] In some embodiments, the fused features are reconstructed using downsampling error to obtain downsampling features, which is referred to as the downsampling space reconstruction module, such as... Figure 8 The downsampling space reconstruction shown can be implemented as follows:

[0104] In step 801, the fused features are downsampled through a convolutional layer with a stride of the first specified stride to obtain the first downsampled tensor. As... Figure 5 In this context, the fused features are the input features. The first specified stride is 2, and the input features are downsampled through a convolutional layer with a stride of 2 to obtain the first downsampled tensor.

[0105] In step 802, the first downsampled tensor is upsampled and processed by a convolutional layer with a stride of the second specified stride to obtain the first upsampled tensor. As... Figure 5 In the diagram, the second specified step size is 1. The first downsampled tensor is upsampled and then processed by a convolutional layer with a step size of 1 to obtain the first upsampled tensor. Based on the size of the modules in the diagram, it can be seen that the resolution of the first upsampled tensor and the fused features are the same.

[0106] In step 803, a first error tensor between the first upsampling tensor and the fused feature is determined. As... Figure 5 In this process, the input features are subtracted from the first upsampling tensor to obtain the first error tensor.

[0107] In step 804, the first error tensor is downsampled through a convolutional layer with a stride of a first specified stride to obtain the downsampling error features. Figure 5 The fourth module is the downsampling error feature.

[0108] In step 805, the downsampling error feature and the first downsampling tensor are added together to obtain the downsampling feature. Figure 5 The fifth module is the downsampling feature.

[0109] Then, in step 603, the downsampled features are reconstructed using upsampling error to obtain a harmonized high-definition image.

[0110] In other embodiments, upsampling error reconstruction is performed on the downsampled features to obtain a harmonized high-resolution image, such as... Figure 9Upsampling error reconstruction can be implemented as follows:

[0111] In step 901, the downsampled features are upsampled and processed through a convolutional layer with a stride of the second specified stride to obtain the second upsampled tensor. As... Figure 5 In the process, the fifth feature module is upsampled to obtain the sixth feature module, namely the second upsampled tensor.

[0112] In step 902, the second upsampled tensor is downsampled using a convolutional layer with a stride of the first specified stride to obtain the second downsampled tensor; as... Figure 5 In the diagram, the second upsampled tensor corresponds to the sixth feature module. After downsampling, the second downsampled tensor corresponds to the seventh feature module. Based on the size of the modules in the diagram, it can be seen that the resolution of the second downsampled tensor and the downsampled feature is the same.

[0113] In step 903, a second error tensor between the downsampling feature and the second downsampling tensor is determined. As... Figure 5 In the process, the second error tensor is obtained by subtracting the fifth feature module from the seventh feature module.

[0114] In step 904, the second error tensor is upsampled to obtain the upsampled error characteristics. Figure 5 The eighth module is the upsampling error feature.

[0115] In step 905, the upsampling error features and the second upsampling tensor are added together and processed by a convolutional layer with a stride of a second specified stride. Based on the added features, a harmonized high-definition image is obtained. Figure 5 The output features are the features obtained by addition.

[0116] Will Figure 8 and Figure 9 By concatenating the two modules, the network can better learn the mapping function to reconstruct a high-resolution image (upsampled space) from a coarse image (downsampled space). The concatenated downsampled space reconstruction module and upsampled space reconstruction module are then called depth projection error reconstruction modules. Finally, concatenating two such depth error reconstruction modules can alleviate or even eliminate the mosaic effect in a blurred image.

[0117] The entire process of the high-resolution image harmonization method includes two convolutional neural networks: a neural network for harmonizing low-resolution images and a neural network for deblurring high-resolution images. It should be noted that both neural networks are trained under supervised supervision. The specific steps of the neural network training method are as follows: Figure 10 As shown:

[0118] In step 1001, a high-resolution sample image and a sample mask image of the high-resolution sample image are acquired. The sample mask image is used to indicate the region to be harmonized in the high-resolution sample image.

[0119] In step 1002, the high-definition sample image and the sample mask image are downsampled to obtain the training image and the mask image of the training image.

[0120] In step 1003, the harmonization network is trained using the training image and the mask image of the training image until the training converges, resulting in a trained harmonization network.

[0121] During the training of this harmonization network, an adaptive loss function based on the foreground region area is proposed. This loss function enables the neural network to respond well to foregrounds of different scales, i.e., the regions to be harmonized, effectively improving the robustness of the model. Its expression is shown in Equation 1:

[0122]

[0123] Where L1 represents the loss value, M represents the mask image of the training image, such as the mask image of the training image in step 1002, and I gt The standard harmonicized image of the training image is represented, such as the training image in step 1002. G represents the harmonicization network mentioned above. G(M, I) represents the decoded image output by the harmonicization network. I represents the training image, such as the training image in step 1002.

[0124] In step 1004, the harmonized image of the high-definition sample image output by the trained harmonization network is upsampled to the resolution of the high-definition sample image to obtain the upsampled image of the high-definition sample image.

[0125] In step 1005, the upsampled image is input into the deblurring network to obtain a high-resolution image of the blurred image.

[0126] In step 1006, the standard harmonized images of the harmonized high-definition image and the high-definition sample image are compared to determine the loss of the deblurring network.

[0127] In step 1007, the network parameters of the deblurring network are updated based on the loss of the deblurring network.

[0128] The same loss function was used during the training of this deblurring network, and its expression is shown in Equation 2:

[0129]

[0130] Where L2 represents the loss value, M2 represents the sample mask image of the high-definition sample image, such as the sample mask image of the high-definition sample image in step 1001, I gt ′ represents the high-resolution sample image used for supervised training, such as the high-resolution sample image in step 1001, G2 represents the deblurring network, G2(M2, I2) represents the decoded image output by the deblurring network, and I2 represents the obfuscated high-resolution image, such as the obfuscated high-resolution image in step 1006.

[0131] In general, the above neural network training method consists of two steps: the first step is to train the neural network on the public dataset iHarmony4, with the training dataset sample as follows: Figure 11 As shown; the second step is to fine-tune the low-resolution harmonizing network and the high-resolution deblurring network together on the user-collected dataset. When the network's peak signal-to-noise ratio (PSNR) no longer improves on the validation set, the network training is considered complete.

[0132] For the neural network used for low-resolution image harmonization, we employed the publicly available dataset iHarmony4 as the pre-training training set during training, and then fine-tuned it on various types of high-resolution harmonization datasets. With this convolutional neural network, we were able to successfully harmonize low-resolution (256*256) images, making the pasted foreground and background appear harmonious in terms of lighting and color.

[0133] For the neural network used for high-resolution image deblurring, the loss function used during training is the same as that used for the neural network for low-resolution image harmonization, which is the loss function in Equation 1. During training, the neural network used for low-resolution image harmonization is fine-tuned on a user-built dataset. Similarly, when the peak signal-to-noise ratio (PSNR) no longer improves on the validation set, the model can be considered to have completed training.

[0134] In another embodiment, the overall framework of the high-definition image harmonization method of this application is as follows: Figure 12 As shown, the first image I after downsampling processing 256 and the first mask M 256 The image is fed into a convolutional network module, which is the low-resolution image harmonicization module mentioned above, to obtain the decoded features. These features are then upsampled to obtain the upsampled decoded features Up(F). 256 The upsampled decoding feature Up(F) is then used. 256 ) and the high-definition image to be harmonized I 1080p High-definition mask image M 1080p and the third image Up(G(I) 256 M 256The features obtained by fusing these three features are then processed by the feature fusion module to obtain the fused features. Finally, the fused features are fed into the depth projection error reconstruction module to obtain the harmonized high-resolution image.

[0135] The above steps truly fill a gap in the field of image processing. As a result, our entire algorithm can efficiently and effectively complete the harmonization of high-resolution images, saving time and resources, improving efficiency, and achieving better image display effects.

[0136] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0137] In some possible implementations, the electronic device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the high-definition image retouching method and neural network training method described above according to various exemplary embodiments of this application. For example, the processor may perform steps such as the high-definition image retouching method and the neural network training method.

[0138] The following reference Figure 13 To describe an electronic device 130 according to this embodiment of the present application. Figure 13 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0139] like Figure 13 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0140] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0141] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0142] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0143] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0144] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 132 including instructions. These instructions can be executed by the processor 131 of the electronic device 700 or the processor 131 of the electronic device 800 to complete the aforementioned high-definition image harmonization method and neural network training method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0145] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor 131, implements either the high-definition image harmonization method or the neural network training method provided in this application.

[0146] In an exemplary embodiment, various aspects of the high-definition image harmonization method and neural network training method provided in this application can also be implemented as a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps of the high-definition image harmonization method and neural network training method according to the various exemplary embodiments of this application described above.

[0147] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, electronic device, or apparatus, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0148] The program products of the methods for harmonizing high-definition images and training neural networks according to the embodiments of this application can be in the form of a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device. However, the program products of this application are not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, electronic device, or apparatus.

[0149] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, electronic device, or apparatus.

[0150] The program code contained on the readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wired, fiber optic, RF, etc., or any suitable combination thereof.

[0151] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0152] It should be noted that although several units or sub-units of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0153] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0154] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable image scaling device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable image scaling device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 An electronic device that specifies the functions in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable image scaling device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction electronic equipment, which is implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable image scaling device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0159] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for harmonizing high-definition images, characterized in that, The method includes: Acquire a high-definition image to be censored and a high-definition mask image of the high-definition image to be censored, wherein the high-definition mask image is used to indicate the area to be censored in the high-definition image to be censored; The high-definition image to be harmonized and the high-definition mask image are respectively downsampled to obtain a first image and a first mask; An encoder using a harmonicization network encodes the first image and the first mask to obtain encoded features; The encoded features are decoded using a decoder with a harmonic network to obtain the decoded features; A second image is generated based on the decoded features; The second image is upsampled to the resolution of the high-definition image to be harmonized to obtain the third image; The decoded features are upsampled to obtain upsampled decoded features; The high-definition image to be harmonized, the high-definition mask image, the third image, and the upsampled decoding features are fused to obtain fused features; The fused features are downsampled through a convolutional layer with a stride of a first specified stride to obtain a first downsampled tensor; The first downsampled tensor is upsampled and processed by a convolutional layer with a stride of a second specified stride to obtain a first upsampled tensor; the first upsampled tensor and the fused feature have the same resolution; Determine the first error tensor between the first upsampling tensor and the fused feature; The first error tensor is downsampled through a convolutional layer with a stride of a first specified stride to obtain the downsampling error features; The downsampling error feature is added to the first downsampling tensor to obtain the downsampling feature; The downsampled features are reconstructed using upsampling error to obtain a harmonized high-definition image.

2. The method according to claim 1, characterized in that, The step of reconstructing the downsampled features by upsampling error to obtain the harmonized high-definition image includes: The downsampled features are upsampled and processed through a convolutional layer with a stride of the second specified stride to obtain a second upsampled tensor; The second upsampled tensor is downsampled by a convolutional layer with a stride of a first specified stride to obtain a second downsampled tensor; the second downsampled tensor and the downsampled feature have the same resolution; Determine the second error tensor between the downsampling feature and the second downsampling tensor; Upsample the second error tensor to obtain the upsampled error characteristics; The upsampling error feature and the second upsampling tensor are added together and processed by a convolutional layer with a stride of the second specified stride to obtain the upsampling feature. Based on the upsampling feature, the harmonized high-definition image is obtained.

3. The method according to claim 1, characterized in that, The harmonization network includes an encoder and a decoder; the encoder includes multiple sequentially connected convolutional neural network layers, which are used to extract feature information of the first image and the first mask.

4. A neural network training method, characterized in that, The neural network is used to harmonize high-definition images in the method for harmonizing high-definition images as described in any one of claims 1-3, the neural network comprising a harmonizing network and a deblurring network, the method comprising: Acquire a high-resolution sample image and a sample mask image of the high-resolution sample image, wherein the sample mask image is used to indicate the region to be harmonized in the high-resolution sample image; The high-resolution sample image and the sample mask image are downsampled respectively to obtain the training image and the mask image of the training image; The harmonization network is trained using the training images and the mask images of the training images until the training converges, resulting in a trained harmonization network. The harmonized image of the high-definition sample image output by the trained harmonization network is upsampled to the resolution of the high-definition sample image to obtain the upsampled image of the high-definition sample image; The upsampled image is input into a deblurring network to obtain a high-resolution image of the already blurred image. The loss of the deblurring network is determined by comparing the harmonized high-definition image with the standard harmonized image of the high-definition sample image. The network parameters of the deblurring network are updated based on the loss of the deblurring network.

5. The method according to claim 4, characterized in that, The loss function used to train the harmonization network is: in, Indicates the loss value. The mask image representing the training image. This represents the standard harmonicized image of the training images. This refers to the harmonized network. This represents the decoded image output by the harmonization network. This refers to the training image; The loss function used to train the deblurring network is: in, Indicates the loss value. The sample mask image representing the high-resolution sample image. This represents high-resolution sample images used for supervised training. This refers to the deblurring network. This represents the decoded image output by the deblurring network. This indicates that the image has been harmonized and is in high definition.

6. The method according to claim 5, characterized in that, Before inputting the upsampled image into the deblurring network to obtain the blurred high-resolution image of the blurred image, the method further includes: The decoding features of the training image and the mask image of the training image extracted from the decoder in the harmonization network are obtained, and the decoding features are upsampled. The high-resolution sample image, the sample mask image of the high-resolution sample image, the upsampled image, and the upsampled decoding features are fused to obtain fused features, which are then input to the upsampled image of the deblurring network.

7. An electronic device, characterized in that, include: A display, a user interface used to display images; Memory is used to store the processor's executable instructions; A processor for executing the executable instructions to implement the high-definition image harmonization method and neural network training method as described in any one of claims 1-6.

Images