Model training and image processing method, medium, device and computing equipment

Abstract

The embodiments of the present disclosure provide a model training and image processing method, medium, apparatus, and computing device. The model training method includes: obtaining multiple sets of training samples, wherein the training samples include sample images to be processed, sample face parsed images, and label-enhanced images; inputting the sample images to be processed and the sample face parsed images into an image enhancement model to obtain sample enhanced images output by the image enhancement model; and adjusting the parameters of the image enhancement model based on the sample enhanced images and the label-enhanced images. Images containing faces can be significantly enhanced without the need for cropping or splicing the images, ensuring the integrity and completeness of the images, and making the face enhancement effect and background enhancement effect in the image more coordinated.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology. More specifically, the embodiments of the present disclosure relate to a model training and image processing method, medium, apparatus and computing device. Background Art

[0002] This section is intended to provide a background or context for the embodiments of the present disclosure. No admission is made that the description herein is prior art by virtue of its inclusion in this section.

[0003] Image enhancement refers to the process of enhancing low-quality images caused by various degradation factors to improve the visual effect of the image.

[0004] In various types of images, people are often the focus. For enhancement of images containing people, the process typically involves cropping the face, enhancing the cropped face image, and then splicing the enhanced face image back into the original image.

[0005] The above solution destroys the integrity of the enhanced image due to cropping and splicing, and the enhanced image has obvious splicing traces, and the display effect of the enhanced image is poor. Summary of the Invention

[0006] The present disclosure provides a model training and image processing method, medium, apparatus and computing device to improve the display effect of enhanced images.

[0007] In a first aspect of the embodiments of the present disclosure, a model training method is provided, comprising:

[0008] Acquire multiple sets of training samples, wherein the training samples include sample images to be processed, sample face parsed images, and label enhanced images;

[0009] Inputting the sample image to be processed and the sample face parsed image into an image enhancement model to obtain a sample enhanced image output by the image enhancement model;

[0010] The parameters of the image enhancement model are adjusted according to the sample enhanced image and the label enhanced image.

[0011] In a second aspect of the embodiments of the present disclosure, there is provided an image processing method, comprising:

[0012] Perform face analysis on the image to be processed to obtain a corresponding face analysis image;

[0013] The image to be processed and the face parsed image are input into an image enhancement model to obtain an enhanced image corresponding to the image to be processed, wherein the image enhancement model is a model trained according to the model training method described in any one of the first aspects.

[0014] In a third aspect of the embodiments of the present disclosure, a model training device is provided, comprising:

[0015] An acquisition module is used to acquire multiple sets of training samples, wherein the training samples include sample images to be processed, sample face parsed images, and label enhanced images;

[0016] a processing module, configured to input the sample image to be processed and the sample face parsed image into an image enhancement model to obtain a sample enhanced image output by the image enhancement model;

[0017] A training module is used to adjust the parameters of the image enhancement model according to the sample enhanced image and the label enhanced image.

[0018] In a fourth aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, comprising:

[0019] An analysis module is used to perform face analysis on the image to be processed to obtain a corresponding face analysis image;

[0020] A processing module is used to input the image to be processed and the face parsed image into an image enhancement model to obtain an enhanced image corresponding to the image to be processed, and the image enhancement model is a model trained according to the model training method described in any one of the first aspects.

[0021] In a fifth aspect of the embodiments of the present disclosure, there is provided a computing device comprising: at least one processor and a memory;

[0022] The memory stores computer-executable instructions;

[0023] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the model training method as described in any one of the first aspects, or the at least one processor executes the image processing method as described in any one of the second aspects.

[0024] In the sixth aspect of the embodiment of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the model training method as described in any one of the first aspects is implemented, or the image processing method as described in any one of the second aspects is implemented.

[0025] In the seventh aspect of the embodiments of the present disclosure, a computer program product is provided, which includes a computer program; when the computer program is executed, it implements the model training method as described in any one of the first aspects, or implements the image processing method as described in any one of the second aspects.

[0026] The model training and image processing methods, media, devices and computing equipment provided by the embodiments of the present disclosure first obtain multiple groups of training samples, which include sample images to be processed, sample face parsed images and label enhanced images. For any group of training samples, the sample images to be processed and the sample face parsed images can be input into the image enhancement model to obtain the sample enhanced images output by the image enhancement model, so that the parameters of the image enhancement model can be adjusted according to the sample enhanced images and the label enhanced images. The image enhancement model can be trained in the above manner for any group of training samples, and finally a trained image enhancement model is obtained. After the image enhancement model is trained, the image enhancement model has the ability to perform image enhancement processing. After the image to be processed and the corresponding face parsed image are input into the image enhancement model, the image enhancement model can output the corresponding enhanced image. Since sample face analysis images are added during the training process of the image enhancement model, and the sample face analysis images include the facial prior information of the sample image to be processed, the solution of the embodiment of the present disclosure can significantly enhance images containing faces without the need for cropping and splicing the images, thereby ensuring the integrity and completeness of the images, and making the face enhancement effect and background enhancement effect in the image more coordinated. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0028] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0029] Figure 2 A flowchart of a model training method provided in an embodiment of the present disclosure;

[0030] Figure 3 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0031] Figure 4 A schematic diagram of the structure of the image enhancement model provided in an embodiment of the present disclosure;

[0032] Figure 5 A schematic diagram of the structure of a face structure embedding module provided in an embodiment of the present disclosure;

[0033] Figure 6 A schematic diagram of inter-frame transmission provided in an embodiment of the present disclosure;

[0034] Figure 7A schematic diagram of the image enhancement model training process provided in an embodiment of the present disclosure;

[0035] Figure 8 A schematic diagram of sample loss calculation provided in an embodiment of the present disclosure;

[0036] Figure 9 A schematic diagram of the image processing process to be processed provided in an embodiment of the present disclosure;

[0037] Figure 10 A schematic diagram of a storage medium provided in an embodiment of the present disclosure;

[0038] Figure 11 A schematic diagram of the structure of a model training device provided in an embodiment of the present disclosure;

[0039] Figure 12 A schematic diagram of the structure of an image processing device provided in an embodiment of the present disclosure;

[0040] Figure 13 A schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure.

[0041] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0042] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0044] According to embodiments of the present disclosure, a model training and image processing method, medium, apparatus, and computing device are proposed. It should be understood that any number of elements in the accompanying drawings is for illustration only and not for limitation, and any nomenclature is for distinction only and does not have any limiting meaning.

[0045] First, the basic concepts involved in this disclosure are introduced.

[0046] Generative Adversarial Networks: Generative Adversarial Networks, abbreviated as GAN, is a type of deep learning technology, commonly used in image generation, style transfer and image enhancement. A generative adversarial network usually contains a generator network and at least one discriminator network.

[0047] Degradation factors: refers to the reasons for poor visual effects of video images, common ones include blur, noise, compression distortion, scaling, etc.

[0048] The principles and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0050] Image enhancement refers to the process of enhancing low-quality images caused by various degradation factors and improving the visual effects of low-quality images. It can refer to enhancing low-quality images caused by a single degradation factor or multiple degradation factors.

[0051] In the mobile internet era, the number of images captured and uploaded by users is growing exponentially. Due to the diverse devices and environments used to capture images, the quality varies, and compression and transmission are common. Blur, noise, and distortion are common in images, reducing their visual quality. Therefore, image enhancement is crucial to improve their visual quality.

[0052] Most of the images captured and uploaded by users are primarily people-oriented. For images containing people, the following two enhancement solutions are available in related technologies:

[0053] The first one is to not consider the faces in the image separately, but to take the image as a whole and perform overall enhancement.

[0054] The second method is to extract the face from the image, enhance the face and background of the image separately, and then fuse the enhanced face and background to obtain the enhanced overall image.

[0055] The inventors have discovered that the first approach, because it enhances the image as a whole, has limited effect on enhancing faces within the image. Face enhancement is crucial to the perception and visual quality of an image, so this approach is less effective in improving the visual quality of the image. While the second approach enhances faces individually, it disrupts the integrity of the image by extracting them and then stitching and fusing them together. The pixels at the stitching point don't align well, which also affects the display quality of the enhanced image.

[0056] Because real-world image enhancement is a blind problem, the degradation factors that lead to low-quality images are unclear, and thus enhancing low-quality images presents significant challenges. Based on this, for images primarily consisting of people, the disclosed embodiments obtain specific prior information about faces and apply it to subsequent image enhancement processes, achieving superior enhancement results while preserving the integrity of the enhanced image.

[0057] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0058] Application Scenario Overview

[0059] First reference Figure 1 An applicable application scenario of the present disclosure is introduced.

[0060] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure, such as Figure 1 As shown, the client 11 and the server 12 are included, and the client 11 and the server 12 are connected via wired or wireless communication.

[0061] The client 11 can be used to produce images and send the images to the server 12, and the server 12 can process the received images. In the embodiment of the present disclosure, it involves a training process for an image enhancement model and an image processing process based on the image enhancement model. During the training process of the image enhancement model, the client 11 can send training samples to the server 12, and the training samples include images for model training. The server 12 trains the image enhancement model based on the training samples. During the image processing process based on the image enhancement model, the client 11 can send low-quality images to the server 12, and the server 12 enhances the low-quality images according to the image enhancement model to obtain corresponding high-quality images, and so on.

[0062] In some embodiments, the client 11 and server 12 may be two independent devices. In other embodiments, the functions of the client 11 and server 12 may be integrated into a single device. Furthermore, the device that performs the image enhancement model training process and the device that performs the image processing process based on the image enhancement model may be the same device or different devices.

[0063] Exemplary Methods

[0064] The following combination Figure 1 For application scenarios, refer to Figure 2To describe the model training method according to the exemplary embodiment of the present disclosure. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0065] Figure 2 A flow chart of the model training method provided in the embodiment of the present disclosure is shown as follows: Figure 2 As shown, the method may include:

[0066] S21, obtaining multiple sets of training samples, where the training samples include sample images to be processed, sample face parsed images, and label enhanced images.

[0067] The execution entity in the embodiments of the present disclosure can be a server or a device with certain computing and data processing capabilities. Multiple sets of training samples are used to process the image enhancement model. Each set of training samples may include a sample image to be processed, a sample face parsed image, and a label-enhanced image. The sample image to be processed is a low-quality image in the training sample, equivalent to the sample data in the training sample. The label-enhanced image is a high-quality image corresponding to the low-quality image, equivalent to the label corresponding to the sample data in the training sample. The sample face parsed image is the prior information of the face in the sample image to be processed.

[0068] S22, inputting the sample image to be processed and the sample face parsed image into the image enhancement model to obtain the sample enhanced image output by the image enhancement model.

[0069] After obtaining the training samples, for any group of training samples, the sample images to be processed and the sample face parsed images in the group of training samples can be input into the image enhancement model. The image enhancement model processes the sample images to be processed and the sample face parsed images and outputs the sample enhanced images.

[0070] Since the image enhancement model is used to perform image enhancement processing, the enhancement processing is to enhance the low-quality sample image to be processed to obtain a corresponding high-quality sample enhanced image.

[0071] S23, adjusting the parameters of the image enhancement model according to the sample enhanced image and the label enhanced image.

[0072] After obtaining the sample enhanced image output by the image enhancement model, the parameters of the image enhancement model can be adjusted according to the difference between the sample enhanced image and the label enhanced image.

[0073] For any set of training samples, the parameters of the image enhancement model can be adjusted according to the schemes of S22 and S23 above. The image enhancement model is trained with multiple sets of training samples until the difference between the sample-enhanced image and the label-enhanced image meets certain conditions, thereby obtaining a trained image enhancement model.

[0074] In the above embodiment, the image enhancement model is trained based on multiple sets of training samples. Figure 3 This paper introduces the image processing process of the image enhancement model.

[0075] Figure 3 A flow chart of the image processing method provided by the embodiment of the present disclosure is shown as follows: Figure 3 As shown, the method may include:

[0076] S31, performing face analysis processing on the image to be processed to obtain a corresponding face analysis image.

[0077] The image to be processed is a low-quality image that needs to be enhanced. The image enhancement model can enhance the image to be processed based on the facial prior information. Therefore, it is necessary to first obtain the facial prior information through face analysis, that is, the facial analysis image corresponding to the image to be processed.

[0078] The face parsing process first performs face detection on the image to be processed to obtain the face area in the image to be processed, and then parses the face area in the image to be processed to obtain the corresponding face parsing image. The face prior information can be reflected by the pixel values ​​of each pixel point on the face parsing image.

[0079] S32: Input the image to be processed and the face parsed image into the image enhancement model to obtain an enhanced image corresponding to the image to be processed.

[0080] After obtaining the face parsing image, the image to be processed and the face parsing image can be input into the image enhancement model. The image enhancement model is Figure 2 The trained model in the embodiment, the image enhancement model, has the ability to enhance the image to be processed based on the face analysis image to obtain a corresponding enhanced image. By processing the image to be processed and the face analysis image with the image enhancement model, a corresponding enhanced image can be obtained.

[0081] The model training method provided by the embodiment of the present disclosure first obtains multiple groups of training samples, which include sample images to be processed, sample face parsed images and label enhanced images. For any group of training samples, the sample images to be processed and the sample face parsed images can be input into the image enhancement model to obtain the sample enhanced images output by the image enhancement model, so that the parameters of the image enhancement model can be adjusted according to the sample enhanced images and the label enhanced images.

[0082] The image enhancement model can be trained using the above method for any set of training samples. After the termination condition of the image enhancement model training is met, the training process is terminated to obtain a trained image enhancement model. The termination condition of the image enhancement model training may include: the loss value between the sample enhanced image and the label enhanced image output by the image enhancement model is less than a preset loss value, or the number of training rounds reaches a preset number. For example, after the sample enhanced image is output by the image enhancement model, a corresponding loss value can be calculated based on the sample enhanced image and the label enhanced image, and then the loss value is compared with the preset loss value. When the loss value is greater than or equal to the preset loss value, it indicates that the image enhancement model has not been trained and needs to continue to the next round of training; when the loss value is less than the preset loss value, it indicates that the termination condition is met and the image enhancement model training is completed. For example, when the number of training rounds of the image enhancement model reaches a preset number, it indicates that the termination condition is met and the image enhancement model training is completed.

[0083] After the image enhancement model is trained, it has the ability to enhance images. After the image to be processed and the corresponding face parsed image are input into the image enhancement model, the image enhancement model can output the corresponding enhanced image. Because sample face parsed images are included in the training process of the image enhancement model, and the sample face parsed images include the facial prior information of the sample image to be processed, the solution of the embodiment of the present disclosure can significantly enhance images containing faces without the need for cropping or splicing the images, and also ensures the integrity and completeness of the images, making the face enhancement effect and background enhancement effect in the image more coordinated.

[0084] Based on the above embodiments, the image enhancement model in the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0085] Figure 4 A schematic diagram of the structure of the image enhancement model provided in the embodiment of the present disclosure is shown in FIG. Figure 4 As shown in FIG, the image enhancement model includes a feature pyramid module, a face structure embedding module, an image reconstruction module, and a global residual fine-tuning and inter-frame transfer module. Different modules are used to perform different image processing. At the same time, in the process of processing the low-quality image 41 through the image enhancement model, it is also necessary to cooperate with the face parsing module to perform face parsing on the low-quality image 41 to obtain the corresponding face parsing image 42, which is used to assist the image enhancement model in processing the low-quality image 41. Figure 4 Each module is introduced separately.

[0086] Face parsing module

[0087] The face parsing module is used to analyze the pixels in the facial regions of low-quality image 41 and determine the category to which each pixel in low-quality image 41 belongs. The face parsing module processes low-quality image 41 in two main steps: first, performing face detection on low-quality image 41 to obtain all facial regions in low-quality image 41; and second, parsing the facial regions to obtain a face parsed image 42.

[0088] The face parsing module outputs a three-dimensional matrix of size (C, H, W), namely, the face parsing image 42, where W and H are the width and height of the low-quality image 41, and C represents the number of categories into which the low-quality image 41 needs to be classified. For example, in a face scene, pixels can be classified into 11 categories, including background, facial skin, left eyebrow, right eyebrow, left eye, right eye, nose, upper lip, mouth, lower lip, and hair. Each category corresponds to a matrix of size (H, W), where 1 indicates that the pixel belongs to that category and 0 indicates that the pixel does not belong to that category. The matrices of all categories are combined to form the face parsing image 42. To reduce the amount of computation, the above parsing process can be performed only on the face area, and pixels in non-face areas are classified as background.

[0089] The functions of the face parsing module can be implemented using a pre-trained face parsing model. For example, multiple sets of face parsing samples can be first obtained. These face parsing samples include sample images and annotation information, which identifies the category of each pixel in the sample images. The face parsing model is then trained using these multiple sets of face parsing samples. Training is terminated when the training termination condition for the face parsing model is met, resulting in a trained face parsing model. For example, the training termination condition can be when the loss value between the sample face parsing image output by the face parsing model and the annotation information is less than or equal to a preset loss value, or when the number of training rounds reaches a preset number, etc. After the face parsing model is trained, the trained face parsing model possesses face parsing capabilities. At this point, the low-quality image 41 is input into the face parsing model to obtain the corresponding face parsing image 42. The face parsing image 42 includes the category information of each pixel in the low-quality image 41. Based on this category information, the category of each pixel in the low-quality image 41 can be determined.

[0090] Feature Pyramid Module

[0091] The feature pyramid module consists of a convolution layer and multiple feature extraction layers. Figure 4 For example, from top to bottom, it includes convolution layer 1, feature extraction layer 1, feature extraction layer 2, feature extraction layer 3, and feature extraction layer 4. Each feature extraction layer extracts the key features of the image and performs downsampling to aggregate the extracted features.

[0092] Each feature extraction layer includes a convolution module and a downsampling module. The convolution module is a module that uses the convolution method to extract features by combining multiple convolution layers, normalization layers, and activation layers. The results of the convolution module can be designed and combined by yourself or implemented using a residual network. The core convolution calculation formula is as follows:

[0093] conv(F;W) i =∑ j∈Ω(i) w Δij f j ,

[0094] i and j represent spatial coordinates, F represents the feature map passed in from the previous layer, W represents the convolution kernel, Ω(i) represents the area around i, and w Δij is the parameter value in the convolution kernel W, f j is the pixel value of the pixel in the feature map F.

[0095] The downsampling module can be implemented using a pooling layer or a convolutional layer with a stride of 2. The downsampling module reduces the size of the input features by half. Therefore, the features extracted by the feature extraction layer will be halved in size as the number of layers increases, resulting in a feature pyramid.

[0096] In some embodiments, the feature pyramid module as a whole can be represented by a pre-trained backbone network. The backbone network may include, for example, MobileNet, Inception-ResNet-v2, etc. These backbone networks will be pre-trained with large image data sets to achieve better feature extraction and improve network effectiveness and training efficiency.

[0097] exist Figure 4 In the example, the feature pyramid module includes a convolution layer and four feature extraction layers. The low-quality image 41 is input into the convolution layer 1 to obtain image F1, image F1 is input into the feature extraction layer 1 to obtain image F2, image F2 is input into the feature extraction layer 2 to obtain image F3, image F3 is input into the feature extraction layer 3 to obtain image F4, and image F4 is input into the feature extraction layer 4 to obtain image F5. The size of image F1 is the same as that of the low-quality image 41, and the sizes of image F2, image F3, image F4 and image F5 are respectively one-half, one-quarter, one-eighth and one-sixteenth of the low-quality image 41.

[0098] Image reconstruction module

[0099] The image reconstruction module consists of multiple reconstruction layers and a convolution layer. Figure 4For example, from bottom to top, there are reconstruction layer 4, reconstruction layer 3, reconstruction layer 2, reconstruction layer 1, reconstruction layer 0, and convolution layer 2. Each reconstruction layer contains a convolution module and an upsampling module. The upsampling module completes the upscaling by a factor of two, and the convolution module completes the mapping of features to enhanced images. The upsampling module is generally implemented using interpolation-based upsampling or deconvolution.

[0100] The lowest reconstruction layer of the image reconstruction module takes the output of the lowest layer of the feature pyramid module as input. Each subsequent reconstruction layer takes the parallel output of the next reconstruction layer and the output of the corresponding feature pyramid layer as the input of the convolution module. After the convolution module outputs the reconstructed features, it is then compared with the face parsing of the same size. Figure 1 The facial structure information is used to modulate the current reconstruction features, resulting in reconstructed features embedded with facial structure information. The features are then passed to the next reconstruction layer. The top layer of the image reconstruction module uses a convolutional layer to map the reconstructed features back to the image, resulting in a reconstructed image.

[0101] For example, Figure 4 In the figure, image F5 is the input of reconstruction layer 4, which upsamples image F5 to obtain image C5, whose size is one-eighth of the low-quality image 41. Then, image C5 and image F4 are input to reconstruction layer 3, which convolves and upsamples image C5 and image F4 to obtain image C4 as output from reconstruction layer 3. Image C4 is passed to the face structure embedding module for processing, outputting image E4. Image E4 and image F3 are input to reconstruction layer 2, which convolves and upsamples image E4 and image F3 to obtain image C3 as output from reconstruction layer 2. Image C3 is passed to the face structure embedding module for processing, outputting image E3. Image E3 and image F2 are the inputs of reconstruction layer 1, and so on.

[0102] Each reconstruction layer performs the above process until reconstruction layer 0 outputs image C1. Image C1 is then input into the face structure embedding module to obtain image E1, which is then processed by convolution layer 2 and mapped back to reconstructed image 43. In the above example, images C1, C2, and C3 are upsampled after being input into the face structure embedding module, resulting in sizes that are one, half, and one-quarter the size of the low-quality image 41, respectively.

[0103] Face structure embedding module

[0104] Figure 5 This is a schematic diagram of the structure of the face structure embedding module provided in the embodiment of the present disclosure. Figure 5 The dotted box in the figure shows the processing of the face structure embedding module. The reconstruction feature refers to the output of the reconstruction layer in the image reconstruction module, such as Figure 4In the example, images C4, C3, C2, and C1 are all reconstructed features. For reconstructed features, the face structure embedding module first uses a convolutional layer to perform feature mapping, and then uses a normalization layer to perform normalization. The implementation formula for the normalization process is as follows:

[0105]

[0106]

[0107] Where H and W represent the height and width of the input feature map (i.e., the image after convolution of the reconstructed features), i and j represent plane coordinates, c represents the channel, x represents a point in the input feature map, μ represents the mean, σ represents the variance, ∈ is a fixed parameter, ∈ is usually set to a minimum value to prevent the denominator in the implementation formula of the normalization process from being 0, and y cij are the standardized features.

[0108] For facial parsing images, the facial structure embedding module first scales them to the same size as the input reconstructed features. A shared convolutional layer is then used for feature extraction and mapping. Finally, two convolutional layers are used to generate γ and β, which have the same width, height, and number of channels as the input reconstructed features. γ and β are then used to modulate the normalized features according to the following formula, achieving adaptive normalization based on the portrait structure and achieving facial structure embedding.

[0109] f cij =y cij ,y cij +β

[0110] The feature embedded with the facial structure is then subjected to a Relu activation layer, a convolutional layer, and finally added to the input of the entire module to obtain the output. The reconstructed feature E of the facial structure embedding is obtained.

[0111] In this embodiment, image C4 and face analysis image 42 are used as inputs of the face structure embedding module to obtain image E4 output by the face structure embedding module, and the size of image E4 is the same as that of image C4; image C3 and face analysis image 42 are used as inputs of the face structure embedding module to obtain image E3 output by the face structure embedding module, and the size of image E3 is the same as that of image C3; image C2 and face analysis image 42 are used as inputs of the face structure embedding module to obtain image E2 output by the face structure embedding module, and the size of image E2 is the same as that of image C2; ​​image C1 and face analysis image 42 are used as inputs of the face structure embedding module to obtain image E1 output by the face structure embedding module, and the size of image E1 is the same as that of image C1.

[0112] Global residual fine-tuning and inter-frame transfer modules

[0113] The global residual fine-tuning and inter-frame transfer module integrates the underlying low-resolution feature maps from the image reconstruction module to calculate the global residual of the generated image, which is then applied to the reconstructed image output by the image reconstruction module. This module firstly uses multi-scale underlying feature fusion to extract residual features from the enhanced image at a global level, enhancing image detail and reducing edge artifacts. Secondly, the underlying feature resolution is smaller than that of the original low-quality image, making it less sensitive to motion and suitable for feature transfer in video scenarios.

[0114] Specifically, Figure 4 For example, the image E4 output by the face structure embedding module is upsampled four times to obtain image E up4 ; Upsample the image E3 output by the face structure embedding module by two times to obtain image E up3 ; Image E up4 、Image E up4 After being connected in parallel with image E2, it is input into a convolution module to obtain the residual feature D with the same dimension as E2. t .

[0115] If the image enhancement model only processes images, it can be used to t An upsampling module and a convolution module are used to obtain the global image residual, and the global image residual is summed with the reconstructed image output by the image reconstruction module to obtain the final enhanced image 45.

[0116] In video scenarios, feature transfer between frames can provide more useful information for video enhancement, while also improving inter-frame coherence and avoiding jitter. It is generally believed that low-scale inter-frame images have less pixel movement and are more suitable for inter-frame fusion. The scale of an image can be understood as the size of the image (i.e., the number of pixels included in the height and width of the image), and low-scale images are smaller images. For example, image A is downsampled once to obtain image B, and image A is downsampled twice to obtain image C. The size of image B is half that of image A, and the size of image C is one-quarter that of image A. Images B and C are two images of different scales obtained based on image A.

[0117] Figure 6 A schematic diagram of inter-frame transmission provided by an embodiment of the present disclosure, such as Figure 6 As shown, the residual feature D of the previous frame t-1 The residual feature D of the current frame tMerge the input into a convolution module and use a sigmoid activation function as the output. According to the output of the activation function, the residual features of the previous frame and the residual features of the current frame are fused to obtain D', and the face structure embedding module is used to embed the face information into the fused features. Figure 4 As shown, the upsampling module and the convolution module are used to obtain the global residual of the current frame, and the global image residual 44 is summed with the reconstructed image 43 output by the image reconstruction module to obtain the final enhanced image 45. The sigmoid function and the fusion process are shown in the following formula, where S x It represents the result of the sigmoid calculation of the value x, and S represents the result of the feature output after the sigmoid function.

[0118]

[0119] D′=D t-1 ×S+(1-S)×D t

[0120] exist Figure 4-Figure 6 In the example, the input to the image enhancement model is a low-quality image and the corresponding face parsed image, and the output is a high-quality image corresponding to the low-quality image. Since the embodiments of the present disclosure involve the training of the image enhancement model and the application processing of the image enhancement model, the structure of the image enhancement model is the same during the training and application processing, and only the parameters in the image enhancement model are different. In the following embodiments, based on Figure 4-Figure 6 The structure of the image enhancement model of the example introduces the process of model training and application processing respectively.

[0121] First combine Figure 7 The training process of the image enhancement model is introduced. During the model training process, the low-quality image input to the model is the sample image to be processed (equivalent to Figure 4 The low-quality image 41 in the face parsing module is output as a sample face parsing image (equivalent to Figure 442); the images output by each layer of the feature pyramid module are sample feature images (image F1 is the first sample feature image, image F2 is the second sample feature image, image F3 is the third sample feature image, image F4 is the fourth sample feature image, and image F5 is the fifth sample feature image); the images output by each reconstruction layer of the image reconstruction module are sample up-sampled images (image C5 is the fifth sample up-sampled image, image C4 is the fourth sample up-sampled image, image C3 is the third sample up-sampled image, image C2 is the second sample up-sampled image, and image C1 is the first sample up-sampled image); the images output by each layer of the face structure embedding module are sample reconstructed feature images (image E4 is the fourth sample reconstructed feature image, image E3 is the third sample reconstructed feature image, image E2 is the second sample reconstructed feature image, and image E1 is the first sample reconstructed feature image).

[0122] Before model training, multiple sets of training samples need to be obtained. The training samples include sample images to be processed and label-enhanced images, as well as sample face parsed images of the sample images to be processed. The sample images to be processed are low-quality images, and the label-enhanced images are the corresponding high-quality images.

[0123] Since it is difficult to obtain paired low-quality images and high-quality images, while it is easier to obtain a single high-quality image, the embodiment of the present disclosure may adopt a method of degrading high-quality images to obtain paired training data.

[0124] Specifically, we first obtain a label-enhanced image, then parse it using the face parsing model to obtain a sample face parsed image. Finally, we degrade the label-enhanced image to obtain the corresponding sample image to be processed.

[0125] To simulate real-world data distribution, when constructing training samples, different degradation factors can be combined to create low-quality images based on the business scenario. Specifically, high-quality images from various scenarios can be collected as label-enhanced images. These label-enhanced images can include both images containing faces and images excluding faces. The ratio of images containing faces to images excluding faces can be set as needed, for example, to 2:1. By including images excluding faces, the image enhancement model can maintain enhanced performance even when processing images excluding faces.

[0126] Then, a degradation operator set is constructed, which includes multiple preset degradation operators, such as random Gaussian noise, random Poisson noise, random Gaussian blur, random motion blur, random instruction JPEG compression, random image scaling, etc. The preset degradation operators are used to perform degradation processing.

[0127] After constructing the set of degradation operators, different preset probability values ​​can be assigned to different preset degradation operators according to the task scenario. Then, in each iteration, different preset degradation operators are randomly selected to act on the label enhanced image to obtain the corresponding sample image to be processed.

[0128] Furthermore, after obtaining the corresponding sample image to be processed and sample face analysis image based on the label enhanced image, the label enhanced image, the sample image to be processed and the sample face analysis image can be simultaneously subjected to random cropping, rotation and mirroring to obtain new training samples, thereby expanding the training sample set.

[0129] Figure 7 A flow chart of the image enhancement model training process provided by the embodiment of the present disclosure is shown as follows: Figure 7 Shown, including:

[0130] S71 , performing feature extraction processing on the sample image to be processed to obtain R sample feature images, where R is a positive integer greater than or equal to 2.

[0131] Feature extraction processing is the process in which the feature extraction module in the image enhancement model processes the sample image to be processed. The feature extraction module includes a convolution layer and R-1 feature extraction layers, which are used to perform feature extraction processing on the input image.

[0132] Specifically, after the sample image to be processed is input into the image enhancement model, the convolution layer first performs convolution processing on the sample image to be processed to obtain the first sample feature image output by the convolution layer.

[0133] Then, starting from the first sample feature image, the i-th sample feature image is sequentially convolved and downsampled to obtain the i+1-th sample feature image, where i is 1, 2, ..., R-1, until R sample feature images are obtained. Figure 4 In the figure, taking R=5 as an example, after the sample image to be processed is input into the image enhancement model, the convolution layer 1 performs convolution processing on the sample image to be processed to obtain the first sample feature image F1; the feature extraction layer 1 performs convolution processing and downsampling processing on the first sample feature image F1 to obtain the second sample feature image F2; the feature extraction layer 2 performs convolution processing and downsampling processing on the second sample feature image F2 to obtain the third sample feature image F3; the feature extraction layer 3 performs convolution processing and downsampling processing on the third sample feature image F3 to obtain the fourth sample feature image F4; the feature extraction layer 4 performs convolution processing and downsampling processing on the fourth sample feature image F4 to obtain the fifth sample feature image F5.

[0134] Since the feature extraction process based on the ith sample feature image includes downsampling, the downsampling process will change the number of pixels in the ith sample feature image. For example, if the number of pixels in the width of the sample image to be processed is M and the number of pixels in the height is N, the size of the ith sample feature image is That is, starting from the first sample feature image, after each convolution and downsampling process, the size of the obtained sample feature image will be reduced by half (that is, the number of pixels included in the width and height is half of that before the process).

[0135] S72, performing image reconstruction processing based on the R sample feature images and the sample face analysis image to obtain R-1 sample reconstructed feature images.

[0136] Image reconstruction processing is Figure 4 The image reconstruction module of the example performs processing, and the image reconstruction module includes R reconstruction layers and a convolution layer. Each reconstruction layer includes a convolution module and an upsampling module. The upsampling module completes the up-amplification by two times, and the upsampling module performs upsampling processing. The upsampling processing can be implemented, for example, by interpolation-based upsampling or deconvolution. The convolution module completes the mapping of features to enhanced images, and the convolution module performs convolution processing.

[0137] First, the R-1th sample reconstructed feature image can be obtained based on the Rth sample feature image, the R-1th sample feature image and the sample face analysis image, where the Rth sample feature image is the lowest feature extraction layer of the feature pyramid module (such as Figure 4 The image output by the feature extraction layer 4 in the image, the R-1th sample feature image is the second to last feature extraction layer (such as Figure 4 The image output by the feature extraction layer 3).

[0138] The Rth sample feature image is the lowest reconstruction layer in the image reconstruction module (such as Figure 4 After the Rth sample feature image is input to the reconstruction layer of the lowest layer, the Rth sample feature image is upsampled according to the reconstruction layer of the lowest layer to obtain the Rth sample upsampled image. Figure 4 For example, after image F5 is input to the reconstruction layer 4, the reconstruction layer 4 outputs image C5, which is the R-th sample upsampled image.

[0139] Then, the Rth sample upsampled image and the R-1th feature image are convolved and upsampled to obtain the R-1th sample upsampled image. Figure 4As shown, after image F4 and image C5 are input to the reconstruction layer 3, the reconstruction layer 3 outputs image C4, which is the R-1th sample up-sampled image.

[0140] After obtaining the R-1th sample upsampled image, convolution processing can be performed on the R-1th sample upsampled image and the sample face parsing image to obtain the R-1th sample reconstructed feature image. Specifically, firstly, upsample the Rth sample feature image to obtain the Rth sample upsampled image, then convolve and upsample the Rth sample upsampled image and the R-1th feature image to obtain the R-1th sample upsampled image, and finally, convolution processing is performed on the R-1th sample upsampled image and the sample face parsing image to obtain the R-1th sample reconstructed feature image. Figure 4 As shown, the image C4 output by the reconstruction layer and the face parsing image are input into the face structure embedding module to obtain the image E4 output by the face structure embedding module. The image E4 is the fourth sample reconstructed feature image.

[0141] After obtaining the R-1th sample reconstructed feature image, starting from the R-1th sample reconstructed feature image, the j-1th sample reconstructed feature image is obtained in sequence based on the jth sample reconstructed feature image, the j-1th sample feature image and the sample face parsed image, where j is R-1, R-2, ..., 2, until the R-1th sample reconstructed feature image is obtained.

[0142] Regarding the process of obtaining the j-1th sample up-sampled image based on any j-th sample reconstructed feature image and the j-1th sample feature image, first, convolution processing and up-sampling processing can be performed on the j-th sample reconstructed feature image and the j-1th sample feature image to obtain the j-1th sample up-sampled image.

[0143] After obtaining the j-1th sample up-sampled image, the j-1th sample reconstructed feature image is obtained based on the j-1th sample up-sampled image and the sample face parsing image. Specifically, the sample face parsing image is first down-sampled and convolved to obtain a first sample face parsing image and a second sample face parsing image. The sizes of the first sample face parsing image and the second sample face parsing image are the same as the size of the j-1th sample up-sampled image. Through down-sampling, the sizes of the processed first sample face parsing image and the second sample face parsing image are the same as the size of the j-1th sample up-sampled image. Then, the j-1th sample reconstructed feature image is convolved and normalized to obtain the j-1th normalized sample reconstructed feature image. The process of convolution and normalization can be found in [1]. Figure 5After obtaining the j-1th standardized sample reconstructed feature image, the j-1th sample reconstructed feature image can be obtained based on the j-1th standardized sample reconstructed feature image, the first sample face parsing image, and the second sample face parsing image.

[0144] S73, reconstructing a feature image based on R-1 samples to obtain a sample enhanced image.

[0145] After obtaining R-1 sample reconstructed feature images, the sample residual features of the sample to be processed can be first obtained based on the R-1 sample reconstructed feature images. Specifically, starting from the R-1 sample reconstructed feature image, the k-th sample reconstructed feature image is sequentially upsampled to obtain the corresponding sample upsampled reconstructed feature image, where k is R-1, R-2, ..., 3, respectively. The size of the sample upsampled reconstructed feature image obtained after the upsampling process is the same as the size of the second sample reconstructed feature image.

[0146] Then, the second sample reconstructed feature image and the upsampled reconstructed feature images of each sample are processed in parallel to obtain the sample parallel reconstructed feature image. The sample parallel reconstructed feature image is convolved to obtain the sample residual feature.

[0147] If the image to be processed is the t-th frame image in the sample video image, the sample residual features can also be updated. Specifically, first obtain the residual features of the t-1th frame image in the sample video image, then obtain the sample video inter-frame parameters, which are preset values, and finally update the sample residual features based on the residual features of the t-1th frame image and the sample video inter-frame parameters. The specific calculation process for updating the sample residual features can be found in Figure 6 The embodiments will not be described in detail here.

[0148] After obtaining the sample residual features, the sample residual features are convolved to obtain the sample convolution residual features, which are equivalent to Figure 4 Then, the sample enhanced image is obtained by reconstructing the feature image based on the sample convolution residual feature and the first sample.

[0149] During the training process, the sample images to be processed and the sample face parsed images are processed by the image enhancement model. After the sample enhanced images are output, the parameters of the image enhancement model need to be adjusted according to the sample enhanced images and the label enhanced images. Figure 8 A sample loss calculation diagram provided in an embodiment of the present disclosure is shown in FIG. Figure 8As shown, first, a sample loss value is obtained between the sample enhanced image and the label enhanced image. In the disclosed embodiment, the sample loss value includes at least one of a sample global loss value, a sample face loss value, a sample eye loss value, and a sample mouth and nose loss value. Then, based on at least one of the sample global loss value, the sample face loss value, the sample eye loss value, and the sample mouth and nose loss value, and the sample loss value weight, a sample loss value is obtained.

[0150] The content loss value is the sample enhancement image Image using the training sample HR and label enhanced image Image g Calculate the L1 loss and the calculation formula is as follows:

[0151] L recon =||Image HR -Image g ||1.

[0152] The perceptual loss value uses the pre-trained VGG network as the feature extractor, and performs HR and Image g Input into the VGG network, extract features and calculate the L1 loss. φ represents the VGG feature extractor. The calculation formula is as follows:

[0153] L percp =||φ(Image HR )-φ(Image g )||1.

[0154] The sample global loss value, sample face loss value, sample eye loss value, and sample mouth and nose loss value all represent the calculation of the discriminator loss in the generative adversarial network. In this scheme, four discriminators are constructed, namely:

[0155] Global discriminator. Taking the sample enhanced image and the label enhanced image as input, it makes a global judgment and calculates the sample global loss value recorded as

[0156] Portrait discriminator. Based on the sample face parsing image, the portrait area is cropped from the sample enhanced image and the label enhanced image, respectively, and input into the portrait discriminator. The sample face loss value is calculated and recorded as

[0157] Eye discriminator. Based on the sample face parsing image, the eye region is cropped from the sample enhanced image and the label enhanced image respectively to calculate the sample eye loss value, which is recorded as

[0158] Mouth and nose discriminator. According to the sample face parsing image, the mouth and nose regions are cropped from the sample enhanced image and the label enhanced image respectively to calculate the sample mouth and nose loss value, which is recorded as

[0159] Finally, for L recon , L percp 、 and Perform weighted summation and construct a complete loss function for training until the generated enhanced image meets the requirements.

[0160] In the above embodiment, the training process of the image enhancement model is introduced. Figure 9 Introduce the image processing process. In the image processing process, the low-quality image input to the model is the image to be processed (equivalent to Figure 4 The low-quality image 41 in the face parsing module is output as a face parsing image (equivalent to Figure 4 The face parsed image 42 in the image is output); the images output by each layer of the feature pyramid module are feature images (image F1 is the first feature image, image F2 is the second feature image, image F3 is the third feature image, image F4 is the fourth feature image, and image F5 is the fifth feature image); the images output by each reconstruction layer of the image reconstruction module are upsampled images (image C5 is the fifth upsampled image, image C4 is the fourth upsampled image, image C3 is the third upsampled image, image C2 is the second upsampled image, and image C1 is the first upsampled image); the images output by each layer of the face structure embedding module are reconstructed feature images (image E4 is the fourth reconstructed feature image, image E3 is the third reconstructed feature image, image E2 is the second reconstructed feature image, and image E1 is the first reconstructed feature image).

[0161] Figure 9 A schematic diagram of the image processing process to be processed provided in the embodiment of the present disclosure is shown in FIG. Figure 9 Shown, including:

[0162] S91, performing feature extraction processing on the image to be processed to obtain R feature images, where R is a positive integer greater than or equal to 2.

[0163] Feature extraction processing is the process of processing the image to be processed by the feature extraction module in the image enhancement model. The feature extraction module includes a convolution layer and R-1 feature extraction layers, which are used to perform feature extraction processing on the input image.

[0164] Specifically, after the image to be processed is input into the image enhancement model, the convolution layer first performs convolution processing on the image to be processed to obtain the first feature image output by the convolution layer.

[0165] Then, starting from the first feature image, the i-th feature image is sequentially convolved and downsampled to obtain the i+1-th feature image, where i is 1, 2, ..., R-1, until R feature images are obtained. Figure 4 In the example, taking R=5 as an example, after the image to be processed is input into the image enhancement model, the convolution layer 1 performs convolution processing on the image to be processed to obtain the first feature image F1; the feature extraction layer 1 performs convolution processing and downsampling processing on the first feature image F1 to obtain the second feature image F2; the feature extraction layer 2 performs convolution processing and downsampling processing on the second feature image F2 to obtain the third feature image F3; the feature extraction layer 3 performs convolution processing and downsampling processing on the third feature image F3 to obtain the fourth feature image F4; the feature extraction layer 4 performs convolution processing and downsampling processing on the fourth feature image F4 to obtain the fifth feature image F5.

[0166] Since the feature extraction process based on the i-th feature image includes downsampling, the downsampling process will change the number of pixels in the i-th feature image. For example, if the number of pixels in the width of the image to be processed is M and the number of pixels in the height is N, the size of the i-th feature image is That is, starting from the first feature image, after each convolution and downsampling process, the size of the obtained feature image will be reduced by half (that is, the number of pixels included in the width and height is half of that before the process).

[0167] S92: Perform image reconstruction processing based on the R feature images and the face analysis image to obtain R-1 reconstructed feature images.

[0168] Image reconstruction processing is Figure 4 The image reconstruction module of the example performs processing, and the image reconstruction module includes R reconstruction layers and a convolution layer. Each reconstruction layer includes a convolution module and an upsampling module. The upsampling module completes the up-amplification by two times, and the upsampling module performs upsampling processing. The upsampling processing can be implemented, for example, by interpolation-based upsampling or deconvolution. The convolution module completes the mapping of features to enhanced images, and the convolution module performs convolution processing.

[0169] First, the R-1th reconstructed feature image can be obtained based on the R-th feature image, the R-1th feature image and the face analysis image, where the R-th feature image is the lowest feature extraction layer of the feature pyramid module (such as Figure 4 The image output by the feature extraction layer 4 in the image, the R-1th feature image is the second to last feature extraction layer (such as Figure 4 The image output by the feature extraction layer 3).

[0170] The Rth feature image is the lowest reconstruction layer in the image reconstruction module (such as Figure 4 After the Rth feature image is input to the reconstruction layer of the lowest layer, the Rth feature image is upsampled according to the reconstruction layer of the lowest layer to obtain the Rth upsampled image. Figure 4 For example, after image F5 is input to the reconstruction layer 4, the reconstruction layer 4 outputs image C5, and image C5 is the Rth up-sampled image.

[0171] Then, the Rth up-sampled image and the R-1th feature image are convolved and up-sampled to obtain the R-1th up-sampled image. Figure 4 As shown, after image F4 and image C5 are input to the reconstruction layer 3, the reconstruction layer 3 outputs image C4, which is the R-1th up-sampled image.

[0172] After obtaining the R-1th upsampled image, convolution processing can be performed on the R-1th upsampled image and the face parsing image to obtain the R-1th reconstructed feature image. Specifically, firstly, upsample the Rth feature image to obtain the Rth upsampled image, then convolve and upsample the Rth upsampled image and the R-1th feature image to obtain the R-1th upsampled image, and finally, convolve the R-1th upsampled image and the face parsing image to obtain the R-1th reconstructed feature image. Figure 4 As shown, the image C4 output by the reconstruction layer and the face parsing image are input into the face structure embedding module to obtain the image E4 output by the face structure embedding module. Image E4 is the fourth reconstructed feature image.

[0173] After obtaining the R-1th reconstructed feature image, starting from the R-1th reconstructed feature image, the j-1th reconstructed feature image is obtained according to the jth reconstructed feature image, the j-1th feature image and the face parsed image, where j is R-1, R-2, ..., 2, until the R-1th reconstructed feature image is obtained.

[0174] Regarding the process of obtaining the j-1th up-sampled image based on any j-th reconstructed feature image and j-1th feature image, first, convolution processing and up-sampling processing can be performed on the j-th reconstructed feature image and the j-1th feature image to obtain the j-1th up-sampled image.

[0175] After obtaining the j-1th up-sampled image, the j-1th reconstructed feature image is obtained based on the j-1th up-sampled image and the face parsing image. Specifically, the face parsing image is first down-sampled and convolved to obtain a first face parsing image and a second face parsing image. The sizes of the first face parsing image and the second face parsing image are the same as the size of the j-1th up-sampled image. Through down-sampling, the sizes of the processed first face parsing image and the second face parsing image are the same as the size of the j-1th up-sampled image. Then, the j-1th reconstructed feature image is convolved and normalized to obtain the j-1th normalized reconstructed feature image. The process of convolution and normalization can be found in Figure 5 After obtaining the j-1th standardized reconstructed feature image, the j-1th reconstructed feature image can be obtained according to the j-1th standardized reconstructed feature image, the first face parsing image and the second face parsing image.

[0176] S93: Obtain an enhanced image based on the R-1 reconstructed feature images.

[0177] After obtaining R-1 reconstructed feature images, the residual features of the image to be processed can be first obtained based on the R-1 reconstructed feature images. Specifically, starting from the R-1th reconstructed feature image, the kth reconstructed feature image is sequentially upsampled to obtain the corresponding upsampled reconstructed feature image, where k is R-1, R-2, ..., 3. The size of the upsampled reconstructed feature image obtained after the upsampling process is the same as the size of the second reconstructed feature image.

[0178] Then, the second reconstructed feature image and each upsampled reconstructed feature image are processed in parallel to obtain a parallel reconstructed feature image. The parallel reconstructed feature image is convolved to obtain a residual feature.

[0179] If the image to be processed is the t-th frame image in the video image, the residual features can also be updated. Specifically, the residual features of the t-1th frame image in the video image are first obtained, and then the video frame parameters are obtained. The video frame parameters are preset values. Finally, the residual features are updated according to the residual features of the t-1th frame image and the video frame parameters. The specific calculation process of updating the residual features can be found in Figure 6 The embodiments will not be described in detail here.

[0180] After obtaining the residual features, the residual features are convolved to obtain the convolution residual features, which are equivalent to Figure 4 Then, the enhanced image can be obtained based on the convolution residual features and the first reconstructed feature image.

[0181] The model training and image processing method provided by the embodiment of the present disclosure first obtains multiple groups of training samples, and the training samples include sample images to be processed, sample face parsed images and label enhanced images. For any group of training samples, the sample images to be processed and the sample face parsed images can be input into the image enhancement model to obtain the sample enhanced images output by the image enhancement model, so that the parameters of the image enhancement model can be adjusted according to the sample enhanced images and the label enhanced images. The image enhancement model can be trained in the above manner for any group of training samples, and finally a trained image enhancement model is obtained. After the image enhancement model is trained, the image enhancement model has the ability to enhance the image. After the image to be processed and the corresponding face parsed image are input into the image enhancement model, the image enhancement model can output the corresponding enhanced image. Since the sample face parsed image is added in the training process of the image enhancement model, the sample face parsed image includes the face prior information of the sample image to be processed. In the subsequent image processing process, by inputting the image to be processed and the corresponding face analysis image, the structural information of the face can be embedded into the image processing process based on the face analysis image. This not only takes the face information into consideration during the image processing process, but also avoids the subsequent cropping and fitting of the image, thereby improving the enhancement effect of images containing faces.

[0182] Exemplary media

[0183] After introducing the method of the exemplary embodiment of the present disclosure, next, reference is made to Figure 10 A storage medium according to an exemplary embodiment of the present disclosure is described.

[0184] Figure 10 A schematic diagram of a storage medium provided in an embodiment of the present disclosure, referring to Figure 10 As shown, the storage medium 100 stores a program product for implementing the above method according to an embodiment of the present disclosure. The program product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0185] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0186] A readable signal medium may include a data signal transmitted in baseband or as part of a carrier wave, which carries readable program code. Such a transmitted data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0187] Program code for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0188] Exemplary devices

[0189] After introducing the medium of the exemplary embodiment of the present disclosure, next, reference is made to Figure 11 and Figure 12 The model training device and image processing device of the exemplary embodiments of the present disclosure are described separately, which are used to implement the method in any of the above method embodiments. Their implementation principles and technical effects are similar and will not be repeated here.

[0190] Figure 11 A schematic diagram of the structure of the model training device provided in the embodiment of the present disclosure is shown as follows: Figure 11 As shown, the device includes:

[0191] An acquisition module 111 is configured to acquire multiple sets of training samples, wherein the training samples include sample images to be processed, sample face parsed images, and label-enhanced images;

[0192] A processing module 112 is configured to input the sample image to be processed and the sample face parsed image into an image enhancement model to obtain a sample enhanced image output by the image enhancement model;

[0193] The training module 113 is configured to adjust the parameters of the image enhancement model according to the sample enhanced image and the label enhanced image.

[0194] In a possible implementation, the processing module 112 is specifically configured to:

[0195] Performing feature extraction processing on the sample image to be processed to obtain R sample feature images, where R is a positive integer greater than or equal to 2;

[0196] Perform image reconstruction processing based on the R sample feature images and the sample face analysis image to obtain R-1 sample reconstructed feature images;

[0197] The feature image is reconstructed according to the R-1 samples to obtain the sample enhanced image.

[0198] In a possible implementation, the processing module 112 is specifically configured to:

[0199] Performing convolution processing on the sample image to be processed to obtain a first sample feature image;

[0200] Starting from the first sample feature image, the i-th sample feature image is sequentially convolved and downsampled to obtain the i+1-th sample feature image, where i is 1, 2, ..., R-1, until the R sample feature images are obtained. The size of the i-th sample feature image is The M is the number of pixels in the width of the sample image to be processed, and the N is the number of pixels in the height of the sample image to be processed.

[0201] In a possible implementation, the processing module 112 is specifically configured to:

[0202] Obtaining an R-1th sample reconstructed feature image according to the R-th sample feature image, the R-1th sample feature image, and the sample face parsed image;

[0203] Starting from the R-1th sample reconstructed feature image, the j-1th sample reconstructed feature image is obtained according to the j-th sample reconstructed feature image, the j-1th sample feature image and the sample face parsed image, where j is R-1, R-2, ..., 2 in sequence until the R-1th sample reconstructed feature image is obtained.

[0204] In a possible implementation, the processing module 112 is specifically configured to:

[0205] Performing upsampling processing on the R-th sample feature image to obtain an R-th sample upsampled image;

[0206] Performing convolution processing and upsampling processing on the R-th sample upsampled image and the R-1-th feature image to obtain the R-1-th sample upsampled image;

[0207] Convolution processing is performed on the R-1th sample up-sampled image and the sample face parsed image to obtain the R-1th sample reconstructed feature image.

[0208] In a possible implementation, the processing module 112 is specifically configured to:

[0209] Performing convolution processing and upsampling processing on the j-th sample reconstructed feature image and the j-1-th sample feature image to obtain a j-1-th sample upsampled image;

[0210] The j-1th sample reconstructed feature image is obtained according to the j-1th sample up-sampled image and the sample face parsed image.

[0211] In a possible implementation, the processing module 112 is specifically configured to:

[0212] Performing downsampling and convolution processing on the sample face parsing image to obtain a first sample face parsing image and a second sample face parsing image, wherein the sizes of the first sample face parsing image and the second sample face parsing image are the same as the size of the j-1th sample upsampled image;

[0213] Performing convolution processing and normalization processing on the j-1th sample reconstructed feature image to obtain a j-1th normalized sample reconstructed feature image;

[0214] The j-1th sample reconstructed feature image is obtained according to the j-1th standardized sample reconstructed feature image, the first sample face parsing image and the second sample face parsing image.

[0215] In a possible implementation, the processing module 112 is specifically configured to:

[0216] Reconstructing a feature image based on the R-1 samples to obtain a sample residual feature of the sample image to be processed;

[0217] Performing convolution processing on the sample residual feature to obtain a sample convolution residual feature;

[0218] The sample enhanced image is obtained according to the sample convolution residual feature and the first sample reconstruction feature image.

[0219] In a possible implementation, the processing module 112 is specifically configured to:

[0220] Perform upsampling processing on the k-th sample reconstructed feature image to obtain a corresponding sample upsampled reconstructed feature image, wherein the size of the sample upsampled reconstructed feature image is the same as the size of the second sample reconstructed feature image, and k is R-1, R-2, ..., 3 in sequence;

[0221] Processing the second sample reconstructed feature image and each of the sample up-sampled reconstructed feature images in parallel to obtain a sample parallel reconstructed feature image;

[0222] The sample parallel reconstructed feature image is subjected to convolution processing to obtain the sample residual feature.

[0223] In a possible implementation, the sample image to be processed is the t-th frame image in the sample video image; the processing module 112 is further configured to:

[0224] Obtaining residual features of the t-1th frame image in the sample video image;

[0225] Get sample video frame parameters;

[0226] The sample residual feature is updated according to the residual feature of the t-1th frame image and the sample video inter-frame parameter.

[0227] In a possible implementation, the training module 113 is specifically configured to:

[0228] Obtaining a sample loss value between the sample-enhanced image and the label-enhanced image;

[0229] The parameters of the image enhancement model are adjusted according to the sample loss value.

[0230] In a possible implementation, the training module 113 is specifically configured to:

[0231] Obtaining at least one of a sample global loss value, a sample face loss value, a sample eye loss value, and a sample mouth and nose loss value according to the sample enhanced image and the label enhanced image;

[0232] The sample loss value is obtained according to at least one of the sample global loss value, the sample face loss value, the sample eye loss value and the sample mouth and nose loss value, and the sample loss value weight.

[0233] In a possible implementation, the acquisition module 111 is specifically configured to:

[0234] Acquire the label-enhanced image;

[0235] Parsing the label-enhanced image according to a face parsing model to obtain the sample face parsed image;

[0236] Degrading the label-enhanced image to obtain the sample image to be processed;

[0237] The multiple groups of training samples are obtained according to the label enhanced image, the sample face parsed image and the sample to-be-processed image.

[0238] In a possible implementation, the acquisition module 111 is specifically configured to:

[0239] Obtaining preset probability values ​​of multiple preset degradation operators;

[0240] Determining at least one preset degradation operator for the label enhanced image according to the preset probability values ​​of the multiple preset degradation operators;

[0241] The label enhanced image is degraded at least once according to the at least one preset degradation operator to obtain at least one sample image to be processed.

[0242] In a possible implementation, the sample face parsed image includes category information of each pixel of the sample image to be processed.

[0243] The model training device provided in the embodiment of the present disclosure can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0244] Figure 12 A schematic diagram of the structure of an image processing device provided in an embodiment of the present disclosure is shown in FIG. Figure 12 As shown, the device includes:

[0245] An analysis module 121 is used to perform face analysis on the image to be processed to obtain a corresponding face analysis image;

[0246] The processing module 122 is used to input the image to be processed and the face analysis image into an image enhancement model to obtain an enhanced image corresponding to the image to be processed. The image enhancement model is a model trained according to a model training method.

[0247] In a possible implementation, the processing module 122 is specifically configured to:

[0248] Performing feature extraction on the image to be processed to obtain R feature images, where R is a positive integer greater than or equal to 2;

[0249] Performing image reconstruction processing based on the R feature images and the face analysis image to obtain R-1 reconstructed feature images;

[0250] The enhanced image is obtained according to the R-1 reconstructed feature images.

[0251] In a possible implementation, the processing module 122 is specifically configured to:

[0252] Performing convolution processing on the image to be processed to obtain a first feature image;

[0253] Starting from the first feature image, the i-th feature image is sequentially convolved and downsampled to obtain the i+1-th feature image, where i is 1, 2, ..., R-1, until the R feature images are obtained. The size of the i-th feature image is The M is the number of pixels in the width of the image to be processed, and the N is the number of pixels in the height of the image to be processed.

[0254] In a possible implementation, the processing module 122 is specifically configured to:

[0255] Obtaining an R-1th reconstructed feature image according to the R-th feature image, the R-1-th feature image, and the face parsed image;

[0256] Starting from the R-1th reconstructed feature image, the j-1th reconstructed feature image is obtained according to the jth reconstructed feature image, the j-1th feature image and the face parsed image, where j is R-1, R-2, ..., 2 in sequence until the R-1th reconstructed feature image is obtained.

[0257] In a possible implementation, the processing module 122 is specifically configured to:

[0258] Performing upsampling processing on the R-th feature image to obtain an R-th upsampled image;

[0259] Performing convolution processing and upsampling processing on the R-th up-sampled image and the R-1-th feature image to obtain an R-1-th up-sampled image;

[0260] Convolution processing is performed on the R-1th up-sampled image and the face parsed image to obtain the R-1th reconstructed feature image.

[0261] In a possible implementation, the processing module 122 is specifically configured to:

[0262] Performing convolution processing and upsampling processing on the j-th reconstructed feature image and the j-1-th feature image to obtain a j-1-th upsampled image;

[0263] The j-1th reconstructed feature image is obtained according to the j-1th up-sampled image and the face parsed image.

[0264] In a possible implementation, the processing module 122 is specifically configured to:

[0265] Performing downsampling and convolution processing on the face parsed image to obtain a first face parsed image and a second face parsed image, wherein the sizes of the first face parsed image and the second face parsed image are the same as the size of the j-1th upsampled image;

[0266] Performing convolution processing and normalization processing on the j-1th reconstructed feature image to obtain a j-1th normalized reconstructed feature image;

[0267] The j-1th reconstructed feature image is obtained according to the j-1th standardized reconstructed feature image, the first face parsing image and the second face parsing image.

[0268] In a possible implementation, the processing module 122 is specifically configured to:

[0269] Obtaining residual features of the image to be processed according to the R-1 reconstructed feature images;

[0270] Performing convolution processing on the residual feature to obtain a convolution residual feature;

[0271] The enhanced image is obtained according to the convolution residual features and the first reconstructed feature image.

[0272] In a possible implementation, the processing module 122 is specifically configured to:

[0273] Perform upsampling on the k-th reconstructed feature image to obtain a corresponding upsampled reconstructed feature image, wherein the size of the upsampled reconstructed feature image is the same as the size of the second reconstructed feature image, and k is R-1, R-2, ..., 3 in sequence;

[0274] Performing parallel processing on the second reconstructed feature image and each of the upsampled reconstructed feature images to obtain a parallel reconstructed feature image;

[0275] Convolution processing is performed on the parallel reconstructed feature image to obtain the residual feature.

[0276] In a possible implementation, the image to be processed is the t-th frame image in the video image; the processing module 122 is further configured to:

[0277] Obtaining residual features of the t-1th frame image in the video image;

[0278] Get video frame parameters;

[0279] The residual feature is updated according to the residual feature of the t-1th frame image and the video inter-frame parameter.

[0280] In a possible implementation, the face parsed image includes category information of each pixel of the image to be processed.

[0281] The image processing device provided in the embodiment of the present disclosure can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0282] Exemplary computing devices

[0283] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, Figure 13 A computing device according to an exemplary embodiment of the present disclosure is described.

[0284] Figure 13 The computing device 130 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0285] like Figure 13 As shown, computing device 130 is implemented as a general-purpose computing device. Components of computing device 130 may include, but are not limited to, the aforementioned at least one processing unit 131, the aforementioned at least one storage unit 132, and a bus 133 connecting various system components (including processing unit 131 and storage unit 132).

[0286] The bus 133 includes a data bus, a control bus, and an address bus.

[0287] The storage unit 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 1323 .

[0288] The storage unit 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0289] The computing device 130 may also communicate with one or more external devices 134 (e.g., a keyboard, a pointing device, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the computing device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 136. Figure 13 As shown, network adapter 136 communicates with other modules of computing device 130 via bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 130, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0290] It should be noted that although several units / modules or sub-units / modules of the model training device and the image processing device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided to be embodied by multiple units / modules.

[0291] Furthermore, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0292] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features in these aspects cannot be combined to benefit. Such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A model training method, comprising: Acquire multiple sets of training samples, wherein the training samples include sample images to be processed, sample face parsed images, and label enhanced images; Inputting the sample image to be processed and the sample face parsed image into an image enhancement model to obtain a sample enhanced image output by the image enhancement model; Adjusting parameters of the image enhancement model according to the sample enhanced image and the label enhanced image; The step of inputting the sample image to be processed and the sample face parsed image into an image enhancement model to obtain a sample enhanced image output by the image enhancement model comprises: Performing feature extraction processing on the sample image to be processed using a feature pyramid module to obtain R sample feature images, where R is a positive integer greater than or equal to 2; Performing image reconstruction processing based on the R sample feature images and the sample face parsed image to obtain R-1 sample reconstructed feature images, wherein the image reconstruction processing adopts an image reconstruction module, and uses the output of the lowest layer of the feature pyramid module as input for the lowest reconstruction layer. Each subsequent reconstruction layer uses the parallel connection of the output of the next reconstruction layer and the output of the corresponding feature pyramid layer as input for the convolution module. After the convolution module outputs the reconstructed features, it is input into the face structure embedding module together with the face parsed image of the same size. The face structure information is used to modulate the current reconstructed features to obtain a reconstructed feature image embedded with the face structure information, which is then transmitted to the next reconstruction layer for processing; The characteristic image is reconstructed according to the R-1 samples to obtain the sample enhanced image.

2. The method according to claim 1, wherein the step of performing feature extraction on the sample image to be processed to obtain R sample feature images comprises: Performing convolution processing on the sample image to be processed to obtain a first sample feature image; Starting from the first sample feature image, the i-th sample feature image is sequentially convolved and downsampled to obtain the i+1-th sample feature image, where i is 1, 2, ..., R-1, until the R sample feature images are obtained. The size of the i-th sample feature image is , M is the number of pixels in the width of the sample image to be processed, and N is the number of pixels in the height of the sample image to be processed.

3. The method according to claim 2, wherein the image reconstruction processing is performed based on the R sample feature images and the sample face analysis image to obtain R-1 sample reconstructed feature images, comprising: Obtaining an R-1th sample reconstructed feature image according to the R-th sample feature image, the R-1-th sample feature image and the sample face parsed image; Starting from the R-1th sample reconstructed feature image, the j-1th sample reconstructed feature image is obtained according to the j-th sample reconstructed feature image, the j-1th sample feature image and the sample face parsed image, where j is R-1, R-2, ..., 2 in sequence until the R-1th sample reconstructed feature image is obtained.

4. The method according to claim 3, wherein obtaining the R-1th sample reconstructed feature image based on the Rth sample feature image, the R-1th sample feature image, and the sample face parsed image comprises: Performing upsampling processing on the R-th sample feature image to obtain an R-th sample upsampled image; Performing convolution processing and upsampling processing on the R-th sample upsampled image and the R-1-th feature image to obtain the R-1-th sample upsampled image; Convolution processing is performed on the R-1th sample up-sampled image and the sample face parsed image to obtain the R-1th sample reconstructed feature image.

5. The method according to claim 3, wherein obtaining the j-1th sample reconstructed feature image based on the j-th sample reconstructed feature image, the j-1th sample feature image, and the sample face parsed image comprises: Performing convolution processing and upsampling processing on the j-th sample reconstructed feature image and the j-1-th sample feature image to obtain a j-1-th sample upsampled image; The j-1th sample reconstructed feature image is obtained according to the j-1th sample up-sampled image and the sample face parsed image.

6. The method according to claim 5, wherein obtaining the j-1th sample reconstructed feature image based on the j-1th sample upsampled image and the sample face parsed image comprises: Performing downsampling and convolution processing on the sample face parsing image to obtain a first sample face parsing image and a second sample face parsing image, wherein the sizes of the first sample face parsing image and the second sample face parsing image are the same as the size of the j-1th sample upsampled image; Performing convolution processing and normalization processing on the j-1th sample reconstructed feature image to obtain a j-1th normalized sample reconstructed feature image; The j-1th sample reconstructed feature image is obtained according to the j-1th standardized sample reconstructed feature image, the first sample face parsing image and the second sample face parsing image.

7. The method according to any one of claims 1 to 6, wherein reconstructing a feature image based on the R-1 samples to obtain the sample enhanced image comprises: Reconstructing a feature image based on the R-1 samples to obtain a sample residual feature of the sample image to be processed; Performing convolution processing on the sample residual feature to obtain a sample convolution residual feature; The sample enhanced image is obtained according to the sample convolution residual feature and the first sample reconstruction feature image.

8. The method according to claim 7, wherein reconstructing the feature image based on the R-1 samples to obtain the sample residual feature of the sample image to be processed comprises: Perform upsampling processing on the k-th sample reconstructed feature image to obtain a corresponding sample upsampled reconstructed feature image, wherein the size of the sample upsampled reconstructed feature image is the same as the size of the second sample reconstructed feature image, and k is R-1, R-2, ..., 3 in sequence; Processing the second sample reconstructed feature image and each of the sample up-sampled reconstructed feature images in parallel to obtain a sample parallel reconstructed feature image; The sample parallel reconstructed feature image is subjected to convolution processing to obtain the sample residual feature.

9. The method according to claim 8, wherein the sample image to be processed is the t-th frame image in the sample video image; the method further comprises: Obtaining residual features of the t-1th frame image in the sample video image; Get sample video frame parameters; The sample residual feature is updated according to the residual feature of the t-1th frame image and the sample video inter-frame parameter.

10. The method according to any one of claims 1 to 6, wherein adjusting parameters of the image enhancement model according to the sample-enhanced image and the label-enhanced image comprises: Obtaining a sample loss value between the sample-enhanced image and the label-enhanced image; The parameters of the image enhancement model are adjusted according to the sample loss value.

11. The method according to claim 10, wherein obtaining a sample loss value between the sample-enhanced image and the label-enhanced image comprises: Obtaining at least one of a sample global loss value, a sample face loss value, a sample eye loss value, and a sample mouth and nose loss value according to the sample enhanced image and the label enhanced image; The sample loss value is obtained according to at least one of the sample global loss value, the sample face loss value, the sample eye loss value and the sample mouth and nose loss value, and the sample loss value weight.

12. The method according to any one of claims 1 to 6, wherein obtaining multiple sets of training samples comprises: Acquire the label-enhanced image; Parsing the label-enhanced image according to a face parsing model to obtain the sample face parsed image; Degrading the label-enhanced image to obtain the sample image to be processed; The multiple groups of training samples are obtained according to the label enhanced image, the sample face parsed image and the sample to-be-processed image.

13. The method according to claim 12, wherein the step of degrading the label-enhanced image to obtain the sample image to be processed comprises: Obtaining preset probability values ​​of multiple preset degradation operators; Determining at least one preset degradation operator for the label enhanced image according to the preset probability values ​​of the multiple preset degradation operators; The label enhanced image is degraded at least once according to the at least one preset degradation operator to obtain at least one sample image to be processed.

14. The method according to any one of claims 1 to 6, wherein the sample face parsed image includes category information of each pixel point of the sample image to be processed.

15. An image processing method, comprising: Perform face analysis on the image to be processed to obtain a corresponding face analysis image; The image to be processed and the face parsed image are input into an image enhancement model to obtain an enhanced image corresponding to the image to be processed, wherein the image enhancement model is a model trained by the model training method according to any one of claims 1 to 14.

16. The method according to claim 15, wherein inputting the image to be processed and the face parsed image into an image enhancement model to obtain an enhanced image corresponding to the image to be processed comprises: Performing feature extraction on the image to be processed to obtain R feature images, where R is a positive integer greater than or equal to 2; Performing image reconstruction processing based on the R feature images and the face analysis image to obtain R-1 reconstructed feature images; The enhanced image is obtained according to the R-1 reconstructed feature images.

17. The method according to claim 16, wherein the step of performing feature extraction on the image to be processed to obtain R feature images comprises: Performing convolution processing on the image to be processed to obtain a first feature image; Starting from the first feature image, the i-th feature image is sequentially convolved and downsampled to obtain the i+1-th feature image, where i is 1, 2, ..., R-1, until the R feature images are obtained. The size of the i-th feature image is , M is the number of pixels in the width of the image to be processed, and N is the number of pixels in the height of the image to be processed.

18. The method according to claim 17, wherein the performing image reconstruction processing based on the R feature images and the face analysis image to obtain R-1 reconstructed feature images comprises: Obtaining an R-1th reconstructed feature image according to the R-th feature image, the R-1-th feature image, and the face parsed image; Starting from the R-1th reconstructed feature image, the j-1th reconstructed feature image is obtained according to the jth reconstructed feature image, the j-1th feature image and the face parsed image, where j is R-1, R-2, ..., 2, until the R-1th reconstructed feature image is obtained.

19. The method according to claim 18, wherein obtaining the R-1th reconstructed feature image based on the Rth feature image, the R-1th feature image, and the face parsed image comprises: Performing upsampling processing on the R-th feature image to obtain an R-th upsampled image; Performing convolution processing and upsampling processing on the R-th up-sampled image and the R-1-th feature image to obtain an R-1-th up-sampled image; Convolution processing is performed on the R-1th up-sampled image and the face parsed image to obtain the R-1th reconstructed feature image.

20. The method according to claim 18, wherein obtaining the j-1th reconstructed feature image according to the j-th reconstructed feature image, the j-1th feature image, and the face parsed image comprises: Performing convolution processing and upsampling processing on the j-th reconstructed feature image and the j-1-th feature image to obtain a j-1-th upsampled image; The j-1th reconstructed feature image is obtained according to the j-1th up-sampled image and the face parsed image.

21. The method according to claim 20, wherein obtaining the j-1th reconstructed feature image based on the j-1th upsampled image and the face parsed image comprises: Performing downsampling and convolution processing on the face parsed image to obtain a first face parsed image and a second face parsed image, wherein the sizes of the first face parsed image and the second face parsed image are the same as the size of the j-1th upsampled image; Performing convolution processing and normalization processing on the j-1th reconstructed feature image to obtain a j-1th normalized reconstructed feature image; The j-1th reconstructed feature image is obtained according to the j-1th standardized reconstructed feature image, the first face parsing image and the second face parsing image.

22. The method according to any one of claims 16 to 21, wherein obtaining the enhanced image based on the R-1 reconstructed feature images comprises: Obtaining residual features of the image to be processed according to the R-1 reconstructed feature images; Performing convolution processing on the residual feature to obtain a convolution residual feature; The enhanced image is obtained according to the convolution residual features and the first reconstructed feature image.

23. The method according to claim 22, wherein obtaining the residual features of the image to be processed based on the R-1 reconstructed feature images comprises: Perform upsampling on the k-th reconstructed feature image to obtain a corresponding upsampled reconstructed feature image, wherein the size of the upsampled reconstructed feature image is the same as the size of the second reconstructed feature image, and k is R-1, R-2, ..., 3 in sequence; Performing parallel processing on the second reconstructed feature image and each of the upsampled reconstructed feature images to obtain a parallel reconstructed feature image; Convolution processing is performed on the parallel reconstructed feature image to obtain the residual feature.

24. The method according to claim 23, wherein the image to be processed is the t-th frame image in a video image; the method further comprising: Obtaining residual features of the t-1th frame image in the video image; Get video frame parameters; The residual feature is updated according to the residual feature of the t-1th frame image and the video inter-frame parameter.

25. The method according to any one of claims 16 to 21, wherein the face parsed image comprises category information of each pixel of the image to be processed.

26. A model training device comprising: An acquisition module is used to acquire multiple sets of training samples, wherein the training samples include sample images to be processed, sample face parsed images, and label enhanced images; a processing module, configured to input the sample image to be processed and the sample face parsed image into an image enhancement model to obtain a sample enhanced image output by the image enhancement model; A training module, configured to adjust parameters of the image enhancement model according to the sample enhanced image and the label enhanced image; The processing module is specifically used for: Performing feature extraction processing on the sample image to be processed using a feature pyramid module to obtain R sample feature images, where R is a positive integer greater than or equal to 2; Performing image reconstruction processing based on the R sample feature images and the sample face parsed image to obtain R-1 sample reconstructed feature images, wherein the image reconstruction processing adopts an image reconstruction module, and uses the output of the lowest layer of the feature pyramid module as input for the lowest reconstruction layer. Each subsequent reconstruction layer uses the parallel connection of the output of the next reconstruction layer and the output of the corresponding feature pyramid layer as input for the convolution module. After the convolution module outputs the reconstructed features, it is input into the face structure embedding module together with the face parsed image of the same size. The face structure information is used to modulate the current reconstructed features to obtain a reconstructed feature image embedded with the face structure information, which is then transmitted to the next reconstruction layer for processing; The characteristic image is reconstructed according to the R-1 samples to obtain the sample enhanced image.

27. The apparatus according to claim 26, wherein the processing module is specifically configured to: Performing convolution processing on the sample image to be processed to obtain a first sample feature image; Starting from the first sample feature image, the i-th sample feature image is sequentially convolved and downsampled to obtain the i+1-th sample feature image, where i is 1, 2, ..., R-1, until the R sample feature images are obtained. The size of the i-th sample feature image is , M is the number of pixels in the width of the sample image to be processed, and N is the number of pixels in the height of the sample image to be processed.

28. The apparatus according to claim 27, wherein the processing module is specifically configured to: Obtaining an R-1th sample reconstructed feature image according to the R-th sample feature image, the R-1-th sample feature image and the sample face parsed image; Starting from the R-1th sample reconstructed feature image, the j-1th sample reconstructed feature image is obtained according to the j-th sample reconstructed feature image, the j-1th sample feature image and the sample face parsed image, where j is R-1, R-2, ..., 2 in sequence until the R-1th sample reconstructed feature image is obtained.

29. The apparatus according to claim 28, wherein the processing module is specifically configured to: Performing upsampling processing on the R-th sample feature image to obtain an R-th sample upsampled image; Performing convolution processing and upsampling processing on the R-th sample upsampled image and the R-1-th feature image to obtain the R-1-th sample upsampled image; Convolution processing is performed on the R-1th sample up-sampled image and the sample face parsed image to obtain the R-1th sample reconstructed feature image.

30. The apparatus according to claim 28, wherein the processing module is specifically configured to: Performing convolution processing and upsampling processing on the j-th sample reconstructed feature image and the j-1-th sample feature image to obtain a j-1-th sample upsampled image; The j-1th sample reconstructed feature image is obtained according to the j-1th sample up-sampled image and the sample face parsed image.

31. The apparatus according to claim 30, wherein the processing module is specifically configured to: Performing downsampling and convolution processing on the sample face parsing image to obtain a first sample face parsing image and a second sample face parsing image, wherein the sizes of the first sample face parsing image and the second sample face parsing image are the same as the size of the j-1th sample upsampled image; Performing convolution processing and normalization processing on the j-1th sample reconstructed feature image to obtain a j-1th normalized sample reconstructed feature image; The j-1th sample reconstructed feature image is obtained according to the j-1th standardized sample reconstructed feature image, the first sample face parsing image and the second sample face parsing image.

32. The apparatus according to any one of claims 26 to 31, wherein the processing module is specifically configured to: Reconstructing a feature image based on the R-1 samples to obtain a sample residual feature of the sample image to be processed; Performing convolution processing on the sample residual feature to obtain a sample convolution residual feature; The sample enhanced image is obtained according to the sample convolution residual feature and the first sample reconstruction feature image.

33. The apparatus according to claim 32, wherein the processing module is specifically configured to: Perform upsampling processing on the k-th sample reconstructed feature image to obtain a corresponding sample upsampled reconstructed feature image, wherein the size of the sample upsampled reconstructed feature image is the same as the size of the second sample reconstructed feature image, and k is R-1, R-2, ..., 3 in sequence; Processing the second sample reconstructed feature image and each of the sample up-sampled reconstructed feature images in parallel to obtain a sample parallel reconstructed feature image; The sample parallel reconstructed feature image is subjected to convolution processing to obtain the sample residual feature.

34. The apparatus according to claim 33, wherein the sample image to be processed is the t-th frame image in the sample video image; and the processing module is further configured to: Obtaining residual features of the t-1th frame image in the sample video image; Get sample video frame parameters; The sample residual feature is updated according to the residual feature of the t-1th frame image and the sample video inter-frame parameter.

35. The apparatus according to any one of claims 26 to 31, wherein the training module is specifically configured to: Obtaining a sample loss value between the sample-enhanced image and the label-enhanced image; The parameters of the image enhancement model are adjusted according to the sample loss value.

36. The apparatus according to claim 35, wherein the training module is specifically configured to: Obtaining at least one of a sample global loss value, a sample face loss value, a sample eye loss value, and a sample mouth and nose loss value according to the sample enhanced image and the label enhanced image; The sample loss value is obtained according to at least one of the sample global loss value, the sample face loss value, the sample eye loss value and the sample mouth and nose loss value, and the sample loss value weight.

37. The apparatus according to any one of claims 26 to 31, wherein the acquisition module is specifically configured to: Acquire the label-enhanced image; Parsing the label-enhanced image according to a face parsing model to obtain the sample face parsed image; Degrading the label-enhanced image to obtain the sample image to be processed; The multiple groups of training samples are obtained according to the label enhanced image, the sample face parsed image and the sample to-be-processed image.

38. The apparatus according to claim 37, wherein the acquisition module is specifically configured to: Obtaining preset probability values ​​of multiple preset degradation operators; Determining at least one preset degradation operator for the label enhanced image according to the preset probability values ​​of the multiple preset degradation operators; The label enhanced image is degraded at least once according to the at least one preset degradation operator to obtain at least one sample image to be processed.

39. The apparatus according to any one of claims 26 to 31, wherein the sample face parsed image comprises category information of each pixel of the sample image to be processed.

40. An image processing device comprising: An analysis module is used to perform face analysis on the image to be processed to obtain a corresponding face analysis image; A processing module is used to input the image to be processed and the face parsed image into an image enhancement model to obtain an enhanced image corresponding to the image to be processed, and the image enhancement model is a model trained according to the model training method according to any one of claims 1 to 14.

41. The apparatus according to claim 40, wherein the processing module is specifically configured to: Performing feature extraction on the image to be processed to obtain R feature images, where R is a positive integer greater than or equal to 2; Performing image reconstruction processing based on the R feature images and the face analysis image to obtain R-1 reconstructed feature images; The enhanced image is obtained according to the R-1 reconstructed feature images.

42. The apparatus according to claim 41, wherein the processing module is specifically configured to: Performing convolution processing on the image to be processed to obtain a first feature image; Starting from the first feature image, the i-th feature image is sequentially convolved and downsampled to obtain the i+1-th feature image, where i is 1, 2, ..., R-1, until the R feature images are obtained. The size of the i-th feature image is , M is the number of pixels in the width of the sample image to be processed, and N is the number of pixels in the height of the sample image to be processed.

43. The apparatus according to claim 42, wherein the processing module is specifically configured to: Obtaining an R-1th reconstructed feature image according to the R-th feature image, the R-1-th feature image, and the face parsed image; Starting from the R-1th reconstructed feature image, the j-1th reconstructed feature image is obtained according to the jth reconstructed feature image, the j-1th feature image and the face parsed image, where j is R-1, R-2, ..., 2, until the R-1th reconstructed feature image is obtained.

44. The apparatus according to claim 43, wherein the processing module is specifically configured to: Performing upsampling processing on the R-th feature image to obtain an R-th upsampled image; Performing convolution processing and upsampling processing on the R-th up-sampled image and the R-1-th feature image to obtain an R-1-th up-sampled image; Convolution processing is performed on the R-1th up-sampled image and the face parsed image to obtain the R-1th reconstructed feature image.

45. The apparatus according to claim 43, wherein the processing module is specifically configured to: Performing convolution processing and upsampling processing on the j-th reconstructed feature image and the j-1-th feature image to obtain a j-1-th upsampled image; The j-1th reconstructed feature image is obtained according to the j-1th up-sampled image and the face parsed image.

46. ​​The apparatus according to claim 45, wherein the processing module is specifically configured to: Performing downsampling and convolution processing on the face parsed image to obtain a first face parsed image and a second face parsed image, wherein the sizes of the first face parsed image and the second face parsed image are the same as the size of the j-1th upsampled image; Performing convolution processing and normalization processing on the j-1th reconstructed feature image to obtain a j-1th normalized reconstructed feature image; The j-1th reconstructed feature image is obtained according to the j-1th standardized reconstructed feature image, the first face parsing image and the second face parsing image.

47. The apparatus according to any one of claims 41 to 46, wherein the processing module is specifically configured to: Obtaining residual features of the image to be processed according to the R-1 reconstructed feature images; Performing convolution processing on the residual feature to obtain a convolution residual feature; The enhanced image is obtained according to the convolution residual features and the first reconstructed feature image.

48. The apparatus according to claim 47, wherein the processing module is specifically configured to: Perform upsampling on the k-th reconstructed feature image to obtain a corresponding upsampled reconstructed feature image, wherein the size of the upsampled reconstructed feature image is the same as the size of the second reconstructed feature image, and k is R-1, R-2, ..., 3 in sequence; Performing parallel processing on the second reconstructed feature image and each of the upsampled reconstructed feature images to obtain a parallel reconstructed feature image; Convolution processing is performed on the parallel reconstructed feature image to obtain the residual feature.

49. The apparatus according to claim 48, wherein the image to be processed is the t-th frame image in a video image; and the processing module is further configured to: Obtaining residual features of the t-1th frame image in the video image; Get video frame parameters; The residual feature is updated according to the residual feature of the t-1th frame image and the video inter-frame parameter.

50. The device according to any one of claims 41 to 46, wherein the face parsed image includes category information of each pixel point of the image to be processed.

51. A computing device comprising: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the model training method as described in any one of claims 1-14, or the at least one processor executes the image processing method as described in any one of claims 15-25.

52. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the model training method according to any one of claims 1 to 14 is implemented, or the image processing method according to any one of claims 15 to 25 is implemented.

53. A computer program product, comprising a computer program; when the computer program is executed, it implements the model training method according to any one of claims 1 to 14, or implements the image processing method according to any one of claims 15 to 25.

Images