Training method of enhanced model, image processing method, device, equipment and medium

Abstract

The present disclosure relates to a kind of training method and device of image enhancement model, image processing method and device, electronic equipment and computer readable medium, belong to image processing technical field.The training method of the image enhancement model includes: obtaining the training data set of the image enhancement model, and the training sample image in the training data set is carried out image degradation processing, obtains input sample image;Attention concentration module is added in image semantic segmentation network to construct initial image enhancement model, and the input sample image is input in the initial image enhancement model and obtains corresponding output sample image;The output sample image is input into discriminator to obtain discriminator loss, and the model parameter in the initial image enhancement model is iteratively updated based on the discriminator loss, obtains the image enhancement model trained.The present disclosure can effectively improve image enhancement quality by training image enhancement model based on the training data synthesized based on degradation model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to a training method and device of an image enhancement model, an image processing method and device, an electronic device, and a computer readable medium. BACKGROUND

[0002] With the rise of images and videos, there are many different types of images and videos on web pages for users to browse. However, these images and videos generally have problems such as low clarity, uneven frame rate, and insufficient image color.

[0003] The improvement of image quality is mainly based on three aspects: color enhancement, deblurring, and image super-resolution. For images or videos with poor visual perception, the saturation, sharpness, and contrast of the picture can be improved through the above three modules, while the noise and blur of the picture are reduced. However, the processing effect cannot be too strong, otherwise it will cause an unrealistic perception.

[0004] Therefore, there is an urgent need in the art for a method that can effectively improve the image enhancement quality.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The purpose of the present disclosure is to provide a training method of an image enhancement model, a training device of an image enhancement model, an electronic device, and a computer readable medium, which can effectively improve the image enhancement quality at least to some extent.

[0007] According to a first aspect of the present disclosure, a training method of an image enhancement model is provided, comprising:

[0008] obtaining a training data set of the image enhancement model, and performing image degradation processing on a training sample image in the training data set to obtain an input sample image of the image enhancement model;

[0009] adding an attention focusing module to an image semantic segmentation network to construct an initial image enhancement model, and inputting the input sample image into the initial image enhancement model to obtain an output sample image corresponding to the input sample image;

[0010] inputting the output sample image into a discriminator to obtain a corresponding discriminator loss, and iteratively updating model parameters in the initial image enhancement model based on the discriminator loss to obtain a trained image enhancement model.

[0011] In an example embodiment of the present disclosure, the image degradation processing on the training sample images in the training data set to obtain the input sample images of the image enhancement model comprises:

[0012] obtaining a self-training degradation method set and a random degradation method set for image degradation processing;

[0013] determining a target self-training degradation method from the self-training degradation method set and a target random degradation method from the random degradation method set;

[0014] degrading the training sample images by the target self-training degradation method and the target random degradation method to obtain the input sample images corresponding to the training sample images.

[0015] In an example embodiment of the present disclosure, the input of the input sample images into the initial image enhancement model to obtain the output sample images corresponding to the input sample images comprises:

[0016] obtaining the feature maps of the input sample images corresponding to each feature extraction layer in the image semantic segmentation network in sequence through the multiple feature extraction layers in the image semantic segmentation network;

[0017] adjusting the channel attention coefficients corresponding to each channel and the region attention coefficients corresponding to each region in the feature map corresponding to each feature extraction layer through the attention concentration module corresponding to each feature extraction layer in the image semantic segmentation network;

[0018] obtaining the output sample images corresponding to the input sample images through the multiple up-sampling layers in the image semantic segmentation network based on the channel attention coefficients and the region attention coefficients corresponding to each feature extraction layer and the feature maps.

[0019] In an example embodiment of the present disclosure, the input of the output sample images into the discriminator to obtain the corresponding discriminator loss and the iterative update of the model parameters in the initial image enhancement model based on the discriminator loss comprises:

[0020] inputting the output sample images into the discriminator and obtaining the first output result of the discriminator through the image semantic segmentation network in the discriminator and the second output result of the discriminator through the cross network in the discriminator;

[0021] iteratively updating the model parameters in the initial image enhancement model based on the first output result of the discriminator and the second output result of the discriminator.

[0022] According to a second aspect of the present disclosure, an image processing method is provided, comprising:

[0023] obtain an original image to be processed, and input the original image into a pre-trained image enhancement model, wherein the image enhancement model is obtained by the training method of the image enhancement model according to any one of the preceding embodiments;

[0024] perform image enhancement processing on the original image by using the image enhancement model to obtain an enhanced image corresponding to the original image.

[0025] In an exemplary embodiment of the present disclosure, the image enhancement processing on the original image by using the image enhancement model to obtain an enhanced image corresponding to the original image comprises:

[0026] obtain, in sequence, feature maps of the original image corresponding to each feature extraction layer in the image semantic segmentation network by using the plurality of feature extraction layers in the image semantic segmentation network;

[0027] determine channel attention coefficients corresponding to each channel and region attention coefficients corresponding to each region in the feature map corresponding to each feature extraction layer by using the attention concentration module corresponding to each feature extraction layer in the image semantic segmentation network;

[0028] obtain an output image of the image enhancement model by using a plurality of up-sampling layers in the image semantic segmentation network based on the channel attention coefficients and the region attention coefficients corresponding to each feature extraction layer and the feature map;

[0029] obtain an enhanced image corresponding to the original image according to the output image of the image enhancement model.

[0030] According to a third aspect of the present disclosure, there is provided an image enhancement model training device, comprising:

[0031] an image degradation processing module configured to obtain a training data set of the image enhancement model, and perform image degradation processing on a training sample image in the training data set to obtain an input sample image of the image enhancement model;

[0032] an enhancement model construction module configured to add an attention concentration module to an image semantic segmentation network to construct an initial image enhancement model, and input the input sample image into the initial image enhancement model to obtain an output sample image corresponding to the input sample image;

[0033] an enhancement model training module configured to input the output sample image into a discriminator to obtain a corresponding discriminator loss, and iteratively update model parameters in the initial image enhancement model based on the discriminator loss to obtain the trained image enhancement model.

[0034] In an example embodiment of the present disclosure, the image degradation processing module comprises:

[0035] a degradation method set obtaining unit configured to obtain a self-training degradation method set and a random degradation method set for image degradation processing;

[0036] a target degradation method determining unit configured to determine a target self-training degradation method from the self-training degradation method set and a target random degradation method from the random degradation method set;

[0037] an image degradation processing unit configured to perform image degradation processing on the training sample image by using the target self-training degradation method and the target random degradation method to obtain an input sample image corresponding to the training sample image.

[0038] In an example embodiment of the present disclosure, the enhancement model construction module comprises:

[0039] a feature map extraction unit configured to sequentially obtain a feature map corresponding to each feature extraction layer in the image semantic segmentation network by using a plurality of feature extraction layers in the image semantic segmentation network;

[0040] an attention coefficient adjustment unit configured to adjust a channel attention coefficient corresponding to each channel and a region attention coefficient corresponding to each region in the feature map corresponding to each feature extraction layer in the image semantic segmentation network by using an attention center module corresponding to each feature extraction layer in the image semantic segmentation network;

[0041] a feature up-sampling unit configured to obtain an output sample image corresponding to the input sample image by using a plurality of up-sampling layers in the image semantic segmentation network based on the channel attention coefficient and the region attention coefficient corresponding to each feature extraction layer and the feature map.

[0042] In an example embodiment of the present disclosure, the enhancement model training module comprises:

[0043] a discriminator loss determining unit configured to input the output sample image into a discriminator, obtain a first output result of the discriminator by using an image semantic segmentation network in the discriminator, and obtain a second output result of the discriminator by using a cross network in the discriminator;

[0044] a model parameter updating unit configured to iteratively update model parameters in the initial image enhancement model based on the first output result of the discriminator and the second output result of the discriminator.

[0045] According to a fourth aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0046] An original image input module is configured to acquire an original image to be processed and input the original image into a pre-trained image enhancement model, wherein the image enhancement model is obtained by the training device of the image enhancement model as described above.

[0047] An image enhancement processing module is configured to perform image enhancement processing on the original image by using the image enhancement model to obtain an enhanced image corresponding to the original image.

[0048] In an exemplary embodiment of the present disclosure, the image enhancement processing module comprises:

[0049] An original image feature map extraction unit is configured to sequentially obtain a feature map corresponding to each feature extraction layer of the original image by using a plurality of feature extraction layers in the image semantic segmentation network;

[0050] An attention coefficient determination unit is configured to determine a channel attention coefficient corresponding to each channel and a region attention coefficient corresponding to each region in the feature map corresponding to each feature extraction layer by using an attention concentration module corresponding to each feature extraction layer in the image semantic segmentation network;

[0051] An output image determination unit is configured to obtain an output image of the image enhancement model by using a plurality of up-sampling layers in the image semantic segmentation network based on the channel attention coefficient and the region attention coefficient corresponding to each feature extraction layer and the feature map;

[0052] An enhanced image determination unit is configured to obtain an enhanced image corresponding to the original image according to the output image of the image enhancement model.

[0053] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the image enhancement model training method or the image processing method as described above by executing the executable instructions.

[0054] According to a sixth aspect of the present disclosure, a computer readable medium is provided, which stores a computer program, wherein the computer program is executed by a processor to implement the image enhancement model training method or the image processing method as described above.

[0055] The exemplary embodiments of the present disclosure can have the following beneficial effects:

[0056] In the training method of the image enhancement model of the example embodiment of the present disclosure, an attention focusing module is added to an image semantic segmentation network to construct an initial image enhancement model, image degradation processing is performed on a training sample image in a training data set to obtain an input sample image of the image enhancement model, then a corresponding discriminator loss is obtained according to an output sample image, and model parameters in the initial image enhancement model are iteratively updated based on the discriminator loss to train the image enhancement model. The training method of the image enhancement model in the example embodiment of the present disclosure considers and designs a more complex degradation space, trains the image enhancement model based on the training data synthesized by the degradation model, and the obtained model achieves very good results on different types of real degradation data, effectively improves the image enhancement quality, saves the labor screening cost, and realizes real-time calculation, thereby optimizing the visual experience when a user browses images or videos.

[0057] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0058] The accompanying drawings, which are incorporated into and form part of the specification, illustrate an embodiment consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0059] Figure 1 A flowchart of the training method of the image enhancement model of the example embodiment of the present disclosure is shown;

[0060] Figure 2 A flowchart of the image degradation processing of the training sample image of the example embodiment of the present disclosure is shown;

[0061] Figure 3 A flowchart of obtaining an output sample image by the initial image enhancement model of the example embodiment of the present disclosure is shown;

[0062] Figure 4 A flowchart of iteratively updating the model parameters based on the discriminator loss of the example embodiment of the present disclosure is shown;

[0063] Figure 5 A general framework diagram of the training method of the image enhancement model according to one specific embodiment of the present disclosure is shown;

[0064] Figure 6 A schematic diagram of the image degradation method according to one specific embodiment of the present disclosure is shown;

[0065] Figure 7 A schematic diagram of a random degradation mode according to one embodiment of the present disclosure is shown;

[0066] Figure 8 A structural schematic diagram of an attention focusing module according to one embodiment of the present disclosure is shown;

[0067] Figure 9 A structural schematic diagram of a discriminator network according to one embodiment of the present disclosure is shown;

[0068] Figure 10 A flowchart of an image processing method according to an example embodiment of the present disclosure is shown;

[0069] Figure 11 A flowchart of obtaining an enhanced image corresponding to an original image by an image enhancement model according to an example embodiment of the present disclosure is shown;

[0070] Figure 12 A block diagram of a training device of an image enhancement model according to an example embodiment of the present disclosure is shown;

[0071] Figure 13 A block diagram of an image processing device according to an example embodiment of the present disclosure is shown;

[0072] Figure 14 A structural schematic diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0073] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the example embodiments to those skilled in the art. The features, structures, or characteristics described in connection with the embodiments can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. Other implements can be utilized and present examples are not intended as a limitation on the scope of the present disclosure.

[0074] Further, the accompanying drawings are included to provide a thorough understanding of the present disclosure and are not intended to be in any way limiting. Like

[0075] The present example embodiment first provides a training method of an image enhancement model. Referring to Figure 1 As shown in the figure, the training method of the image enhancement model can include the following steps:

[0076] Step S110. Obtain a training data set of the image enhancement model, and perform image degradation processing on the training sample images in the training data set to obtain input sample images of the image enhancement model.

[0077] Step S120. Add an attention focusing module to the image semantic segmentation network to construct an initial image enhancement model, and input the input sample images into the initial image enhancement model to obtain output sample images corresponding to the input sample images.

[0078] Step S130. Input the output sample images into the discriminator to obtain the corresponding discriminator loss, and iteratively update the model parameters in the initial image enhancement model based on the discriminator loss to obtain the trained image enhancement model.

[0079] In the training method of the image enhancement model of the present example embodiment of the present disclosure, by adding an attention focusing module to the image semantic segmentation network to construct an initial image enhancement model, and by performing image degradation processing on the training sample images in the training data set to obtain input sample images of the image enhancement model, then according to the output sample images, the corresponding discriminator loss is obtained, and the model parameters in the initial image enhancement model are iteratively updated based on the discriminator loss to train the image enhancement model. The training method of the image enhancement model in the present example embodiment of the present disclosure considers and designs a more complex degradation space, trains the image enhancement model based on the training data synthesized by the degradation model, and the obtained model achieves very good results on different types of real degradation data, effectively improves the image enhancement quality, saves the cost of manual screening, and can realize real-time calculation, and optimizes the visual experience when the user browses the image or video.

[0080] Next, the above steps of the present example embodiment will be described in more detail. Figures 2 to 4 The above steps of the present example embodiment will be described in more detail.

[0081] In step S110, a training data set of the image enhancement model is obtained, and image degradation processing is performed on the training sample images in the training data set to obtain input sample images of the image enhancement model.

[0082] In the example embodiment, first, the training data set of the image enhancement model needs to be obtained. For example, the DIV2K data set (image super-resolution reconstruction data set), the Flickr2K data set (image super-resolution reconstruction data set), the WED data set (image super-resolution reconstruction data set), and a large number of face images derived from the FFHQ (Flickr Faces Hight Quality) data set can be used as the training data set. After determining the training data set of the image enhancement model, image degradation processing needs to be performed on the training sample images in the training data set to obtain the input sample images of the image enhancement model.

[0083] In the example embodiment, as shown in Figure 2 The image degradation processing of the training sample images in the training data set to obtain the input sample images of the image enhancement model can include the following steps:

[0084] Step S210. Obtain a self-training degradation method set and a random degradation method set for image degradation processing.

[0085] The image degradation processing can adopt two ways of self-training degradation and random degradation. The random degradation method set can include noise, blur, scaling, compression, etc., and the self-training degradation method set can include shallow blur, middle blur, and high blur, where shallow, middle, and high represent the degree of blur.

[0086] Step S220. Determine a target self-training degradation method from the self-training degradation method set and a target random degradation method from the random degradation method set.

[0087] A self-training degradation method is randomly determined from the self-training degradation method set as a target self-training degradation method, and a random degradation method is randomly determined from the random degradation method set as a target random degradation method.

[0088] Step S230. Perform image degradation processing on the training sample images by the target self-training degradation method and the target random degradation method to obtain the input sample images corresponding to the training sample images.

[0089] After the training sample images are processed by the target self-training degradation method and the target random degradation method in a random superposition manner, the input sample images corresponding to the training sample images are obtained, which are used to input into the network.

[0090] In step S120, the attention focusing module is added to the image semantic segmentation network to construct an initial image enhancement model, and an input sample image is input into the initial image enhancement model to obtain an output sample image corresponding to the input sample image.

[0091] In the example embodiment, the backbone network of the image enhancement model adopts a Unet+NATM structure. The Unet (image semantic segmentation network) is a segmentation network with a U-shaped symmetric structure, with feature extraction layers on the left side and up-sampling layers on the right side. The feature maps obtained by each feature extraction layer of the Unet network are connected to the corresponding up-sampling layer, so that each layer of feature map is effectively used in subsequent calculations. The NATM (Net Attention Model) is used for attention processing of features to strengthen the perception learning ability of the model. By applying the attention mechanism to the Unet segmentation network, the attention to the salient regions and the suppression of irrelevant background regions can be better achieved.

[0092] In the example embodiment, as shown in Figure 3 the input sample image is input into the initial image enhancement model to obtain an output sample image corresponding to the input sample image, which can include the following steps:

[0093] Step S310. The input sample image is sequentially obtained in each feature extraction layer corresponding to the feature extraction layer through a plurality of feature extraction layers in the image semantic segmentation network.

[0094] First, the input sample image is sequentially obtained in each feature extraction layer corresponding to the feature extraction layer through a plurality of feature extraction layers on the left side of the image semantic segmentation network.

[0095] Step S320. The channel attention coefficients corresponding to each channel and the region attention coefficients corresponding to each region in the feature map corresponding to the feature extraction layer are adjusted by the attention focusing module corresponding to each feature extraction layer in the image semantic segmentation network.

[0096] When the feature map obtained by each feature extraction layer is connected to the corresponding up-sampling layer, the channel attention coefficients corresponding to each channel and the region attention coefficients corresponding to each region in the feature map corresponding to each feature extraction layer can be adjusted by the attention focusing module corresponding to each feature extraction layer.

[0097] The channel attention coefficient can be used to adjust the channel-focused attention of the feature map. Channel-focused attention is a top-down, conscious attention, classified as active attention. It refers to attention that is focused on a specific object with a predetermined purpose and task-dependent. The regional attention coefficient can be used to adjust the spatial-focused attention of the feature map. Spatial-focused attention is a bottom-up, conscious attention, classified as passive attention. Saliency-based attention is driven by external stimuli, does not require active intervention, and is unrelated to the task. Pooling and gating mechanisms can be roughly viewed as bottom-up, saliency-based attention mechanisms.

[0098] Step S330. Based on the channel attention coefficients and regional attention coefficients corresponding to each feature extraction layer and the feature map, an output sample image corresponding to the input sample image is obtained through multiple upsampling layers in the image semantic segmentation network.

[0099] Based on the channel attention coefficient and regional attention coefficient corresponding to each feature extraction layer, the channel focus attention and spatial focus attention of the feature map corresponding to each feature extraction layer are adjusted, connected to the corresponding upsampling layer, and then the output sample image corresponding to the input sample image is obtained through multiple upsampling layers.

[0100] In step S130, the output sample image is input into the discriminator to obtain the corresponding discriminator loss, and the model parameters in the initial image enhancement model are iteratively updated based on the discriminator loss to obtain a trained image enhancement model.

[0101] The discriminator can be used to judge the quality of the generated image, and the discriminator loss is used to iteratively update the model parameters in the initial image enhancement model. The discriminant network can enhance the difference of the discrimination through parallel loss calculation.

[0102] In this example implementation, Figure 4 As shown in Figure 1, the output sample image is input into the discriminator to obtain the corresponding discriminator loss, and the model parameters in the initial image enhancement model are iteratively updated based on the discriminator loss. Specifically, the following steps may be included:

[0103] Step S410: Input the output sample image into the discriminator, and obtain the first output result of the discriminator through the image semantic segmentation network in the discriminator, and obtain the second output result of the discriminator through the cross network in the discriminator.

[0104] In the example embodiment, the discriminator is designed to identify the original training sample image as "true" and the output sample image as "false", and the goal of the generator composed of the image semantic segmentation network and the network attention focusing module is to confuse the discriminator to misidentify the output sample image generated by the image semantic segmentation network as "true", so as to achieve the purpose of image reconstruction.

[0105] In the example embodiment, the training sample image and the output sample image are input into the discriminator, and the features of the image are extracted by the discriminator network to determine whether the image input into the discriminator is "true" or "false". Specifically, "0" or "False" can be used to represent "false", and "1" or "True" can be used to represent "true". The discriminator network can be divided into two branches. One branch uses the Unet mode to obtain the overall features of the image through the image semantic segmentation network, and obtains the first output result of the discriminator according to the overall features of the image. The other branch uses the cross network to obtain the detailed features of the image and obtains the second output result of the discriminator. By combining the output results of the Unet network and the cross network, the overall features and the detailed features of the image can be better fused, thereby improving the enhancement effect of the image.

[0106] Step S420. Iteratively update the model parameters in the initial image enhancement model based on the first output result of the discriminator and the second output result of the discriminator.

[0107] Finally, based on the first output result and the second output result of the discriminator, the overall loss of the image enhancement model can be obtained. The neural network parameters in the image enhancement model can be iteratively updated through the back propagation of the overall loss of the image enhancement model, so that the image semantic segmentation network generates an image that is closer to the original training sample image. The iteration stops when the discriminator cannot distinguish whether the input image is the image generated by the image semantic segmentation network or the original training sample image, thereby realizing the training process of the image enhancement model and obtaining the trained image enhancement model.

[0108] As Figure 5 Fig. 1 shows the overall framework of the training method of the image enhancement model in one specific embodiment of the present disclosure, which is an example illustration of the above steps in the example embodiment. The overall framework describes the model training process. The image enhancement model adopts the Unet+NATM structure, and the specific training method includes: after the training sample image is randomly superimposed by self-training degradation and random degradation, an input sample image 501 is obtained. After the input sample image 501 is input into the network, the network generates an output sample image 502 through Unet. After the output sample image 502 is discriminated by the discriminator, the model is iteratively trained based on the discriminator loss.

[0109] AsFigure 6 The figure shows a schematic diagram of an image degradation method in a specific embodiment of the present disclosure. Image degradation processing can include two methods: self-training degradation and random degradation. Among them, the random degradation method can include noise, blurring, scaling and compression, and the self-training degradation method can include shallow blurring, mid-level blurring and high-level blurring.

[0110] like Figure 7 Figure 1 shows a schematic diagram of random degradation methods in a specific embodiment of the present disclosure. Random degradation methods can be categorized into four types: blur, scaling, noise, and compression. Blur can be categorized into Gaussian blur, mean blur, and anisotropic blur; scaling can be categorized into bilinear scaling, trilinear scaling, and regional scaling; noise can be categorized into Gaussian noise, color noise, Poisson noise, and grayscale noise; and compression can include JEPG compression.

[0111] like Figure 8 Figure 1 shows a schematic diagram of the attention module in one embodiment of the present disclosure. This NATM module can be used to perform feature attention processing, enhancing the model's perceptual learning capabilities. Applying the attention mechanism to the Unet segmentation network can effectively focus on salient regions and suppress irrelevant background regions.

[0112] like Figure 9 The figure shows a schematic diagram of the structure of the discriminator network in a specific embodiment of the present disclosure. The discriminator network can be divided into two branches. One branch uses the Unet model to obtain the overall features of the output sample image through the image semantic segmentation network and obtains the first output result of the discriminator based on the overall features of the image; the other branch uses a cross network to obtain the detailed features of the output sample image and obtain the second output result of the discriminator. The input size of the two branches of the discriminator network is 512x512 respectively. By combining the discriminant results output by the Unet network and the cross network and then providing feedback, the overall and detailed features of the image can be better integrated, thereby improving the image enhancement effect.

[0113] On the other hand, this exemplary embodiment also provides an image processing method. Figure 10 As shown, the above image processing method may include the following steps:

[0114] Step S1010: Obtain the original image to be processed, and input the original image into a pre-trained image enhancement model.

[0115] Among them, the image enhancement model can be Figures 1 to 4 The training method of the image enhancement model in is obtained.

[0116] Step S1020. Perform image enhancement processing on the original image through the image enhancement model to obtain an enhanced image corresponding to the original image.

[0117] In the example embodiment, as shown in Figure 11 the image enhancement model to obtain an enhanced image corresponding to the original image, which can include the following steps:

[0118] Step S1110. Obtain the feature map corresponding to each feature extraction layer of the original image through the multiple feature extraction layers in the image semantic segmentation network.

[0119] First, obtain the feature map corresponding to each feature extraction layer of the original image through the multiple feature extraction layers on the left side of the image semantic segmentation network.

[0120] Step S1120. Determine the channel attention coefficients corresponding to each channel and the region attention coefficients corresponding to each region in the feature map corresponding to the feature extraction layer through the attention concentration module corresponding to each feature extraction layer in the image semantic segmentation network.

[0121] When connecting the feature map obtained by each feature extraction layer to the corresponding up-sampling layer, the channel attention coefficients corresponding to each channel and the region attention coefficients corresponding to each region in the feature map corresponding to each feature extraction layer can be adjusted through the attention concentration module corresponding to each feature extraction layer.

[0122] Step S1130. Obtain the output image of the image enhancement model through the multiple up-sampling layers in the image semantic segmentation network based on the channel attention coefficients and the region attention coefficients corresponding to each feature extraction layer and the feature map.

[0123] After adjusting the channel focused attention and the spatial focused attention of the feature map corresponding to each feature extraction layer based on the channel attention coefficients and the region attention coefficients corresponding to each feature extraction layer, connecting to the corresponding up-sampling layer, and then obtaining the output image corresponding to the original image through the multiple up-sampling layers.

[0124] Step S1140. Obtain the enhanced image corresponding to the original image according to the output image of the image enhancement model.

[0125] Specifically, the output image of the image enhancement model can be directly used as the enhanced image corresponding to the original image, or the image interpolation method can be used to perform size enlargement processing on the output image of the image enhancement model to obtain the final enhanced image.

[0126] It should be noted that although the various steps of the methods of the present disclosure are described in a particular order in the accompanying drawings, this is not required or implied that the steps must be performed in that particular order, or that all of the steps shown must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into a single step, and / or a single step can be broken down into multiple steps, etc.

[0127] Further, the present disclosure also provides a training device of an image enhancement model. Referring to Figure 12 As shown in the figure, the training device of the image enhancement model can include an image degradation processing module 1210, an enhancement model construction module 1220, and an enhancement model training module 1230. Among them:

[0128] The image degradation processing module 1210 can be used to obtain a training data set of an image enhancement model, and perform image degradation processing on a training sample image in the training data set to obtain an input sample image of the image enhancement model;

[0129] The enhancement model construction module 1220 can be used to add an attention focusing module to an image semantic segmentation network to construct an initial image enhancement model, and input the input sample image into the initial image enhancement model to obtain an output sample image corresponding to the input sample image;

[0130] The enhancement model training module 1230 can be used to input the output sample image into a discriminator to obtain a corresponding discriminator loss, and iteratively update the model parameters in the initial image enhancement model based on the discriminator loss to obtain a trained image enhancement model.

[0131] In some exemplary embodiments of the present disclosure, the image degradation processing module 1210 can include a degradation method set obtaining unit, a target degradation method determining unit, and an image degradation processing unit.

[0132] Among them:

[0133] The degradation method set obtaining unit can be used to obtain a self-training degradation method set and a random degradation method set for image degradation processing;

[0134] The target degradation method determining unit can be used to determine a target self-training degradation method from the self-training degradation method set, and determine a target random degradation method from the random degradation method set;

[0135] The image degradation processing unit can be used to perform image degradation processing on the training sample image through the target self-training degradation method and the target random degradation method to obtain an input sample image corresponding to the training sample image.

[0136] In some example embodiments of the present disclosure, the enhanced model construction module 1220 can include a feature map extraction unit, an attention coefficient adjustment unit, and a feature upsampling unit. Among them:

[0137] The feature map extraction unit can be used to obtain the feature map of the input sample image corresponding to each feature extraction layer in the image semantic segmentation network in sequence through a plurality of feature extraction layers in the image semantic segmentation network;

[0138] The attention coefficient adjustment unit can be used to adjust the channel attention coefficients corresponding to each channel and the region attention coefficients corresponding to each region in the feature map corresponding to each feature extraction layer through the attention concentration module corresponding to each feature extraction layer in the image semantic segmentation network;

[0139] The feature upsampling unit can be used to obtain the output sample image corresponding to the input sample image through a plurality of upsampling layers in the image semantic segmentation network based on the channel attention coefficients and the region attention coefficients corresponding to each feature extraction layer and the feature map.

[0140] In some example embodiments of the present disclosure, the enhanced model training module 1230 can include a discriminator loss determination unit and a model parameter updating unit. Among them:

[0141] The discriminator loss determination unit can be used to input the output sample image into the discriminator, and obtain the first output result of the discriminator through the image semantic segmentation network in the discriminator, and obtain the second output result of the discriminator through the cross network in the discriminator;

[0142] The model parameter updating unit can be used to iteratively update the model parameters in the initial image enhancement model based on the first output result of the discriminator and the second output result of the discriminator.

[0143] Further, the present disclosure also provides an image processing device. Referring to Figure 13 The image processing device can include an original image input module 1310 and an image enhancement processing module 1320. Among them:

[0144] The original image input module 1310 can be used to obtain the original image to be processed, and input the original image into the pre-trained image enhancement model, wherein the image enhancement model is obtained by the image enhancement model training device as above;

[0145] The image enhancement processing module 1320 can be used to perform image enhancement processing on the original image through the image enhancement model to obtain an enhanced image corresponding to the original image.

[0146] In some example embodiments of the present disclosure, the image enhancement processing module 1320 can include an original image feature map extraction unit, an attention coefficient determination unit, an output image determination unit, and an enhanced image determination unit. Among them:

[0147] The original image feature map extraction unit can be configured to sequentially obtain a feature map of the original image corresponding to each feature extraction layer in the image semantic segmentation network through a plurality of feature extraction layers in the image semantic segmentation network;

[0148] The attention coefficient determination unit can be configured to determine the channel attention coefficients corresponding to each channel and the region attention coefficients corresponding to each region in the feature map corresponding to each feature extraction layer in the image semantic segmentation network through the attention concentration module corresponding to each feature extraction layer in the image semantic segmentation network;

[0149] The output image determination unit can be configured to obtain the output image of the image enhancement model through a plurality of up-sampling layers in the image semantic segmentation network based on the channel attention coefficients and the region attention coefficients corresponding to each feature extraction layer and the feature map;

[0150] The enhanced image determination unit can be configured to obtain the enhanced image corresponding to the original image according to the output image of the image enhancement model.

[0151] The specific details of each module / unit in the above image enhancement model training device and image processing device have been described in detail in the corresponding method embodiment part, and will not be repeated here.

[0152] Figure 14 A structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present disclosure is shown.

[0153] It should be noted that, Figure 14 The computer system 1400 of the electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0154] As Figure 14 shown, the computer system 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1402 or programs loaded from a storage portion 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for system operation are also stored. The CPU 1401, the ROM 1402, and the RAM 1403 are connected to each other through a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0155] The following components are connected to the I / O interface 1405: an input part 1406 including a keyboard, a mouse, etc.; an output part 1407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 1408 including a hard disk, etc.; and a communication part 1409 including a network interface card such as a LAN card, a modem, etc. The communication part 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as necessary. A removable media 1411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1410 as necessary, so that a computer program read therefrom is installed in the storage part 1408 as necessary.

[0156] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication part 1409, and / or installed from the removable media 1411. When the computer program is executed by the central processing unit (CPU) 1401, various functions defined in the system of the present application are executed.

[0157] Note that the computer-readable medium shown in the disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In the disclosure, the computer-readable signal medium can include a data signal that propagates in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal can take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can be used to carry or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including, but not limited to, wireless, wireline, optical fiber, RF, etc., or any suitable combination of the above.

[0158] The flow diagrams and block diagrams in the drawings are illustrations of possible architectures, functions, and operations of systems, methods, and computer program products in accordance with various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0159] As another aspect, the present application also provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method described in the above embodiments.

[0160] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, such a division is not mandatory. Indeed, according to embodiments of the present disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into modules embodied by several modules.

[0161] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the present disclosure including modifications and alterations thereof. It is intended that the present application encompass all such variations as fall within the scope of the present disclosure, which is defined by the following claims.

[0162] It is to be understood that the present disclosure is not limited to the precise construction described above and shown in the drawings, and that various modifications and changes can be effected therein by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims appended hereto.

Claims

1. A training method for an image enhancement model, characterized in that: include: Obtaining a training data set for the image enhancement model, and performing image degradation processing on training sample images in the training data set to obtain input sample images for the image enhancement model; The performing image degradation processing on the training sample images in the training data set to obtain the input sample images of the image enhancement model includes: obtaining a self-training degradation method set and a random degradation method set for image degradation processing; determining a target self-training degradation method from the self-training degradation method set, and determining a target random degradation method from the random degradation method set; performing image degradation processing on the training sample images using the target self-training degradation method and the target random degradation method to obtain the input sample images corresponding to the training sample images; Adding an attention focus module to the image semantic segmentation network to construct an initial image enhancement model, and inputting the input sample image into the initial image enhancement model to obtain an output sample image corresponding to the input sample image; The output sample image is input into a discriminator to obtain a corresponding discriminator loss, and the model parameters in the initial image enhancement model are iteratively updated based on the discriminator loss to obtain the trained image enhancement model.

2. The image enhancement model training method according to claim 1, characterized in that: Inputting the input sample image into the initial image enhancement model to obtain an output sample image corresponding to the input sample image includes: Obtaining, in sequence, a feature map corresponding to each feature extraction layer of the input sample image through multiple feature extraction layers in the image semantic segmentation network; Adjusting the channel attention coefficient corresponding to each channel and the region attention coefficient corresponding to each region in the feature map corresponding to the feature extraction layer through the attention concentration module corresponding to each feature extraction layer in the image semantic segmentation network; Based on the channel attention coefficients and the regional attention coefficients corresponding to each of the feature extraction layers and the feature map, an output sample image corresponding to the input sample image is obtained through multiple upsampling layers in the image semantic segmentation network.

3. The image enhancement model training method according to claim 1, characterized in that: Inputting the output sample image into a discriminator to obtain a corresponding discriminator loss, and iteratively updating model parameters in the initial image enhancement model based on the discriminator loss, comprising: Inputting the output sample image into a discriminator, and obtaining a first output result of the discriminator through an image semantic segmentation network in the discriminator, and obtaining a second output result of the discriminator through a cross network in the discriminator; Iteratively updating model parameters in the initial image enhancement model based on the first output result of the discriminator and the second output result of the discriminator.

4. An image processing method, characterized in that: include: Obtaining an original image to be processed, and inputting the original image into a pre-trained image enhancement model, wherein the image enhancement model is obtained by the image enhancement model training method according to any one of claims 1 to 3; The image enhancement model is used to perform image enhancement processing on the original image to obtain an enhanced image corresponding to the original image.

5. The image processing method according to claim 4, characterized in that The performing image enhancement processing on the original image by using the image enhancement model to obtain an enhanced image corresponding to the original image includes: Obtaining, in sequence, a feature map of the original image corresponding to each feature extraction layer through multiple feature extraction layers in the image semantic segmentation network; Determine, by means of an attention concentration module corresponding to each feature extraction layer in the image semantic segmentation network, a channel attention coefficient corresponding to each channel and a region attention coefficient corresponding to each region in the feature map corresponding to the feature extraction layer; Based on the channel attention coefficients and the regional attention coefficients corresponding to the feature extraction layers and the feature map, obtaining an output image of the image enhancement model through multiple upsampling layers in the image semantic segmentation network; An enhanced image corresponding to the original image is obtained according to the output image of the image enhancement model.

6. A training device for an image enhancement model, characterized in that: include: An image degradation processing module is used to obtain a training data set for the image enhancement model and perform image degradation processing on the training sample images in the training data set to obtain input sample images for the image enhancement model; The performing image degradation processing on the training sample images in the training data set to obtain the input sample images of the image enhancement model includes: obtaining a self-training degradation method set and a random degradation method set for image degradation processing; determining a target self-training degradation method from the self-training degradation method set, and determining a target random degradation method from the random degradation method set; performing image degradation processing on the training sample images using the target self-training degradation method and the target random degradation method to obtain the input sample images corresponding to the training sample images; An enhancement model construction module is used to add an attention focus module to the image semantic segmentation network to construct an initial image enhancement model, and input the input sample image into the initial image enhancement model to obtain an output sample image corresponding to the input sample image; The enhanced model training module is used to input the output sample image into the discriminator to obtain the corresponding discriminator loss, and iteratively update the model parameters in the initial image enhancement model based on the discriminator loss to obtain the trained image enhancement model.

7. The image enhancement model training device according to claim 6, characterized in that: The enhanced model building module includes: A feature map extraction unit, configured to sequentially obtain a feature map corresponding to each feature extraction layer of the input sample image through multiple feature extraction layers in the image semantic segmentation network; An attention coefficient adjustment unit is configured to adjust the channel attention coefficient corresponding to each channel and the region attention coefficient corresponding to each region in the feature map corresponding to the feature extraction layer through an attention concentration module corresponding to each feature extraction layer in the image semantic segmentation network; A feature upsampling unit is used to obtain an output sample image corresponding to the input sample image through multiple upsampling layers in the image semantic segmentation network based on the channel attention coefficient and the regional attention coefficient corresponding to each feature extraction layer and the feature map.

8. The image enhancement model training device according to claim 6, characterized in that: The enhanced model training module includes: a discriminator loss determination unit, configured to input the output sample image into the discriminator, obtain a first output result of the discriminator through an image semantic segmentation network in the discriminator, and obtain a second output result of the discriminator through a cross network in the discriminator; A model parameter updating unit is used to iteratively update the model parameters in the initial image enhancement model based on the first output result of the discriminator and the second output result of the discriminator.

9. An image processing device, characterized in that: include: an original image input module, configured to obtain an original image to be processed and input the original image into a pre-trained image enhancement model, wherein the image enhancement model is obtained by the image enhancement model training device as claimed in claim 6; The image enhancement processing module is used to perform image enhancement processing on the original image through the image enhancement model to obtain an enhanced image corresponding to the original image.

10. The image processing device according to claim 9, wherein The image enhancement processing module includes: An original image feature map extraction unit is used to sequentially obtain a feature map corresponding to each feature extraction layer of the original image through multiple feature extraction layers in the image semantic segmentation network; An attention coefficient determination unit is configured to determine, through an attention concentration module corresponding to each feature extraction layer in the image semantic segmentation network, a channel attention coefficient corresponding to each channel and a region attention coefficient corresponding to each region in the feature map corresponding to the feature extraction layer; an output image determining unit, configured to obtain an output image of the image enhancement model through a plurality of upsampling layers in the image semantic segmentation network based on the channel attention coefficients and the region attention coefficients corresponding to the feature extraction layers and the feature map; The enhanced image determining unit is used to obtain an enhanced image corresponding to the original image according to the output image of the image enhancement model.

11. An electronic device, characterized in that: include: processor; as well as A memory for storing one or more programs, which, when executed by the processor, enables the processor to implement the training method of the image enhancement model as described in any one of claims 1 to 3, or the image processing method as described in any one of claims 4 to 5.

12. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the training method of the image enhancement model as described in any one of claims 1 to 3 or the image processing method as described in any one of claims 4 to 5 is implemented.

Images