Image quality enhancement method and device, electronic equipment and storage medium

Abstract

The image quality enhancement method, device, electronic equipment and storage medium provided by the present application relate to the technical field of image processing. The method comprises the following steps: based on the semantic prior of a preset service scenario, performing semantic segmentation on an original high-quality image in an original image set to obtain a semantic segmentation result; according to the semantic segmentation result, performing data enhancement on an original low-quality image in the original image set to obtain a data enhancement result; inputting the data enhancement result into a pre-constructed image quality enhancement model for training to obtain a trained image quality enhancement model; and using the trained image quality enhancement model to perform quality enhancement on an image in the preset service scenario, thereby avoiding directly using the original low-quality image as input data for training the image quality enhancement model and improving the actual application effect of the trained image quality enhancement model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image quality enhancement method and device, electronic equipment and storage medium. BACKGROUND

[0002] The image quality enhancement technology is essentially to enhance all edge texture details in the image. The current scheme is to directly input a low-quality image into a quality enhancement model for training, and then use the corresponding high-quality image as a label to optimize the model parameters to obtain a final quality enhancement model. When the quality enhancement model is applied in an actual business scenario, it can only generally process the entire image, and the enhancement effect is not good, which is difficult to meet the business requirements. SUMMARY

[0003] In order to overcome the deficiencies of the prior art, the embodiments of the present application provide an image quality enhancement method, device, electronic equipment and storage medium, which can.

[0004] The technical scheme of the present application can be implemented as follows:

[0005] In a first aspect, the present application provides an image quality enhancement method, which comprises:

[0006] Based on the semantic prior of the preset business scenario, the original high-quality image in the original image set is subjected to semantic segmentation to obtain a semantic segmentation result;

[0007] According to the semantic segmentation result, the original low-quality image in the original image set is subjected to data enhancement to obtain a data enhancement result;

[0008] The data enhancement result is input into a pre-constructed image quality enhancement model for training to obtain a trained image quality enhancement model;

[0009] The trained image quality enhancement model is used to perform quality enhancement on the image in the preset business scenario.

[0010] Optionally, the semantic segmentation result includes a first target semantic region mask image, a second target semantic region mask image and a transition region mask image, and the step of data enhancement of the original low-quality image in the original image set according to the semantic segmentation result to obtain a data enhancement result comprises:

[0011] Based on the first target semantic region mask image, the original low-quality image and the original high-quality image are fused to obtain a first low-quality image, and the first low-quality image has clear semantic regions;

[0012] fuse the original low-quality image and the original high-quality image based on the second target semantic region mask image to obtain a second low-quality image, the second low-quality image having a clear background region;

[0013] fuse the original low-quality image and the original high-quality image based on the transition region mask image to obtain a third low-quality image, the third low-quality image having a transition region meeting a preset definition, and the data enhancement result comprising the first low-quality image, the second low-quality image and the third low-quality image.

[0014] Optionally, the step of fusing the original low-quality image and the original high-quality image based on the first target semantic region mask image to obtain a first low-quality image comprises:

[0015] processing the original high-quality image using the first target semantic region mask image to obtain a high-quality first target semantic region;

[0016] processing the original low-quality image and the original high-quality image using the second target semantic region mask image respectively to obtain a low-quality second target semantic region and a high-quality second target semantic region;

[0017] performing weighted fusion on the low-quality second target semantic region and the high-quality second target semantic region, and combining the fusion result with the high-quality semantic image to obtain the first low-quality image.

[0018] Optionally, the step of fusing the original low-quality image and the original high-quality image based on the second target semantic region mask image to obtain a second low-quality image further comprises:

[0019] processing the original high-quality image using the second target semantic region mask image to obtain a high-quality second target semantic region;

[0020] processing the original low-quality image and the original high-quality image using the first target semantic region mask image respectively to obtain a low-quality first target semantic region and a high-quality first target semantic region;

[0021] performing weighted fusion on the low-quality first target semantic region and the high-quality first target semantic region, and combining the fusion result with the high-quality second target semantic region to obtain the second low-quality image.

[0022] Optionally, the step of fusing the original low-quality image and the original high-quality image based on the transition region mask image to obtain a third low-quality image further comprises:

[0023] generate a non-transition region mask image based on the first target semantic region mask image, the second target semantic region mask image, and the transition region mask image;

[0024] process the original high-quality image using the non-transition region mask image to obtain a high-quality non-transition region;

[0025] process the original low-quality image and the original high-quality image using the transition region mask image respectively to obtain a low-quality transition region and a high-quality transition region;

[0026] perform weighted fusion on the low-quality transition region and the high-quality transition region, and combine the fusion result with the high-quality non-transition region to obtain the third low-quality image.

[0027] Optionally, the data enhancement result includes a plurality of enhanced image sets corresponding to different weighting parameters, and the step of inputting the data enhancement result into the pre-constructed image quality enhancement model for training to obtain the trained image quality enhancement model includes:

[0028] in order from small to large, each of the enhanced image sets is input into the pre-constructed image quality enhancement model for training to obtain the image quality enhancement model to be adjusted.

[0029] Optionally, the method further includes:

[0030] in the case where the weighting parameter corresponding to the enhanced image set input into the model is a preset maximum value, the enhanced image set input into the model is updated using the original low-quality image, and the trained image quality enhancement model is adjusted using the updated enhanced image set to minimize the loss function of the trained image quality enhancement model.

[0031] In a second aspect, the present application provides an image quality enhancement device, which includes:

[0032] a semantic segmentation module configured to perform semantic segmentation on an original high-quality image in an original image set based on semantic priors of a preset service scenario to obtain a semantic segmentation result;

[0033] a data enhancement module configured to perform data enhancement on an original low-quality image in the original image set according to the semantic segmentation result to obtain a data enhancement result;

[0034] a model training module configured to input the data enhancement result into a pre-constructed image quality enhancement model for training to obtain a trained image quality enhancement model;

[0035] a quality enhancement module configured to perform quality enhancement on images in the preset service scenario using the trained image quality enhancement model.

[0036] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the image quality enhancement method according to the first aspect.

[0037] In a fourth aspect, the present application provides a computer readable storage medium storing a computer program, and the computer program is executed to implement the image quality enhancement method according to the first aspect.

[0038] Compared with the prior art, the image quality enhancement method provided by the present application has the following advantages: based on the semantic prior of a preset service scenario, the original high-quality images in an original image set are subjected to semantic segmentation to obtain a semantic segmentation result; according to the semantic segmentation result, the original low-quality images in the original image set are subjected to data enhancement to obtain a data enhancement result; the data enhancement result is input into a pre-constructed image quality enhancement model for training to obtain a trained image quality enhancement model; and the trained image quality enhancement model is used to perform quality enhancement on images in the preset service scenario. Since the data used for training the image quality enhancement model in the present application is obtained by subjecting the original high-quality images to semantic segmentation and then using the semantic segmentation result to perform data enhancement on the original low-quality images, the original low-quality images are not directly used as input data for training the image quality enhancement model, thereby improving the actual application effect of the trained image quality enhancement model. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0040] Figure 1 An existing image quality enhancement model training example diagram provided for the embodiments of the present application;

[0041] Figure 2 A flowchart of an image quality enhancement method provided for the embodiments of the present application;

[0042] Figure 3 An image quality enhancement model training example diagram provided for the embodiments of the present application;

[0043] Figure 4 A functional unit block diagram of an image quality enhancement device provided for the embodiments of the present application;

[0044] Figure 5A structural schematic block diagram of an electronic device provided by an embodiment of the present application is shown.

[0045] Icon: 100-image quality enhancement device; 101-semantic segmentation module; 102-data enhancement module; 103-model training module; 104-quality enhancement module; 200-electronic device; 210-memory; 220-processor. DETAILED DESCRIPTION

[0046] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0047] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0048] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0049] In addition, if the terms "first", "second" and the like are used, they are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0050] It should be noted that: the features in the embodiments of the present application can be combined with each other without conflict.

[0051] As Figure 1 shown, the existing image quality enhancement method is usually to directly input the original low-quality image into the quality enhancement model, output the reconstructed image, and then use the corresponding original high-quality image as the label to optimize the model parameters to obtain the final quality enhancement model.

[0052] The final quality enhancement model can only generally process the whole image when applied in actual business scenarios, and cannot perform specific enhancement on specific regions, resulting in poor enhancement effect in the target region and difficulty in meeting business requirements.

[0053] In order to improve the image quality enhancement effect in the business scenario, an image quality enhancement method is provided in the embodiments of the present application, which will be described in detail below.

[0054] Please refer to Figure 2 The image quality enhancement method includes steps S101-S104.

[0055] S101, based on the semantic prior of the preset business scenario, performing semantic segmentation on the original high-quality image in the original image set to obtain a semantic segmentation result.

[0056] The preset business scenario can be an entertainment live streaming scenario, a city traffic scenario, etc. A semantic segmentation model corresponding to the preset business scenario can be used to segment multiple semantic regions from the image and obtain the corresponding semantic region mask image.

[0057] For example, in a city traffic scenario, a semantic segmentation model corresponding to the business scenario can be used to segment multiple semantic regions such as a pedestrian region, a vehicle region, a road region, and a background region, and obtain the corresponding pedestrian region mask image, vehicle region mask image, road region mask image, and background region mask image.

[0058] The original image set includes a plurality of image pairs, each image pair including one frame of original low-quality image and one frame of original high-quality image, and the content of the original low-quality image and the original high-quality image belonging to the same image pair is the same. In order to ensure the accuracy of the semantic segmentation result, the original high-quality image containing more detailed information can be selected for semantic segmentation.

[0059] Illustratively, assuming that the preset business scenario is an entertainment live streaming scenario, in which the audience usually pays more attention to the host, the image of the scenario can be divided into two semantic regions of host portrait and background. For the original high-quality image in each image pair contained in the original image set, a portrait segmentation model can be used to segment the host portrait region mask image F∈R H×W and the background region mask image B∈R H×W The segmentation process can be as follows:

[0060] For a position (i, j) on the original high-quality image, if the pixel at the position belongs to the host portrait region, then in the host portrait region mask image F∈R H×W , the pixel F(i, j) at the position is 1, and in the second target semantic region mask image B∈R H×W , the pixel B(i, j) at the position is 0.

[0061] Similarly, if the pixel at the position belongs to the background region, then in the host portrait region mask image F∈R H×W , the pixel F(i, j) at the position is 0, and in the second target semantic region mask image B∈R H×W , the pixel B(i, j) at the position is 1.

[0062] In the segmentation of the anchor person portrait region mask image F ∈ R H×W And the background region mask image B ∈ R H×W , for the transition region between the anchor person portrait region and the background region, the corresponding transition region mask image S ∈ R H×W also needs to be generated.

[0063] For the position (i, j) on the original high-quality image, if the pixel at this position belongs to the transition region between the anchor person portrait region and the background region, the pixel S(i, j) at this position on the transition region mask image S ∈ R H×W is 1, otherwise, if the pixel at this position does not belong to the transition region, the pixel S(i, j) at this position on the transition region mask image S ∈ R H×W is 0.

[0064] The transition region mask image S ∈ R H×W can also be Gaussian blurred to expand the range of the boundary region, and after Gaussian blur, the pixels of each position higher than the set threshold ε are set to 1, and the pixels of other positions are set to 0, to obtain the final transition region mask image S ∈ R H×W .

[0065] From the above segmentation process, on the anchor person portrait region mask image F ∈ R H×W , for the pixel F(i, j) at position (i, j), if F(i, j) is 1, it represents that the position belongs to the anchor person portrait region, and if F(i, j) is 0, it represents that the position does not belong to the anchor person portrait region.

[0066] On the background region mask image B ∈ R H×W , for the pixel B(i, j) at position (i, j), if B(i, j) is 1, it represents that the position belongs to the background region, and if B(i, j) is 0, it represents that the position does not belong to the background region.

[0067] On the transition region mask image S ∈ R H×W , for the pixel S(i, j) at position (i, j), if S(i, j) is 1, it represents that the position belongs to the transition region between the anchor person portrait region and the background region, and if S(i, j) is 0, it represents that the position does not belong to the transition region.

[0068] S102, according to the semantic segmentation result, data augmentation is performed on the original low-quality image in the original image set to obtain a data augmentation result.

[0069] Since any image in any business scenario can be divided into at least two semantic regions, the focus of image quality enhancement is often the target semantic region which is more relevant to the purpose of the image among all semantic regions.

[0070] For example, an image in a city traffic scenario can be divided into a pedestrian region, a vehicle region, a road region, a background region and the like, and road safety is mainly related to the pedestrian region and the vehicle region, so the pedestrian region and the vehicle region can be taken as the target semantic regions for image quality enhancement.

[0071] For an entertainment live streaming scenario, an image in the scenario can be divided into a host portrait region and a background region, and the host portrait region and the background region can be taken as the target semantic regions for image quality enhancement.

[0072] In addition, in order to guarantee the overall effect of the image quality enhanced image, the transition region between the target semantic regions needs to be paid attention to.

[0073] In an embodiment of the present application, the semantic segmentation result includes a first target semantic region mask image, a second target semantic region mask image and a transition region mask image.

[0074] The first target semantic region mask image, the second target semantic region mask image and the transition region mask image are used to perform data enhancement on the original low-quality image in each image pair contained in the original image set, to obtain a data enhancement result for training the image quality enhancement model.

[0075] The data enhancement result corresponding to the original low-quality image in each image pair contained in the original image set includes a first low-quality image, a second low-quality image and a third low-quality image, wherein the first low-quality image has a clear first target semantic region and a second target semantic region satisfying a preset clarity, the second low-quality image has a clear second target semantic region and a first target semantic region satisfying a preset clarity, and the third low-quality image has a clear non-transition region and a transition region satisfying a preset clarity.

[0076] The implementation process of step S102 will be described in detail below based on the following settings.

[0077] Suppose that the preset business scenario is an entertainment live streaming scenario, and the original low-quality image in each image pair contained in the original image set is I LR ∈R H×W , and the corresponding original high-quality image is I HR ∈R H×W .

[0078] The host portrait region mask image F ∈ R HR ∈R H×W is divided from I H×Wand background region mask image B∈R H×W These are the first target semantic region mask image and the second target semantic region mask image, respectively.

[0079] From I HR ∈R H×W The above also segments a transition region mask image, S∈R, for the transition region between the anchor's portrait area and the background area. H×W .

[0080] Step S102 may include sub-steps S102-1 to S102-3.

[0081] S102-1, Based on the first target semantic region mask image, the original low-quality image and the original high-quality image are fused to obtain the first low-quality image.

[0082] Understandably, in entertainment live streaming scenarios, the first low-quality image includes a clear image of the anchor and a background area that meets the preset sharpness, which is related to the weighting parameters during the fusion process.

[0083] In a possible implementation, step S102-1 can be implemented as follows:

[0084] S102-1a, the original high-quality image is processed using the mask image of the first target semantic region to obtain the high-quality first target semantic region.

[0085] For any position on the original high-quality image, if the pixel at that position is 1 on the first target semantic region mask image, then the pixel at that position on the original high-quality image is retained; if the pixel at that position is 0 on the first target semantic region mask image, then the pixel at that position on the original high-quality image is not retained. The positions of the retained pixels on the original high-quality image constitute the high-quality first target semantic region.

[0086] Understandably, in entertainment live streaming scenarios, the mask image F∈R of the streamer's portrait region is used. H×W For the original high-quality image I HR ∈R H×W Processing was performed to obtain a high-quality anchor portrait area. in This indicates pixel-wise multiplication.

[0087] S102-1b, using the mask image of the second target semantic region to process the original low-quality image and the original high-quality image respectively, to obtain the low-quality second target semantic region and the high-quality second target semantic region;

[0088] For any position on the original low-quality image, if the pixel of the position on the second target semantic region mask image is 1, the pixel of the position on the original low-quality image is reserved, if the pixel of the position on the second target semantic region mask image is 0, the pixel of the position on the original low-quality image is not reserved, and the positions of the reserved pixels on the original low-quality image form the low-quality second target semantic region.

[0089] Understandably, in the entertainment live broadcast scene, the background region mask image B ∈ R H×W The original low-quality image I LR ∈ R H×W is processed to obtain the low-quality background region

[0090] For any position on the original high-quality image, if the pixel of the position on the second target semantic region mask image is 1, the pixel of the position on the original high-quality image is reserved, if the pixel of the position on the second target semantic region mask image is 0, the pixel of the position on the original high-quality image is not reserved, and the positions of the reserved pixels on the original high-quality image form the high-quality second target semantic region.

[0091] Understandably, in the entertainment live broadcast scene, the background region mask image B ∈ R H×W The original high-quality image I HR ∈ R H×W is processed to obtain the high-quality background region

[0092] S102-1c, the low-quality second target semantic region and the high-quality second target semantic region are weightedly fused, and the fusion result is combined with the high-quality first target semantic region to obtain a first low-quality image.

[0093] In the entertainment live broadcast scene, the first low-quality image The high-quality anchor person portrait region The low-quality background region And the high-quality background region Satisfy the following formula:

[0094]

[0095] Wherein, alpha ∈ [0, 1] represents a weighting parameter.

[0096] Understandably, when alpha = 0, the above formula can be degenerated into That is, the anchor person portrait region and the background region on the first low-quality image are both from the original high-quality image, and correspondingly, when subsequent model training is performed, the model can learn the identity mapping;

[0097] When a = 1, the above formula can degenerate into That is, the background region in the original high-quality image is completely replaced by the background region of the original low-quality image.

[0098] When 0 < a < 1, the background region on the first low-quality image is obtained by weighted summation of the background region of the original low-quality image and the background region of the original high-quality image.

[0099] Obviously, when the value of a gradually increases from 0 to 1, the quality of the background region on the first low-quality image gradually decreases, and correspondingly, when the model is trained, the difficulty of model learning gradually increases with the increase of the training step number, so that the trained model is convergent after reaching the quality of the original low-quality image.

[0100] S102-2, based on the second target semantic region mask image, fusing the original low-quality image and the original high-quality image to obtain a second low-quality image.

[0101] It can be understood that in the entertainment live broadcast scene, the second low-quality image includes a clear background region and an anchor portrait region meeting a preset definition, and the preset definition is related to the weighting parameter in the fusion processing.

[0102] In a possible implementation, the implementation process of step S102-2 can be as follows:

[0103] S102-2a, processing the original high-quality image by using the second target semantic region mask image to obtain a high-quality second target semantic region.

[0104] For any position on the original high-quality image, if the pixel of the position on the second target semantic region mask image is 1, the pixel of the position on the original high-quality image is retained, and if the pixel of the position on the second target semantic region mask image is 0, the pixel of the position on the original high-quality image is not retained, and the positions of the retained pixels on the original high-quality image form the high-quality second target semantic region.

[0105] It can be understood that in the entertainment live broadcast scene, the background region mask image B ∈ R H×W The original high-quality image I HR ∈ R H×W is processed to obtain a high-quality background region

[0106] S102-2b, processing the original low-quality image and the original high-quality image by using the first target semantic region mask image respectively to obtain a low-quality first target semantic region and a high-quality first target semantic region.

[0107] For any position on the original low-quality image, if the pixel of the position on the first target semantic region mask image is 1, the pixel of the position on the original low-quality image is reserved, if the pixel of the position on the first target semantic region mask image is 0, the pixel of the position on the original low-quality image is not reserved, and the positions of the reserved pixels on the original low-quality image form the low-quality first target semantic region.

[0108] Understandably, in the entertainment live broadcast scene, the anchor portrait region mask image F is used to process the original low-quality image I H×W LR ∈R H×W to obtain the low-quality anchor portrait region

[0109] For any position on the original high-quality image, if the pixel of the position on the first target semantic region mask image is 1, the pixel of the position on the original high-quality image is reserved, if the pixel of the position on the first target semantic region mask image is 0, the pixel of the position on the original high-quality image is not reserved, and the positions of the reserved pixels on the original high-quality image form the high-quality first target semantic region.

[0110] Understandably, in the entertainment live broadcast scene, the anchor portrait region mask image F is used to process the original high-quality image I H×W HR ∈R H×W to obtain the high-quality anchor portrait region

[0111] S102-2c, the low-quality first target semantic region and the high-quality first target semantic region are weightedly fused, and the fusion result is combined with the high-quality second target semantic region to obtain a second low-quality image.

[0112] In the entertainment live broadcast scene, the first low-quality image The high-quality background region The low-quality anchor portrait region And the high-quality anchor portrait region Satisfy the following formula:

[0113]

[0114] Wherein, alpha ∈ [0, 1] represents a weighting parameter.

[0115] Understandably, when alpha = 0, the anchor portrait region and the background region on the second low-quality image are both from the original high-quality image, and correspondingly, when subsequent model training is performed, the model can learn the identity mapping;

[0116] ​​When a = 1, that is, the anchor portrait region in the original high-quality image is completely replaced by the anchor portrait region of the original low-quality image.

[0117] When 0 < a < 1, the anchor portrait region on the second low-quality image is obtained by weighted summation of the anchor portrait region of the original low-quality image and the anchor portrait region of the original high-quality image.

[0118] Obviously, when the value of a gradually increases from 0 to 1, the quality of the anchor portrait region on the second low-quality image gradually decreases, and correspondingly, when the model is trained, the difficulty of model learning gradually increases with the increase of the training step number, so that the trained model is convergent after reaching the quality of the original low-quality image.

[0119] S102-3, based on the transition region mask image, fusing the original low-quality image and the original high-quality image to obtain a third low-quality image.

[0120] It can be understood that in the entertainment live broadcast scene, the third low-quality image includes a clear non-transition region and a transition region (a region between the anchor portrait region and the background region) meeting a preset definition, and the preset definition is related to the weighting parameter in the fusion processing.

[0121] In a possible implementation, the implementation process of step S102-3 can be as follows:

[0122] S102-3a, generating a non-transition region mask image based on the first target semantic region mask image, the second target semantic region mask image and the transition region mask image.

[0123] In the entertainment live broadcast scene, the non-transition region mask image can be represented as F+B-S, that is, first combining the anchor portrait region mask image F e R H×W and the background region mask image B e R H×W , and then removing the transition region mask image S e R H×W from the combined result.

[0124] S102-3b, processing the original high-quality image using the non-transition region mask image to obtain a high-quality non-transition region.

[0125] For any position on the original high-quality image, if the pixel of the position on the non-transition region mask image is 1, the pixel of the position on the original high-quality image is retained, and if the pixel of the position on the non-transition region mask image is 0, the pixel of the position on the original high-quality image is not retained, and the positions of the retained pixels on the original high-quality image form the high-quality non-transition region.

[0126] It can be understood that in the entertainment live broadcast scene, the non-transition region mask image F+B-S is used to process the original high-quality image I HR∈R H×W processing, to obtain high-quality non-transition region

[0127] S102-3c, processing the original low-quality image and the original high-quality image respectively by using the transition region mask image, to obtain low-quality transition region and high-quality transition region.

[0128] For any position on the original low-quality image, if the pixel of the position on the transition region mask image is 1, the pixel of the position on the original low-quality image is reserved, if the pixel of the position on the transition region mask image is 0, the pixel of the position on the original low-quality image is not reserved, and the positions of the reserved pixels on the original low-quality image constitute the low-quality transition region.

[0129] It can be understood that, in the entertainment live scene, the transition region mask image S∈R H×W processing the original low-quality image I LR ∈R H×W processing, to obtain low-quality transition region

[0130] For any position on the original high-quality image, if the pixel of the position on the transition region mask image is 1, the pixel of the position on the original high-quality image is reserved, if the pixel of the position on the transition region mask image is 0, the pixel of the position on the original high-quality image is not reserved, and the positions of the reserved pixels on the original high-quality image constitute the high-quality transition region.

[0131] It can be understood that, in the entertainment live scene, the transition region mask image S∈R H×W processing the original high-quality image I HR ∈R H×W processing, to obtain high-quality transition region

[0132] S102-3d, performing weighted fusion on the low-quality transition region and the high-quality transition region, and combining the fusion result with the high-quality non-transition region, to obtain a third low-quality image.

[0133] In the entertainment live scene, the third low-quality image high-quality non-transition region low-quality transition region and high-quality transition region satisfy the following formula:

[0134]

[0135] wherein, α∈[0, 1] represents a weighting parameter.

[0136] Understandably, when α = 0, both the transition region and the non-transition region on the third low-quality image come from the original high-quality image. Accordingly, when training the model later, the model can learn the identity mapping.

[0137] When α = 1, the transition regions in the original high-quality image are completely replaced by the transition regions in the original low-quality image.

[0138] When 0 < α < 1, the transition region on the third low-quality image is obtained by weighted summation of the transition regions of the original low-quality image and the original high-quality image.

[0139] Obviously, when α As the value gradually increases from 0 to 1, the image quality of the transition region on the third low-quality image gradually decreases. Correspondingly, during model training, as the number of training steps increases, the learning difficulty of the model gradually increases, so that the trained model converges after reaching the image quality of the original low-quality image.

[0140] The following continues... Figure 2 Steps S103 and S104 will be described.

[0141] S103, input the data augmentation results into the pre-built image quality enhancement model for training, and obtain the trained image quality enhancement model.

[0142] The data augmentation results include multiple augmented image sets with different weighting parameters, and the weighting parameter is α mentioned above.

[0143] Assuming the preset business scenario is an entertainment live streaming scenario, let I be the original low-quality image in each image pair contained in the original image set. LR ∈R H×W The corresponding original high-quality image is I. HR ∈R H×W Any enhanced image set in the data augmentation results obtained from steps S101 to S102 above is Each enhanced image set The corresponding α is different.

[0144] Let the reconstructed image output by the image enhancement model be I. SR ∈R H×W The loss function of the image quality enhancement model is L.

[0145] The training objective of the image enhancement model is:

[0146] min L(I HR I SR )=min L(I HR ,Φ(Θ(I LR )))

[0147] wherein Φ represents the image quality enhancement model, Θ(I LR ) represents data augmentation on the original low-quality image.

[0148] In a possible implementation, the step S103 comprises sub-steps S103-1-S103-2.

[0149] S103-1, in ascending order of the weighting parameters, input each enhanced image set into the pre-constructed image quality enhancement model for training to obtain the trained image quality enhancement model.

[0150] It can be understood that, according to the value of α from 0 to 1, the plurality of enhanced image sets are input into the pre-constructed image quality enhancement model for training in turn.

[0151] Since the quality of each enhanced image set input in turn is gradually reduced, the learning difficulty of the image quality enhancement model is gradually increased, so that the trained image quality enhancement model is converged when the quality of the original low-quality image is reached, that is, α = 1.

[0152] S103-2, in the case that the weighting parameter corresponding to the input model of the enhanced image set is a preset maximum value, the enhanced image set of the input model is updated using the original low-quality image, and the trained image quality enhancement model is adjusted using the updated enhanced image set to minimize the loss function of the trained image quality enhancement model.

[0153] wherein the preset maximum value can be 1, when the weighting parameter α = 1, the enhanced image set corresponding to the weighting parameter is updated, and the updated image set is

[0154] Since the trained image quality enhancement model is converged, by controlling the proportion of the original low-quality image I LR ∈R H×W , the trained image quality enhancement model is fine-tuned to make the loss function satisfy min L(I HR , I SR ) = min L(I HR , Φ(Θ(I LR ))).

[0155] The image quality model training process in the embodiment of the present application is compared with Figure 1 ​The existing image quality model training process shown in the figure is different in that, for any one original low-quality image, a semantic segmentation model is used based on its corresponding original high-quality image to segment different semantic regions and transition regions, and then different clarity combinations are performed on each semantic region and transition region to achieve data enhancement of the original low-quality image, such as Figure 3 As shown in the figure, in the entertainment live broadcast scene, the clear portrait region is combined with the blurred background region, the blurred portrait region is combined with the clear background region, and the blurred transition region is combined with the clear non-transition region.

[0156] By adjusting the difficulty of data enhancement, the difficulty of model learning is gradually increased to avoid the problem of model not converging due to too large initial difficulty. In the whole process, the model gradually learns to process and optimize different semantic regions, so as to have a targeted optimization effect on the enhanced region.

[0157] S104, using the trained image quality enhancement model to perform quality enhancement on the image in the preset business scenario.

[0158] In the entertainment live broadcast scene, since the semantic prior related to the entertainment live broadcast scene has been introduced in the image quality enhancement model training process, the trained image quality enhancement model can perform specific enhancement on the anchorperson portrait region and background region in the live broadcast scene, and can also maintain the transition details between semantic regions.

[0159] In order to perform the corresponding steps in the above method embodiments and various possible implementations, an implementation of an image quality enhancement device 100 is given below.

[0160] Please refer to Figure 4 The image quality enhancement device 100 includes a semantic segmentation module 101, a data enhancement module 102, a model training module 103, and a quality enhancement module 104.

[0161] The semantic segmentation module 101 is configured to perform semantic segmentation on the original high-quality images in the original image set based on the semantic prior of the preset business scenario to obtain a semantic segmentation result.

[0162] The data enhancement module 102 is configured to perform data enhancement on the original low-quality images in the original image set according to the semantic segmentation result to obtain a data enhancement result.

[0163] The model training module 103 is configured to input the data enhancement result into a pre-constructed image quality enhancement model for training to obtain a trained image quality enhancement model.

[0164] The quality enhancement module 104 is configured to use the trained image quality enhancement model to perform quality enhancement on the image in the preset business scenario.

[0165] Optionally, the semantic segmentation result includes a first target semantic region mask image, a second target semantic region mask image, and a transition region mask image. The data augmentation module 102 is specifically configured to fuse the original low-quality image and the original high-quality image based on the first target semantic region mask image to obtain a first low-quality image, the first low-quality image having a clear semantic region; fuse the original low-quality image and the original high-quality image based on the second target semantic region mask image to obtain a second low-quality image, the second low-quality image having a clear background region; and fuse the original low-quality image and the original high-quality image based on the transition region mask image to obtain a third low-quality image, the third low-quality image having a transition region meeting a preset definition. The data augmentation result includes the first low-quality image, the second low-quality image, and the third low-quality image.

[0166] Optionally, when the data augmentation module 102 is configured to fuse the original low-quality image and the original high-quality image based on the first target semantic region mask image to obtain the first low-quality image, the data augmentation module 102 is specifically configured to process the original high-quality image using the first target semantic region mask image to obtain a high-quality first target semantic region; process the original low-quality image and the original high-quality image using the second target semantic region mask image to obtain a low-quality second target semantic region and a high-quality second target semantic region; perform weighted fusion on the low-quality second target semantic region and the high-quality second target semantic region, and combine the fusion result with the high-quality first target semantic region to obtain the first low-quality image.

[0167] Optionally, when the data augmentation module 102 is configured to fuse the original low-quality image and the original high-quality image based on the second target semantic region mask image to obtain the second low-quality image, the data augmentation module 102 is specifically configured to process the original high-quality image using the second target semantic region mask image to obtain a high-quality second target semantic region; process the original low-quality image and the original high-quality image using the first target semantic region mask image to obtain a low-quality first target semantic region and a high-quality first target semantic region; perform weighted fusion on the low-quality first target semantic region and the high-quality first target semantic region, and combine the fusion result with the high-quality second target semantic region to obtain the second low-quality image.

[0168] Optionally, the data enhancement module 102 is specifically configured to generate a non-transition region mask image based on the first target semantic region mask image, the second target semantic region mask image and the transition region mask image when fusing the original low-quality image and the original high-quality image based on the transition region mask image to obtain a third low-quality image; process the original high-quality image by using the non-transition region mask image to obtain a high-quality non-transition region; process the original low-quality image and the original high-quality image by using the transition region mask image respectively to obtain a low-quality transition region and a high-quality transition region; perform weighted fusion on the low-quality transition region and the high-quality transition region, and combine the fusion result with the high-quality non-transition region to obtain the third low-quality image.

[0169] Optionally, the data enhancement result includes a plurality of enhanced image sets corresponding to different weighting parameters, and the model training module 103 is specifically configured to input each enhanced image set into the pre-constructed image quality enhancement model in turn according to the weighting parameters from small to large to obtain the trained image quality enhancement model.

[0170] Optionally, the model training module 103 is further specifically configured to update the enhanced image set input into the model by using the original low-quality image in a case where the weighting parameter corresponding to the enhanced image set input into the model is a preset maximum value, and adjust the trained image quality enhancement model by using the updated enhanced image set to minimize the loss function of the trained image quality enhancement model.

[0171] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the image quality enhancement device 100 described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0172] Further, the embodiment of the present application also provides an electronic device 200, please refer to Figure 5 The electronic device 200 can include a memory 210 and a processor 220.

[0173] The processor 220 can be a general central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the image quality enhancement method provided in the foregoing method embodiments.

[0174] The memory 210 can be a ROM, or other type of static storage device that can store static information and instructions; a RAM, or other type of dynamic storage device that can store information and instructions; an Electrically Erasable Programmable-Only MEMory (EEPROM), a Compact disc Read-Only MEMory (CD-ROM) or other optical disk storage, an optical storage such as a compact disc, a laser disc, an optical disc, a digital versatile disc, a blu-ray disc, or the like; a magnetic storage medium such as a diskette, or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto. The memory 210 can exist in the sole possession of the processor 220 via a communication bus. The memory 210 can also be integrated with the processor 220. The memory 210 is configured to store machine executable instructions for implementing the embodiments of the present application. The processor 220 is configured to execute the machine executable instructions stored in the memory 210 to implement the above-described method embodiments.

[0175] The embodiments of the present application further provide a computer readable storage medium including a computer program, and the computer program can be used to perform the related operations in the image quality enhancement method provided by the above-described method embodiments when executed.

[0176] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image quality enhancement method, characterized in that, The method includes: Based on the semantic prior of the preset business scenario, semantic segmentation is performed on the original high-quality images in the original image set to obtain semantic segmentation results. The semantic segmentation results include a first target semantic region mask image, a second target semantic region mask image, and a transition region mask image. Based on the first target semantic region mask image, the original low-quality image and the original high-quality image in the original image set are fused to obtain a first low-quality image. The first low-quality image has a clear first target semantic region and a second target semantic region that meets the preset clarity. Based on the mask image of the second target semantic region, the original low-quality image and the original high-quality image are fused to obtain a second low-quality image. The second low-quality image has a clear second target semantic region and a first target semantic region that meets the preset clarity. Based on the transition region mask image, the original low-quality image and the original high-quality image are fused to obtain a third low-quality image, which has a clear non-transition region and a transition region that meets the preset clarity. The first low-quality image, the second low-quality image, and the third low-quality image are used as data augmentation results. The data augmentation results are input into a pre-built image quality enhancement model for training to obtain a trained image quality enhancement model. The trained image quality enhancement model is used to enhance the image quality of images in a preset business scenario.

2. The method as described in claim 1, characterized in that, The step of fusing the original low-quality image and the original high-quality image in the original image set based on the first target semantic region mask image to obtain the first low-quality image includes: The original high-quality image is processed using the first target semantic region mask image to obtain a high-quality first target semantic region; The original low-quality image and the original high-quality image are processed using the second target semantic region mask image to obtain a low-quality second target semantic region and a high-quality second target semantic region; The low-quality second target semantic region and the high-quality second target semantic region are weighted and fused, and the fusion result is combined with the high-quality first target semantic region to obtain the first low-quality image.

3. The method as described in claim 1, characterized in that, The step of fusing the original low-quality image and the original high-quality image based on the second target semantic region mask image to obtain the second low-quality image further includes: The original high-quality image is processed using the mask image of the second target semantic region to obtain a high-quality second target semantic region; The original low-quality image and the original high-quality image are processed using the first target semantic region mask image to obtain a low-quality first target semantic region and a high-quality first target semantic region. The low-quality first target semantic region and the high-quality first target semantic region are weighted and fused, and the fusion result is combined with the high-quality second target semantic region to obtain the second low-quality image.

4. The method as described in claim 1, characterized in that, The step of fusing the original low-quality image and the original high-quality image based on the transition region mask image to obtain a third low-quality image further includes: Based on the first target semantic region mask image, the second target semantic region mask image, and the transition region mask image, a non-transition region mask image is generated; The original high-quality image is processed using the non-transition region mask image to obtain a high-quality non-transition region; The original low-quality image and the original high-quality image are processed using the transition region mask image to obtain low-quality transition regions and high-quality transition regions; The low-quality transition region and the high-quality transition region are weighted and fused, and the fusion result is combined with the high-quality non-transition region to obtain the third low-quality image.

5. The method as described in claim 1, characterized in that, The data augmentation results include multiple augmented image sets corresponding to different weighting parameters. The step of inputting the data augmentation results into a pre-constructed image quality enhancement model for training to obtain the trained image quality enhancement model includes: According to the weighting parameters from smallest to largest, each enhanced image set is sequentially input into the pre-constructed image quality enhancement model for training, resulting in the trained image quality enhancement model.

6. The method as described in claim 5, characterized in that, The method further includes: When the weighted parameters corresponding to the enhanced image set of the input model are at the preset maximum value, the enhanced image set of the input model is updated using the original low-quality image, and the trained image quality enhancement model is adjusted using the updated enhanced image set to minimize the loss function of the trained image quality enhancement model.

7. An image quality enhancement device, characterized in that, The device includes: The semantic segmentation module is used to perform semantic segmentation on the original high-quality images in the original image set based on the semantic prior of the preset business scenario, and obtain the semantic segmentation result. The semantic segmentation result includes a first target semantic region mask image, a second target semantic region mask image, and a transition region mask image. The data augmentation module is used to fuse the original low-quality image and the original high-quality image in the original image set based on the first target semantic region mask image to obtain a first low-quality image, wherein the first low-quality image has a clear first target semantic region and a second target semantic region that meets a preset clarity; to fuse the original low-quality image and the original high-quality image based on the second target semantic region mask image to obtain a second low-quality image, wherein the second low-quality image has a clear second target semantic region and a first target semantic region that meets the preset clarity; and to fuse the original low-quality image and the original high-quality image based on the transition region mask image to obtain a third low-quality image, wherein the third low-quality image has a clear non-transition region and a transition region that meets the preset clarity. The model training module is used to take the first low-quality image, the second low-quality image, and the third low-quality image as data augmentation results, input the data augmentation results into a pre-built image quality enhancement model for training, and obtain the trained image quality enhancement model. The image quality enhancement module is used to enhance the image quality of images in preset business scenarios using a trained image quality enhancement model.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the image quality enhancement method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed, implements the image quality enhancement method as described in any one of claims 1 to 6.

Images