Image quality enhancement method and apparatus, device, medium, product

Abstract

The application relates to an image quality enhancement method and device, equipment, medium and product, the method comprising: identifying the quality level of a to-be-processed image and whether a face image exists; when the quality of the to-be-processed image belongs to a low level or a face image exists, generating a high dynamic range image of the to-be-processed image by using a first conversion enhancement model, otherwise, generating a high dynamic range image of the to-be-processed image by using a second conversion enhancement model, wherein the image effect expressiveness of the second conversion enhancement model is higher than that of the first conversion enhancement model. The application can distinguish different situations such as different qualities and whether a face image exists, use a conversion enhancement model with corresponding image effect expressiveness for different situations to perform corresponding quality enhancement processing, obtain a corresponding high dynamic range image, so that different conversion enhancement models can be trained in a targeted manner, and a high dynamic range image with better quality can be obtained for the to-be-processed image.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image quality enhancement method, apparatus, device, medium, or product thereof. Background Technology

[0002] To meet the demand for high-quality imaging, smartphones have gradually shifted from supporting only Standard Dynamic Range (SDR) images to supporting High Dynamic Range (HDR) images, marking a transition from the SDR era to the HDR era. However, due to the ongoing replacement of related software and hardware, HDR images still have a relatively low usage rate. Secondly, there are significant differences in camera functions and processing power among different smartphones, with low-end phones producing less than ideal HDR image quality. Furthermore, the vast amount of SDR images available online remains the primary source of information. Therefore, SDR to HDR image conversion technology is an important image processing technique.

[0003] The typical method for acquiring HDR images involves fusing SDR images with different exposures to create a single HDR image. This approach aims to capture more image information under varying exposure conditions, ensuring accurate image data is present in both bright and dark areas, resulting in a clearer, more colorful, and more balanced high-quality image. However, due to the complexity of the production process and the large amount of data required, it is generally only used in professional photography and has not been widely adopted by the general consumer market. Furthermore, since HDR images are obtained by fusing SDR images with different exposures, it is practically impossible to obtain multi-frame data of the original scene, making the conversion of a single SDR image to an HDR image virtually impossible. In recent years, the explosive development of AI technology has made the conversion of a single SDR image to an HDR image possible. Because of the high value of HDR image conversion technology, both academia and industry have been dedicated to the research and practical application of SDR to HDR conversion.

[0004] With the rapid development of AI, various SDR to HDR image conversion technologies based on CNN (Convolutional Neural Network) have emerged and achieved good results. However, these algorithms do not demonstrate sufficient robustness and expressiveness in practical applications. For example, HDRUNet uses dequantization to enhance the bit depth of the image and recovers overexposed details based on the difference in noise distribution between HDR and SDR images. Its advantages are: it can better recover overexposed details by utilizing differences in image noise, thus improving the expressiveness of HDR images; adding an image denoising module can improve image quality, especially for dark areas. Its disadvantages are: the network's generalization ability is relatively weak, and the noise distribution has a significant impact on the image effect; it overemphasizes the ability to recover overexposed areas, making images outside the training data distribution prone to artificial artifacts and color tone variations. For general internet data or open-scene shooting, the input image quality may be inherently insufficient. Therefore, many current algorithms repair, restore, or improve the resolution of the input image before SDR to HDR conversion. The advantage of this operation is that the image quality of the converted HDR image will be more obvious after the input image quality is improved. However, if there are some artificial traces or abnormal effects in the repair or image quality improvement results, the HDR conversion will also have some shortcomings. In addition, SDR to HDR is also an image quality improvement algorithm. If too many image quality improvement algorithms are added and combined together, it will inevitably lead to overdoing it.

[0005] As mentioned above, the main problems with SDR to HDR image conversion are as follows: the robustness and generalization ability of SDR to HDR models are low in different scenarios, especially when faces are present in the original image; existing defects in the original image may become more obvious, such as image tortuosity, noise, and edge ringing; from the perspective of image quality, single enhancement of SDR to HDR images cannot bring about a strong image quality, and the current fusion of multiple image quality enhancement methods has poor stability. Summary of the Invention

[0006] The purpose of this application is to solve the above-mentioned problems by providing an image quality enhancement method and corresponding apparatus, devices, non-volatile readable storage media, and computer program products.

[0007] According to one aspect of this application, an image quality enhancement method is provided, comprising the following steps:

[0008] Identify the image quality level of the image to be processed and whether it contains human faces;

[0009] When the image quality of the image to be processed is low or contains a face image, a first conversion enhancement model is used to generate a high dynamic range image of the image to be processed; otherwise, a second conversion enhancement model is used to generate a high dynamic range image of the image to be processed. The image performance of the second conversion enhancement model is higher than that of the first conversion enhancement model.

[0010] Optionally, the image quality level of the image to be processed and the presence of a face image are identified, including:

[0011] The image to be processed is compressed and repaired, and its corresponding compression ratio is predicted.

[0012] Determine whether the compression ratio is higher than a preset threshold. If it is higher than the preset threshold, determine that the image to be processed is a low-level image; otherwise, determine that it is a high-level image.

[0013] Face detection is performed on the repaired image to determine whether a face image exists in it.

[0014] Optionally, image compression and restoration are performed on the image to be processed, and its corresponding compression ratio is predicted, including:

[0015] The backbone network in the image compression and restoration model is used to extract features from the image to be processed to obtain image semantic features, and image restoration is performed on the image to be processed based on the image semantic features.

[0016] The control prediction network in the image compression and restoration model is used to predict the corresponding compression ratio based on the image semantic features, and the compression ratio is used to control the image restoration process of the backbone network.

[0017] Optionally, after generating the high dynamic range image of the image to be processed, the process includes:

[0018] The high dynamic range image is subjected to super-resolution enhancement processing to obtain the enhanced image of the image to be processed.

[0019] Optionally, before identifying the image quality level of the image to be processed and whether a face image exists, the following steps are included:

[0020] Obtain two instances of the conversion enhancement model, which will be used as the first conversion enhancement model and the second conversion enhancement model, respectively.

[0021] The first conversion enhancement model and the second conversion enhancement model are trained to convergence using different training sets. The image performance corresponding to the difference between the sample images and the supervised images in the first training set used by the first conversion enhancement model is lower than that corresponding to the difference between the sample images and the supervised images in the second training set used by the second conversion enhancement model.

[0022] Optionally, the image performance is reflected in the difference in the color gamut range of pixel values; the larger the color gamut range, the better the image performance.

[0023] Optionally, before identifying the image quality level of the image to be processed and whether a face image exists, the following steps are included:

[0024] The camera unit of the terminal device is activated to acquire an environmental preview video;

[0025] Target recognition is performed on the image frames in the environmental preview video to determine the presence of product images within them;

[0026] The image frame containing the product image is used as the image to be processed.

[0027] According to another aspect of this application, an image quality enhancement device is provided, comprising:

[0028] The image recognition module is configured to identify the image quality level of the image to be processed and whether a human face image exists;

[0029] The image enhancement module is configured to generate a high dynamic range image of the image to be processed using a first conversion enhancement model when the image quality of the image to be processed is of a low level or when a face image is present; otherwise, it uses a second conversion enhancement model to generate a high dynamic range image of the image to be processed. The image effect performance of the second conversion enhancement model is higher than that of the first conversion enhancement model.

[0030] According to another aspect of this application, an image quality enhancement device is provided, including a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the image quality enhancement method described in this application.

[0031] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the image quality enhancement method in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, executes the steps included in the method.

[0032] According to another aspect of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0033] Compared to existing technologies, this application identifies the image quality level of the image to be processed and whether the image contains a face. It then distinguishes between different situations and applies two conversion enhancement models with different image performance capabilities to convert the image into a high dynamic range image. Specifically, when the image quality level is low or a face image is present, a first conversion enhancement model with weaker image performance capability is used to convert the image. In other cases, a second conversion enhancement model with stronger image performance capability is used. This is equivalent to treating low-quality images and images containing faces as exceptions handled by the first conversion enhancement model, while other cases are handled by the second conversion enhancement model. Both the first and second conversion enhancement models can be trained separately, exhibiting stronger robustness when converting images of corresponding characteristics, and ultimately achieving optimal image quality.

[0034] Specifically, when processing facial images and low-quality images, the relatively weaker image performance of the first conversion enhancement model ensures that the color difference before and after conversion is controllable and that various imperfections are less likely to occur. For human skin tones, it can achieve a more natural appearance. On the other hand, the second conversion enhancement model, due to its stronger image performance, can ensure greater color gamut expansion and color enhancement for images that do not contain facial images and other high-quality images. It can achieve a wider brightness range, provide more image details, and display more vibrant colors, making the converted high dynamic range image more vivid and bright. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A schematic diagram of the network architecture corresponding to the image enhancement service used in this application;

[0037] Figure 2 This is a schematic diagram of an exemplary image quality enhancement architecture of this application;

[0038] Figure 3 This is a flowchart illustrating one embodiment of the image quality enhancement method of this application.

[0039] Figure 4 The images provided are examples, with the original image on the left and the resulting image on the right after transformation processing by the first transformation enhancement model.

[0040] Figure 5 The images provided are examples, with the original image on the left and the resulting image on the right after transformation processing by the second transformation enhancement model.

[0041] Figure 6 This is a flowchart illustrating the specific process of identifying the image to be processed in the embodiments of this application;

[0042] Figure 7 This is a schematic flowchart of another embodiment of the image quality enhancement method exemplified in this application;

[0043] Figure 8 The images are provided as examples, with the left side showing the result obtained by general interpolation on the original image, and the right side showing the result obtained by super-resolution enhancement processing according to this application.

[0044] Figure 9 This is a schematic diagram illustrating the process of training the first and second conversion enhancement models in an embodiment of this application;

[0045] Figure 10 This is a schematic diagram of the process for obtaining the image to be processed in an embodiment of this application;

[0046] Figure 11 This is a schematic block diagram of the image quality enhancement device of this application;

[0047] Figure 12 This is a schematic diagram of the structure of an image quality enhancement device used in this application. Detailed Implementation

[0048] Please see Figure 1 The network architecture adopted in an exemplary application scenario of this application includes terminal devices 83 and 84, a media server 82, and a business server 81. The business server 81 can be used to deploy online stores, forming independent websites within an e-commerce platform. The media server 82 can be used to deploy image conversion services. When users on the terminal devices 83 and 84 access the pages of the online stores, they can use the image conversion service of the media server 82 through the business server 81 to convert the images submitted by the terminal devices 83 and 84 from standard dynamic range images to high dynamic range images. The media server 82 can run a computer program product programmed according to the image quality enhancement method of this application to provide the image conversion service, so as to convert the images to be processed into corresponding high dynamic range images for output.

[0049] It should be pointed out that, Figure 1The application scenario described herein is merely an example of platform-based deployment. In other exemplary application scenarios, the computer program product described in this application can also run on any computer device with sufficient computing power. For example, it can run on terminal devices 83 and 84 to enhance images from user-specified local images or image frames in environmental preview videos obtained after the local camera unit is activated, so that the corresponding images or videos meet the standards for high dynamic range imaging. The terminal devices 83 and 84 can be various mobile terminals, including but not limited to smartphones, tablets, laptops, and other forms of personal computers.

[0050] The images to be processed in this application are not limited in content and can be any content. For example, in e-commerce scenarios, they can be product-related images submitted by merchant users or consumer users, and in terminal devices, they can be real-life images taken in any life scenario.

[0051] Please see Figure 2 In accordance with the inventive spirit of this application, an exemplary image quality enhancement architecture is also provided, which can be used to enhance the image quality of the image to be processed. The image quality enhancement architecture includes an image compression and restoration model, a face detection model, a first conversion enhancement model, a second conversion enhancement model, and an image super-resolution enhancement model. The image compression and restoration model is used to perform image compression and restoration on the image to be processed and predict its corresponding compression ratio; the face detection model is used to predict whether the image to be processed contains a face image. The first conversion enhancement model and the second conversion enhancement model are models with the same model architecture, but they are trained using different training sets. The first conversion enhancement model has weaker image effect performance and is used to process low-quality images to be processed that are detected and restored by the image compression and restoration model, or to process restored images to be processed that contain face images even though they are low-quality. The second conversion enhancement model has stronger image effect performance and is used to process all restored images to be processed that are not suitable for processing by the first conversion enhancement model. The image super-resolution enhancement model can be used to perform further super-resolution enhancement processing on the high dynamic range images obtained after imaging by the first conversion enhancement model or the second conversion enhancement model, so that the final output image achieves the optimal effect.

[0052] Figure 2 The image quality enhancement architecture given is merely exemplary. In other embodiments, the architecture can be appropriately modified. For example, the image super-resolution enhancement model can be omitted, and the high dynamic range image output by the first or second conversion enhancement model can be directly used; or, any other feasible detection method can be used for face recognition and image quality level determination, etc., and so on, which can be flexibly implemented.

[0053] Figure 2 The various models used in the image quality enhancement architecture shown are recommended to be neural network models based on deep learning with corresponding functions. Models cited or potentially cited in this application, including traditional machine learning models or deep learning models, can be deployed on a remote server and remotely invoked from the client, or deployed on a client with sufficient device capabilities for direct invocation, unless explicitly specified. In some embodiments, when running on the client, the corresponding intelligence can be acquired through transfer learning to reduce the requirements on client hardware resources and avoid excessive consumption of client hardware resources.

[0054] Please see Figure 3 According to an image quality enhancement method provided in this application, in one embodiment, the method includes the following steps:

[0055] Step S1100: Identify the image quality level of the image to be processed and whether a face image exists;

[0056] When image quality enhancement processing is required for an original image, it can be provided as the image to be processed in this application, and then the business logic of this application can be used to perform the corresponding processing.

[0057] In order to determine which method should be used to enhance the image quality of the image to be processed, image quality level recognition and face image detection can be performed on the image to be processed, so as to decide which method to use to enhance the image quality of the image to be processed based on the image quality level and whether a face image is present.

[0058] The image quality level can be set to two or more. For ease of understanding, it can be understood as two, that is, including low level and high level, or low level and non-low level. In other words, multiple high levels are allowed so that they can correspond to different image quality enhancement processing methods.

[0059] There are many possible methods for identifying image quality levels. In one embodiment, the sharpness of the image to be processed can be determined by applying various sharpness evaluation algorithms, and it can be divided into different levels according to a sharpness threshold. Optional sharpness evaluation algorithms include, but are not limited to, the Brenner gradient function algorithm, the Tenengrad gradient function algorithm, the variance function algorithm, the quadratic blur algorithm, etc., which can be flexibly used by those skilled in the art. In another embodiment, a deep learning-based neural network model can be used. For example, an image feature extractor followed by a classifier can be used. The image feature extractor extracts the deep semantic information of the image to be processed, and then the classifier determines whether the image quality belongs to a low or high level. Of course, the neural network model should be pre-trained to a convergent state using corresponding training samples. The training samples include images of high or low image quality. Then, the corresponding loss value is calculated using the corresponding high or low level label, and the gradient of the neural network model is updated according to the loss value. After the neural network model is trained to a convergent state, it can be put into use.

[0060] When the image quality of the image to be processed is identified as being of a low level, it can be determined that the first conversion enhancement model of this application will be used to perform an imaging conversion from a standard dynamic range image to a high dynamic range image, so as to obtain a high dynamic range image of the image to be processed through the first conversion enhancement model.

[0061] When the image quality of the image to be processed is not low, further face recognition is performed to determine whether a face image exists in the image. Face recognition can be performed using various mature neural network models. For example, in the course of technological development, models such as LeNet, AlexNet, VGG, ResNet, and SqueezeNet have been developed, all of which can be used for face recognition. These models extract image features from the face region in the image to determine whether it belongs to a face and output the corresponding judgment result.

[0062] When face recognition determines that the image to be processed contains a face image, the first conversion enhancement model of this application can be used to perform an imaging conversion from a standard dynamic range image to a high dynamic range image, so as to obtain a high dynamic range image of the image to be processed through the first conversion enhancement model.

[0063] If the image to be processed is neither of low quality nor contains a face image, then the second conversion enhancement model of this application can be used to perform an imaging conversion from a standard dynamic range image to a high dynamic range image, so as to obtain a high dynamic range image of the image to be processed through the second conversion enhancement model.

[0064] As can be seen, in this step, by identifying the image quality of the image to be processed and whether there are human faces in the image content, the process of image quality enhancement processing of the image to be processed is controlled to ensure that the correct enhancement and conversion processing is performed on the image according to different situations.

[0065] Step S1200: When the image quality of the image to be processed is low or contains a face image, a high dynamic range image of the image to be processed is generated using a first conversion enhancement model; otherwise, a high dynamic range image of the image to be processed is generated using a second conversion enhancement model, wherein the image effect performance of the second conversion enhancement model is higher than that of the first conversion enhancement model.

[0066] For images of low quality or containing human faces, a first conversion enhancement model is used to convert the image from standard dynamic range to high dynamic range to obtain a corresponding high dynamic range image.

[0067] For other images that are not suitable for processing by the first conversion enhancement model, the second conversion enhancement model is used to convert the images from the standard dynamic range to the high dynamic range to obtain the corresponding high dynamic range images.

[0068] The first conversion enhancement model and the second conversion enhancement model are deep learning-based neural network models with the same network architecture, but they are trained using different training sets, so that the first conversion enhancement model and the second conversion enhancement model have different image performance. The second conversion enhancement model, as a strong effect model, has better image performance than the first conversion enhancement model, which is a weak effect model.

[0069] The image performance is mainly reflected in the difference in pixel value color gamut range. It is easy to understand that the larger the color gamut range, the better the image performance, and vice versa. When a high dynamic range (HDR) image has a low color gamut, the conversion from a standard dynamic range (SLR) image to an HDR image will not produce significant distortion for color gamut-sensitive content such as human skin tones, as the change in color gamut is not substantial. The same applies to other image content. Its main drawback is that the limited color gamut makes it difficult to achieve a significant improvement in color quality; therefore, it only provides a limited enhancement to the image being processed. Conversely, when a HDR image has a high color gamut, the conversion from SLR to HDR can lead to color casts due to the expansion and enhancement of the original color gamut during the conversion process. This is more noticeable in the visual effect of human skin tones. However, on the other hand, the expansion and enhancement of the color gamut can also result in a more vivid and detailed high-quality image, providing a wider brightness range, richer image details, and more vibrant colors, making it more pleasing to the user's visual perception. It is evident that, given the different image performance capabilities corresponding to different color gamut ranges, the application of color gamuts to images of different qualities and content has its own advantages and disadvantages. Differentiating the images to be processed in different situations and applying different corresponding processing methods can ensure that the best image quality enhancement effect can be obtained in each situation, which is more practical.

[0070] The first and second conversion enhancement models are trained for different scenarios, gaining advantages in training because each model can be trained under supervision using training samples corresponding to that scenario. After learning representations from similar training samples, they achieve stronger robustness, accurately representing features of similar images and performing effective image conversion processing, resulting in excellent image conversion effects. Both models can leverage their respective strengths within their specialized image quality levels to enhance the image quality of the processed image, ultimately achieving superior image enhancement and conversion capabilities and ensuring the provision of a comprehensive, optimal image quality enhancement service.

[0071] As can be seen from the above embodiments, this application identifies the image quality level of the image to be processed and whether the image contains a face, distinguishes different situations, and applies two conversion enhancement models with different image effect performance to convert the image to be processed into a high dynamic range image. Specifically, when the image quality level is low or when a face image is present, the first conversion enhancement model with weaker image effect performance is used to convert the image to be processed. In other cases, the second conversion enhancement model with stronger image effect performance is used for conversion. This is equivalent to treating the low image quality level and the case containing a face image as exceptions and having the first conversion enhancement model handle them specially, while leaving other cases to the second conversion enhancement model for specialized handling. Both the first and second conversion enhancement models can be trained separately for specific purposes, and each has stronger robustness when converting images of corresponding properties, and can obtain the best image quality effect overall.

[0072] Specifically, when processing facial images and low-quality images, the relatively weaker image enhancement capabilities of the first conversion and enhancement model ensure controllable color differences before and after conversion, reducing the likelihood of various imperfections and achieving a more natural representation of skin tone. The second conversion and enhancement model, with its stronger image enhancement capabilities, ensures greater color gamut expansion and color enhancement for images that do not contain facial images or other high-quality images, achieving a wider brightness range, providing more image details, and displaying more vibrant colors, resulting in a more vivid and lifelike high dynamic range image. Figure 4 As shown, Figure 4 The image on the left is the original image to be processed, and the image on the right is the high dynamic range image after the image quality has been improved by the first conversion enhancement model. It can be seen that when the first conversion enhancement model processes the image to be processed containing facial images, the skin color of the face is relatively natural. Figure 5 The left and right images are the images before and after processing by the second transformation enhancement model. Figure 5 As can be seen, in the absence of facial images, the second transformation enhancement model can achieve very high-quality image enhancement effects. Regardless of brightness, color gamut, or detail, the image on the right is significantly improved compared to the image on the left.

[0073] Based on any embodiment of this application, please refer to Figure 6 Identify the image quality level of the image to be processed and whether it contains a face image, including:

[0074] Step S1110: Perform image compression and restoration on the image to be processed, and predict its corresponding compression ratio;

[0075] In this embodiment, you can refer to Figure 2The architecture shown employs an image compression and restoration model to identify the image quality level of the image to be processed. It first performs image compression and restoration on the image to be processed and predicts its corresponding compression rate.

[0076] Image compression leading to quality degradation is a common problem. This degradation typically manifests as color blocks, color noise, and edge ringing. Simply relying on robust conversion enhancement models or other SDR to HDR models cannot repair or avoid these issues. Therefore, such images will inevitably experience negative optimization and deterioration after SDR to HDR conversion. Consequently, this embodiment recommends using a model specifically designed for image compression and restoration to perform image compression and restoration on the image to be processed.

[0077] An exemplary image compression and restoration model can be the FBCNN model, which is suitable for restoring JPEG format images. The network consists of two parts: a backbone network and a control prediction network. The backbone network is similar to an encoder-decoder Unet structure, with an encoding path responsible for feature extraction and a decoding path responsible for reconstructing the image and improving image quality. Unlike typical end-to-end structures, the control prediction network uses features calculated by the backbone network's encoding path to predict the compression ratio of the current image in a multilayer perceptron predictor. For example, since JPEG compression ranges from [0, 100], a fixed value within this range can be predicted as the compression ratio. This compression ratio is then input to a controller in another multilayer perceptron structure, which outputs weight control parameters γ and bias control parameters β. These parameters control the proportion and bias of the weights in the backbone network's decoding path, allowing the backbone network to perform different degrees of image restoration based on the current compression situation, significantly improving the robustness and expressiveness of the FBCNN model.

[0078] Based on the principles of the FBCNN model, in one embodiment, image compression and restoration of the image to be processed can be achieved through the following specific process, while simultaneously obtaining the corresponding compression ratio:

[0079] Step S1111: Use the backbone network in the image compression and restoration model to extract features from the image to be processed to obtain image semantic features, and perform image restoration on the image to be processed based on the image semantic features;

[0080] Specifically, when the image to be processed is input into the encoding path of the backbone network, the encoders at each scale gradually decrease to extract the image features corresponding to each scale. The image features extracted at the current scale are submitted to the encoders at smaller scales for further image feature extraction. At the same time, the corresponding scale decoders in the decoding path of the backbone network are provided through cross-layer connections as a reference for restoring the image features at the corresponding scale.

[0081] After the last encoder obtains its corresponding image features, the decoders in the decoding path, with progressively increasing scales, perform scale-by-scale decoding. Each decoder, during decoding, uses the image features restored by its relatively small-scale decoder as a basis, and refers to the image features provided by the encoder at the corresponding scale to restore the image features, thereby obtaining the image features restored at the corresponding scale, completing the image restoration at the corresponding scale, and then providing it to the higher-scale decoder for further restoration, and so on, until the image to be processed is finally restored and the restored image to be processed is output.

[0082] The image features output by the decoder at the smallest scale in the encoding path are provided to the predictor in the control prediction network to calculate the corresponding compression ratio, while the weights and biases in the decoder are controlled by the controller in the control prediction network based on the corresponding parameters calculated by the controller based on the compression ratio.

[0083] Step S1112: Using the control prediction network in the image compression and restoration model, the corresponding compression ratio is predicted based on the image semantic features, and the compression ratio is used to control the image restoration process of the backbone network.

[0084] After the control prediction network obtains the image features, it performs a full connection on the network to predict the corresponding compression ratio. The controller then generates weight control parameters and bias control parameters based on the compression ratio and transmits these parameters to the decoders at each scale in the decoding path, enabling each decoder to repair the image at the corresponding scale in relation to the compression ratio.

[0085] As can be seen from the above process, by using the FBCNN model and predicting its compression rate to repair the image to be processed, the image quality of the image to be processed can be initially improved. At the same time, the predicted compression rate can also serve as a decision-making basis for determining which transformation enhancement model to use, which is more efficient.

[0086] Step S1120: Determine whether the compression ratio is higher than a preset threshold. If it is higher than the preset threshold, determine that the image to be processed is a low-level image; otherwise, it is a high-level image.

[0087] To determine which conversion enhancement model to use for image quality enhancement of the image to be processed, a threshold can be preset corresponding to the compression rate. This threshold can be an empirical threshold and can be flexibly set. For example, corresponding to the JPEG compression range [0, 100], the threshold can be set to a fixed value such as 50 or 60. Then, the compression rate of the image to be processed is compared with this threshold. If it is higher than the threshold, it indicates that the original image compression rate is high and the image quality loss is large. It is suitable to use the first conversion enhancement model with weaker image quality performance for processing, and the image to be processed can be identified as a low-level image. Otherwise, it indicates that the compression rate of the image to be processed meets expectations, and it is identified as a high-level image and provided to the second conversion enhancement model with stronger image quality performance for processing.

[0088] It is easy to understand that in some embodiments, multiple interval thresholds can be further subdivided, and multiple conversion enhancement models with different image effect performance can be set accordingly to process the images to be processed at different levels corresponding to different interval thresholds.

[0089] In accordance with the fact that this embodiment performs image restoration on the image to be processed, the restored image to be processed can be used as input data for the first conversion enhancement model or the second conversion enhancement model, so that the corresponding conversion enhancement model can work on the basis of the restored image and obtain a better image quality enhancement effect.

[0090] Step S1130: Perform face detection on the repaired image to determine whether a face image exists in it.

[0091] In addition to processing images of different quality levels based on their original image quality, considering the sensitivity of the human eye to color gamut changes in skin tone during the transition from standard dynamic range to high dynamic range, further processing is performed on the repaired images, such as... Figure 2 As shown, a face detection model can also be used to perform face detection on the repaired image to determine whether a face image exists in it.

[0092] In this embodiment, RetinaFace is recommended as the face detection model. RetinaFace uses feature pyramids to address the issue of varying face sizes. It fuses image feature information obtained from the first and second pyramids, resulting in better robustness and recall of the face detection model, thus accurately determining whether a given image contains a face. In this embodiment, RetinaFace is considered to have a face in the image if it detects a face bounding box larger than 1 / 8 of the image width.

[0093] Due to the unique characteristics of human faces and skin tones in SDR-to-HDR images, images with clearly defined faces can only be enhanced using the first conversion enhancement model, which has a weaker enhancement effect.

[0094] As can be seen from the above embodiments, during the image compression and restoration process of the image to be processed, its compression ratio is predicted simultaneously. Then, the image quality level is determined based on the compression ratio. First, a suitable conversion enhancement model is determined for the image to be processed after conversion and restoration based on the image quality level. For images with low image quality, the first conversion enhancement model, which has weaker image performance, is used. For images with high image quality, a face detection model is further used to identify images containing clear faces. Images containing faces in the restored image to be processed are also assigned to the first conversion enhancement model, which also has weaker image performance. The remaining cases are then processed by the second conversion enhancement model. Thus, based on the characteristics of the image performance of different conversion enhancement models, and according to the image quality and content of the image to be processed, effective differentiation and correspondence are achieved. This ensures that, under various conditions, the corresponding conversion enhancement model can be used to convert the restored image to be processed, improving it from the standard dynamic range to the high dynamic range. Each conversion enhancement model can achieve the maximum conversion optimization effect under its corresponding condition.

[0095] Based on any embodiment of this application, please refer to Figure 7 After generating the high dynamic range image of the image to be processed, the process includes:

[0096] Step S1300: Perform super-resolution enhancement processing on the high dynamic range image to obtain the enhanced image of the image to be processed.

[0097] like Figure 2 As shown, for any high dynamic range image obtained after conversion by a conversion enhancement model, an image super-resolution enhancement model can be used to further enhance the high dynamic range image.

[0098] In this embodiment, Real-ESRGAN is recommended as the image super-resolution enhancement model. Its main structure includes a downsampling module, an upsampling module, and a super-resolution backbone. The network backbone is mainly obtained by stacking RRDB structures. RRDB is a structure that combines ResNet and DenseNet, and it is widely used in the field of low-resolution images, with good performance and effectiveness. The main feature of Real-ESRGAN is that when creating image de-degradation data pairs, it uses blurring, image compression, resolution change, and noise increase. More importantly, it adopts two de-degradation methods. Combined with the characteristics of the generative model in the GAN (Generative Adversarial Network) architecture, the Real-ESRGAN model has very good robust performance and can significantly improve image clarity.

[0099] In this application, the image super-resolution enhancement model uses a 2x super-resolution method, but the output can be downsampled by 1 / 2 according to actual needs to ensure that the resolution is consistent with the input image, while obtaining the image quality improvement effect of super-resolution details and noise removal.

[0100] As can be seen from the above embodiments, in the final stage, the image super-resolution enhancement model is used to perform a final enhancement on the high dynamic range image obtained from the image transformation. This avoids the drawbacks of super-resolution operations causing changes in the structure, noise distribution, and color of image details, and prevents the situation where a stable high dynamic range image is not obtained when the image obtained from the super-resolution operation is first sent to the transformation enhancement model. In addition, the image super-resolution enhancement model itself also has a certain image compression and restoration capability, which can effectively supplement situations that the image compression and restoration model cannot handle. Figure 8 As shown, the left side is the result image obtained by using a general interpolation algorithm on a given image, while the right side is the enhanced image obtained by converting the image to be processed and performing super-resolution enhancement operation according to this application. The advantages of this embodiment are particularly obvious when comparing the two.

[0101] Based on any embodiment of this application, please refer to Figure 9 Before identifying the image quality level of the image to be processed and whether a face image exists, the process includes:

[0102] Step S2100: Obtain two instances of the conversion enhancement model, which will be used as the first conversion enhancement model and the second conversion enhancement model, respectively.

[0103] To facilitate the provision of a corresponding first conversion enhancement model and a second conversion enhancement model for this application, the selection of the corresponding conversion enhancement model can be determined first, and then two instances can be obtained and trained differently so that each instance can obtain the corresponding image effect expressiveness and be suitable for the first conversion enhancement model and the second conversion enhancement model.

[0104] An exemplary conversion enhancement model uses the FHDR model. FHDR employs a framework combining classic DenseNet stacked structures with convolutional feature layers. The FBB (Feedback Block) used within it employs a structure similar to a recurrent regression network, where the internal features from the previous forward propagation can serve as input for the next propagation. This allows the same SDR image to undergo multiple forward propagations, significantly improving the robustness, generalization ability, and expressiveness of the entire conversion enhancement model. For open-world applications, the highly stable FHDR model ensures effective application operation.

[0105] Step S2200: Train the first conversion enhancement model and the second conversion enhancement model to convergence using different training sets respectively. The image performance corresponding to the difference between the sample images and the supervised images in the first training set used by the first conversion enhancement model is lower than the image performance corresponding to the difference between the sample images and the supervised images in the second training set used by the second conversion enhancement model.

[0106] Since the first and second conversion enhancement models have identical network architectures, their training principles are the same. The only difference lies in the training sets they use. Specifically, the image quality corresponding to the difference between the sample images used as input to the first training set and the supervision images used as supervision labels in the first training set is lower than that corresponding to the difference between the sample images and supervision images in the second training set used by the second conversion enhancement model. To illustrate, when selecting training samples from the first training set, the differences in brightness, color, and detail between the supervision images and their corresponding sample images are minimal, and a subset of training samples containing face images can be used to generalize face image features. However, when selecting training samples from the second training set, the differences in brightness, color, and detail between the supervision images and their corresponding sample images are greater. By iteratively training the first conversion enhancement model to convergence using training samples from the first training set, and by iteratively training the second conversion enhancement model to convergence using training samples from the second training set, the two conversion enhancement models can exhibit different image quality enhancement capabilities after performing image quality enhancement on the same image to be processed. Thus, the two instances can be put into use in the online inference stage.

[0107] As mentioned earlier, different image effects correspond to different color gamut ranges. Since the two instances need to represent different color gamut ranges, their data processing will differ. Specifically, the vector dimensions of the input data are the same for both instances, but in the final output, the second conversion enhancement model can convert image data with a larger color gamut range based on a larger vector dimension, and the first conversion enhancement model can similarly convert image data with a relatively smaller color gamut range based on a relatively smaller vector dimension. It is easy to understand that the resulting high dynamic range images will also differ in storage size. The storage size of the high dynamic range image of the second conversion enhancement model will be larger than that of the high dynamic range image of the first conversion enhancement model.

[0108] As can be seen from the above embodiments, by using two isomorphic transformation enhancement models and training them with corresponding and different training sets, the high dynamic range images obtained by the two transformation enhancement models can achieve different image performance. Each transformation enhancement model is specifically trained and has stronger output robustness when processing corresponding types of images. It is also easier to converge during training. When applied to image enhancement processing of the image to be processed, it can also achieve the best image quality improvement effect.

[0109] Based on any embodiment of this application, please refer to Figure 10 Before identifying the image quality level of the image to be processed and whether a face image exists, the process includes:

[0110] Step S3100: Activate the camera unit of the terminal device to acquire an environmental preview video;

[0111] The computer program product implemented according to the image quality enhancement method of this application can be installed in a terminal device such as a smartphone for operation, or embedded as a built-in functional module of the camera unit therein, thereby serving the image generation of the camera unit.

[0112] Once the user activates the camera unit in the terminal device, the camera unit begins to work, acquiring an environmental preview video through its camera. The environmental preview video is generated by capturing ambient light at a certain frame rate, cached in the image space of local memory, and can be rendered onto the display screen.

[0113] Step S3200: Perform target recognition on the image frames in the environmental preview video to determine that there are product images in them;

[0114] Users can pre-activate or automatically activate the HDR function to enhance the image quality of each image frame in the environmental preview video. The HDR function is implemented by the operation of the computer program product.

[0115] In this embodiment, in order to specifically serve the needs of merchants on e-commerce platforms in creating product images, before applying the HDR function, each image frame in the environmental preview video is specifically identified to determine whether there is a product image in it, so as to capture image frames containing product images for image quality enhancement processing.

[0116] To determine whether an image frame contains a product image, a pre-trained product classification model can be used. This model consists of an image feature extraction model based on CNN or ResNet, followed by a binary classifier, and trained until convergence using appropriate training samples. For each image frame, its encoding is input into the product classification model, where the image feature extraction model extracts its image features, and then the binary classifier determines whether it contains a product image.

[0117] Step S3300: The image frame containing the product image is taken as the image to be processed.

[0118] For image frames containing product images in the environmental prediction video, these are used as the images to be processed. The corresponding steps of the image quality enhancement method described in the previous embodiments of this application are applied to perform corresponding processing, and finally, a corresponding high dynamic range image or super-resolution enhanced image is obtained, which replaces the original image frame and is output to the display, so that the product image displayed on the display is clearer and has better image quality.

[0119] As can be seen from the above embodiments, by performing product image recognition on the environmental preview video obtained by the camera unit, a more refined solution can be provided for relevant application scenarios of e-commerce platforms. While minimizing the consumption of the limited computing resources of the terminal device, image quality enhancement processing can be performed on the image frames in the environmental preview video containing product images in a timely manner, so that the final obtained product images are of higher quality, thereby meeting the needs of e-commerce platforms to publish product images and improving user experience.

[0120] Please see Figure 11According to one aspect of this application, an image quality enhancement device includes an image recognition module 1100 and an image enhancement module 1200. The image recognition module 1100 is configured to identify the image quality level of an image to be processed and whether a face image exists. The image enhancement module 1200 is configured to generate a high dynamic range image of the image to be processed using a first conversion enhancement model when the image quality of the image to be processed is low or a face image exists; otherwise, it generates a high dynamic range image of the image to be processed using a second conversion enhancement model, wherein the image effect performance of the second conversion enhancement model is higher than that of the first conversion enhancement model.

[0121] Based on any embodiment of this application, the image recognition module 1100 includes: a compression and repair unit, configured to perform image compression and repair on the image to be processed and predict its corresponding compression ratio; a level recognition unit, configured to determine whether the compression ratio is higher than a preset threshold, and if it is higher than the preset threshold, determine that the image to be processed is a low-level image, otherwise it is a high-level image; and a face recognition unit, configured to perform face detection on the repaired image to be processed to determine whether a face image exists therein.

[0122] Based on any embodiment of this application, the compression and restoration unit includes: a restoration processing subunit, configured to use the backbone network in the image compression and restoration model to extract features from the image to be processed to obtain image semantic features, and to perform image restoration on the image to be processed according to the image semantic features; and a compression ratio prediction subunit, configured to use the control prediction network in the image compression and restoration model to predict the corresponding compression ratio according to the image semantic features, and to use the compression ratio to control the image restoration process of the backbone network.

[0123] Based on any embodiment of this application, the image quality enhancement device of this application further includes: a super-resolution enhancement module, configured to perform super-resolution enhancement processing on the high dynamic range image to obtain an enhanced image of the image to be processed.

[0124] Based on any embodiment of this application, the image quality enhancement device of this application further includes: an instance acquisition module, configured to acquire two instances of the conversion enhancement model, respectively serving as a first conversion enhancement model and a second conversion enhancement model; and a training implementation module, configured to train the first conversion enhancement model and the second conversion enhancement model to convergence using different training sets, wherein the image performance corresponding to the difference between the sample images and their supervision images in the first training set used by the first conversion enhancement model is lower than the image performance corresponding to the difference between the sample images and their supervision images in the second training set used by the second conversion enhancement model.

[0125] Based on any embodiment of this application, the image performance is reflected in the difference in the color gamut range of pixel values; the larger the color gamut range, the better the image performance.

[0126] Based on any embodiment of this application, the image quality enhancement device of this application further includes: a video acquisition module, configured to activate the camera unit of the terminal device to acquire an environmental preview video; a target recognition module, configured to perform target recognition on the image frames in the environmental preview video to determine that there is a product image therein; and an image determination module, configured to use the image frame containing the product image as the image to be processed.

[0127] Another embodiment of this application also provides an image quality enhancement device. For example... Figure 12 The diagram shows the internal structure of an image enhancement device. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database stores information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement an image enhancement method.

[0128] The processor of the image enhancement device provides computing and control capabilities, supporting the operation of the entire device. The memory of the device can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the image enhancement method of this application. The network interface of the device is used for communication with a terminal.

[0129] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the image quality enhancement device to which the solution of this application is applied. A specific image quality enhancement device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0130] In this embodiment, the processor is used to execute... Figure 11 The specific functions of each module are described, and the memory stores the program code and various data required to execute the above modules or sub-modules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the image quality enhancement device of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0131] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the image quality enhancement method of any embodiment of this application.

[0132] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0133] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0134] In summary, this application can distinguish between different image qualities and different situations such as the presence of facial images, and uses a conversion enhancement model with corresponding image effect performance to perform corresponding image quality enhancement processing for different situations, thereby obtaining corresponding high dynamic range images. This allows different conversion enhancement models to be trained in a targeted manner, thus obtaining high dynamic range images with better image quality for the images to be processed.

Claims

1. An image quality enhancement method characterized by comprising: include: Identifying the image quality level and the presence of facial images in the image to be processed includes: performing image compression and restoration on the image to be processed, employing an image compression and restoration model including a backbone network and a control prediction network, wherein the control prediction network predicts the corresponding compression ratio based on the image semantic features extracted by the backbone network, and uses the compression ratio to generate control parameters to adjust the image restoration process of the backbone network; determining whether the compression ratio is higher than a preset threshold, and if it is higher than the preset threshold, determining that the image to be processed is a low-level image, otherwise it is a high-level image; and performing face detection on the restored image to determine whether facial images are present. When the image quality of the image to be processed is low or contains a face image, a first conversion enhancement model is used to generate a high dynamic range image of the image to be processed; otherwise, a second conversion enhancement model is used to generate a high dynamic range image of the image to be processed. The image effect performance of the second conversion enhancement model is higher than that of the first conversion enhancement model. Wherein, the first conversion enhancement model is a model trained to convergence using the first training set, and the second conversion enhancement model is a model trained to convergence using the second training set. The difference in image performance between the sample images and the supervised images corresponding to the first training set is smaller than the difference in image performance between the sample images and the supervised images corresponding to the second training set.

2. The image quality enhancement method according to claim 1, characterized in that, Image compression and restoration are performed on the image to be processed, employing an image compression and restoration model including a backbone network and a control prediction network. The control prediction network predicts the corresponding compression ratio based on the image semantic features extracted by the backbone network, and uses this compression ratio to generate control parameters to adjust the image restoration process of the backbone network, including: The backbone network in the image compression and restoration model is used to extract features from the image to be processed to obtain image semantic features, and image restoration is performed on the image to be processed based on the image semantic features. The control prediction network in the image compression and restoration model is used to predict the corresponding compression ratio based on the image semantic features, and the compression ratio is used to control the image restoration process of the backbone network.

3. The image quality enhancement method according to claim 1, characterized in that, After generating the high dynamic range image of the image to be processed, the process includes: The high dynamic range image is subjected to super-resolution enhancement processing to obtain the enhanced image of the image to be processed.

4. The image quality enhancement method according to any one of claims 1 to 3, characterized in that, Before identifying the image quality level of the image to be processed and whether a face image exists, the process includes: Obtain two instances of the conversion enhancement model, which will be used as the first conversion enhancement model and the second conversion enhancement model, respectively. The first conversion enhancement model and the second conversion enhancement model are trained to convergence using different training sets. The image performance corresponding to the difference between the sample images and the supervised images in the first training set used by the first conversion enhancement model is lower than that corresponding to the difference between the sample images and the supervised images in the second training set used by the second conversion enhancement model.

5. The image quality enhancement method according to any one of claims 1 to 3, characterized in that, The image performance is reflected in the difference in the color gamut range of pixel values; the larger the color gamut range, the better the image performance.

6. The image quality enhancement method according to any one of claims 1 to 3, characterized in that, Before identifying the image quality level of the image to be processed and whether a face image exists, the process includes: The camera unit of the terminal device is activated to acquire an environmental preview video; Target recognition is performed on the image frames in the environmental preview video to determine the presence of product images within them; The image frame containing the product image is used as the image to be processed.

7. An image quality enhancement device, characterized in that, include: The image recognition module is configured to identify the image quality level of the image to be processed and whether a face image exists. This includes: performing image compression and restoration on the image to be processed using an image compression and restoration model comprising a backbone network and a control prediction network; the control prediction network predicting the corresponding compression ratio based on the image semantic features extracted by the backbone network, and using this compression ratio to generate control parameters to adjust the image restoration process of the backbone network; determining whether the compression ratio is higher than a preset threshold; if it is higher than the preset threshold, determining the image to be processed as a low-quality image; otherwise, determining it as a high-quality image; and performing face detection on the restored image to determine whether a face image exists within it. The image enhancement module is configured to generate a high dynamic range image of the image to be processed using a first conversion enhancement model when the image quality of the image to be processed is of a low level or when a face image is present; otherwise, it uses a second conversion enhancement model to generate a high dynamic range image of the image to be processed. The image effect performance of the second conversion enhancement model is higher than that of the first conversion enhancement model. Wherein, the first conversion enhancement model is a model trained to convergence using the first training set, and the second conversion enhancement model is a model trained to convergence using the second training set. The difference in image performance between the sample images and the supervised images corresponding to the first training set is smaller than the difference in image performance between the sample images and the supervised images corresponding to the second training set.

8. The image quality enhancement device according to claim 7, characterized in that, Image compression and restoration are performed on the image to be processed, employing an image compression and restoration model including a backbone network and a control prediction network. The control prediction network predicts the corresponding compression ratio based on the image semantic features extracted by the backbone network, and uses this compression ratio to generate control parameters to adjust the image restoration process of the backbone network, including: The backbone network in the image compression and restoration model is used to extract features from the image to be processed to obtain image semantic features, and image restoration is performed on the image to be processed based on the image semantic features. The control prediction network in the image compression and restoration model is used to predict the corresponding compression ratio based on the image semantic features, and the compression ratio is used to control the image restoration process of the backbone network.

9. An image quality enhancement device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 6.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 6, which, when invoked by a computer, executes the steps included in the corresponding method.

Images