An image enhancement method, device and electronic equipment

Abstract

Embodiments of the present application provide an image enhancement method and device and electronic equipment to solve the problem of low image quality such as color distortion, contrast distortion, detail loss, etc. in the prior art. For each pixel point in the enhanced image, the first pixel value of the pixel point and the second pixel value of the pixel point corresponding to the pixel point in the to-be-enhanced image are obtained, the probability that the to-be-enhanced image is a non-natural image and the probability that the to-be-enhanced image is a natural image are respectively taken as the weights corresponding to the second pixel value and the first pixel value, and the target pixel value of the corresponding pixel point of the pixel point in the fused target image is determined. Since the to-be-enhanced image and the enhanced image are fused according to the probability that the to-be-enhanced image is a non-natural image and the probability that the to-be-enhanced image is a natural image, the fused target image is generated, so the risk of color distortion, contrast distortion, detail loss, etc. caused by directly enhancing the image can be reduced, thereby the image quality can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to computer vision image technology, and in particular to an image enhancement method and device and electronic equipment. BACKGROUND

[0002] In recent years, with the continuous improvement of display brightness and display color gamut of display devices, the requirements for high contrast and wide color gamut images are also increasing. Due to the limitations of shooting scenes and shooting devices, the position and lighting conditions of the light source are uncontrollable in many cases. The areas of underexposure or overexposure on the image are almost invisible to details. At present, many images obtained by shooting or recording may have problems such as overexposure, underexposure, low saturation, low contrast, and poor visual experience.

[0003] In order to enable people to experience high-quality pictures on devices with high display capabilities without using higher-end devices to re-shoot and produce video resources, the related art usually uses an image enhancement method to process and enhance the contrast and color of the image. However, the image enhancement method enhances all input images. However, non-natural images are generally made by the author according to his own intention and are not limited by shooting devices, conditions, etc., so the contrast and color of such images are already appropriate. If non-natural images are enhanced without selection, color distortion, contrast distortion, and loss of details may occur. SUMMARY

[0004] Embodiments of the present application provide an image enhancement method, device and electronic equipment to solve the problem of low image quality such as color distortion, contrast distortion, and loss of details in the prior art.

[0005] In a first aspect, embodiments of the present application provide an image enhancement method, which comprises:

[0006] obtaining a to-be-enhanced image, inputting the to-be-enhanced image into a pre-trained non-natural image recognition model, obtaining probabilities that the to-be-enhanced image is a non-natural image and a natural image output by the non-natural image recognition model, and obtaining an enhanced image corresponding to the to-be-enhanced image;

[0007] For each pixel point in the enhanced image, obtaining a first pixel value of the pixel point in the enhanced image and a second pixel value of a pixel point corresponding to the pixel point in the to-be-enhanced image, taking the probabilities that the to-be-enhanced image is a non-natural image and a natural image as weights corresponding to the second pixel value and the first pixel value respectively, and determining a target pixel value of a corresponding pixel point in a fused target image of the pixel point.

[0008] In a second aspect, embodiments of the present application also provide an image enhancement device, which comprises:

[0009] receiving an image to be enhanced, inputting the image to be enhanced into a pre-trained non-natural image recognition model, obtaining probabilities that the image to be enhanced is a non-natural image and that the image to be enhanced is a natural image output by the non-natural image recognition model, and obtaining an enhanced image corresponding to the image to be enhanced;

[0010] processing the enhanced image, for each pixel point in the enhanced image, obtaining a first pixel value of the pixel point in the enhanced image and a second pixel value of a pixel point corresponding to the pixel point in the image to be enhanced, taking the probabilities that the image to be enhanced is a non-natural image and that the image to be enhanced is a natural image as weights corresponding to the second pixel value and the first pixel value respectively, and determining a target pixel value of a pixel point corresponding to the pixel point in a target image after fusion.

[0011] In a third aspect, an electronic device is provided, and the electronic device comprises:

[0012] a processor and a memory;

[0013] the memory is configured to store instructions executable by the processor;

[0014] the processor is configured to execute the instructions to implement the image enhancement method according to any one of the preceding aspects.

[0015] In the embodiment of the present application, the image to be enhanced is obtained, the image to be enhanced is input into the pre-trained non-natural image recognition model, the probability that the image to be enhanced is a non-natural image and a natural image output by the non-natural image recognition model is obtained, the enhanced image corresponding to the image to be enhanced is obtained, for each pixel point in the enhanced image, the first pixel value of the pixel point in the enhanced image and the second pixel value of the corresponding pixel point in the image to be enhanced are obtained, the probability that the image to be enhanced is a non-natural image and a natural image is respectively taken as the weight corresponding to the second pixel value and the first pixel value, and the target pixel value of the corresponding pixel point of the pixel point in the fused target image is determined. In the embodiment of the present application, the probability that the image to be enhanced is a natural image and a non-natural image is obtained, the enhanced image corresponding to the image to be enhanced is obtained, for each pixel point in the enhanced image, the first pixel value of the pixel point and the second pixel value of the corresponding pixel point in the image to be enhanced are obtained, the probability that the image to be enhanced is a non-natural image and a natural image is respectively taken as the weight corresponding to the second pixel value and the first pixel value, and the target pixel value of the corresponding pixel point of the pixel point in the fused target image is determined. Since the image to be enhanced and the enhanced image are fused according to the probability that the image to be enhanced is a non-natural image and a natural image to generate the fused target image, the risk of color distortion, contrast distortion, detail loss and the like caused by directly enhancing the image can be reduced, so that the image quality can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 A VR device schematic diagram provided for the embodiments of the present application;

[0018] Figure 2 A vehicle-mounted device schematic diagram provided for the embodiments of the present application;

[0019] Figure 3 A television schematic diagram provided for the embodiments of the present application;

[0020] Figure 4 An image enhancement process schematic diagram provided for the embodiments of the present application;

[0021] Figure 5 A structure schematic diagram of a non-natural image recognition model provided for the embodiments of the present application;

[0022] Figure 6A content enhancement model training process schematic diagram provided by an embodiment of the present application;

[0023] Figure 7 A content enhancement model structure schematic diagram provided by an embodiment of the present application;

[0024] Figure 8 A module structure schematic diagram inside a content enhancement model provided by an embodiment of the present application;

[0025] Figure 9 A feature extraction network structure schematic diagram provided by an embodiment of the present application;

[0026] Figure 10 A detailed image enhancement process schematic diagram provided by an embodiment of the present application;

[0027] Figure 11 A process schematic diagram of training a non-natural image recognition model provided by an embodiment of the present application;

[0028] Figure 12 A process schematic diagram of training a perception enhancement model provided by an embodiment of the present application;

[0029] Figure 13 A perception enhancement model structure schematic diagram provided by an embodiment of the present application;

[0030] Figure 14 An image enhancement device structure schematic diagram provided by an embodiment of the present application;

[0031] Figure 15 An electronic device structure schematic diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] The application will be described in further detail below with reference to the drawings. It should be apparent that the described embodiments are only a part of all embodiments of the application, and are not all-inclusive of all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0033] In recent years, with the continuous improvement of display brightness and display color gamut of display devices, the requirements for high contrast and wide color gamut images are also increasing. Due to the limitations of shooting scenes and shooting equipment, etc., in many cases, the position and lighting conditions of the light source are uncontrollable. In such cases, the areas of the image that are underexposed or overexposed have almost no details, and many images obtained by shooting or recording may have problems such as overexposure, underexposure, low saturation, low contrast, etc., resulting in poor visual experience.

[0034] In order to make people experience high-quality pictures on devices with high display capabilities without re-shooting video resources using higher-end devices, the related art usually uses an image enhancement method to process and enhance the contrast and color of images. However, the image enhancement method enhances all input images. However, non-natural images are generally made by the author according to his own intention and are not limited by the shooting device, condition, etc., so the contrast and color of such images are already appropriate. If non-natural images are not selectively enhanced, color distortion, contrast distortion, loss of details, and other problems may occur.

[0035] In order to improve image quality, the embodiments of the present application provide an image enhancement method, device and electronic equipment. The present application is mainly applied to image display related scenarios. The image enhancement method provided by the embodiments of the present application is applied to an electronic equipment, which can be a television, a vehicle-mounted device, a VR device, a mobile phone, a tablet computer or a server, etc. The television is as shown in Figure 1 The VR device is as shown in Figure 2 The vehicle-mounted device is as shown in Figure 3 The electronic equipment enhances the to-be-enhanced image. If the electronic equipment is a television, a vehicle-mounted device, a VR device, a mobile phone, a tablet computer or a server, etc. with a display, the electronic equipment can control the display of the electronic equipment to display the enhanced image. If the electronic equipment is a server or other smart device without a display, the electronic equipment can control the display connected to the electronic equipment to display the enhanced image, or send the enhanced image to a device with a display function, and the device with a display function displays the target image.

[0036] Figure 4 An image enhancement process provided by the embodiments of the present application is shown in the figure. The process includes the following steps:

[0037] S401: Obtain a to-be-enhanced image, input the to-be-enhanced image into a pre-trained non-natural image recognition model, obtain the probability that the to-be-enhanced image output by the non-natural image recognition model is a non-natural image and a natural image, and obtain an enhanced image corresponding to the to-be-enhanced image.

[0038] The image enhancement method provided by the embodiments of the present application is applied to an electronic equipment, which can be a television, a vehicle-mounted device, a VR device, a mobile phone, a tablet computer or a server, etc.

[0039] In the embodiments of the present application, the electronic equipment is taken as a television for example. In order to improve image quality, the television can obtain a to-be-enhanced image, which can be collected by a collection unit of the television. The to-be-enhanced image can be a natural image or a non-natural image.

[0040] In order to improve the image quality, the television locally stores a pre-trained non-natural image recognition model. After receiving the image to be enhanced, the television can input the image to be enhanced into the pre-trained non-natural image recognition model, and obtain the output of the pre-trained non-natural image recognition model. The output is the probability that the image to be enhanced is a non-natural image and the probability that the image to be enhanced is a natural image.

[0041] The non-natural image recognition model described in the embodiments of the present application can filter the input image. Since natural images and non-natural images have obvious differences in texture, edges, etc., the non-natural image recognition model can accurately determine the probability that the image to be enhanced is a non-natural image and a natural image.

[0042] The television can also obtain an enhanced image corresponding to the image to be enhanced. Specifically, in order to obtain the enhanced image corresponding to the image to be enhanced, the image to be enhanced can be input into a pre-trained image enhancement model, and the output of the image enhancement model is obtained. The output of the image enhancement model is the enhanced image corresponding to the image to be enhanced. Specifically, how to obtain the enhanced image of a certain image is a prior art, which will not be described here.

[0043] Figure 5 A structure diagram of a non-natural image recognition model provided in the embodiments of the present application.

[0044] From Figure 5 It can be seen that the non-natural image recognition model is composed of a 3*3 convolution layer (Conv), an adaptive maximum pooling layer (Adaptive max pooling), a maximum pooling layer (Max pooling) and an activation function. The activation function used in the non-natural image recognition model is Sigmoid. Specifically, the structure of the non-natural image recognition model is 3*3 Conv, Adaptive max pooling, 3*3 Conv, Max pooling, 3*3 Conv, Max pooling, 3*3 Conv and Sigmoid in turn. Among them, Figure 5 The 2, 16, 32 and 64 appearing in the above formula are channel numbers.

[0045] S402: For each pixel point in the enhanced image, obtain the first pixel value of the pixel point in the enhanced image and the second pixel value of the pixel point corresponding to the pixel point in the image to be enhanced. The probability that the image to be enhanced is a non-natural image and the probability that the image to be enhanced is a natural image are respectively taken as the weights corresponding to the second pixel value and the first pixel value, and the target pixel value of the corresponding pixel point of the pixel point in the fused target image is determined.

[0046] To improve the image quality, the television can fuse the to-be-enhanced image and the enhanced image. Specifically, for each pixel point in the enhanced image, a first pixel value of the pixel point in the enhanced image is obtained, and a second pixel value of a pixel point corresponding to the pixel point in the to-be-enhanced image is obtained. The enhanced image is obtained according to the to-be-enhanced image, and the enhanced image and the to-be-enhanced image have the same size. Therefore, each pixel point in the enhanced image has a pixel point with the same position in the to-be-enhanced image, and the pixel point with the same position is the pixel point corresponding to the pixel point. To improve the image quality, the television can take the probability that the to-be-enhanced image is a natural image as the weight corresponding to the first pixel value, and take the probability that the to-be-enhanced image is a non-natural image as the weight corresponding to the second pixel value, to determine a target pixel value of a pixel point corresponding to the pixel point in a target image after fusion. In this way, the target pixel value of each pixel point in the target image can be determined, and the target image after fusion can be accurately and effectively determined.

[0047] After obtaining the target image, the television can control the display of the television to display the target image, or control other connected displays to display the target image.

[0048] That is, if the probability that the to-be-enhanced image is a non-natural image is 1, the target pixel value of each pixel point in the generated target image is consistent with the pixel value of the corresponding pixel point in the to-be-enhanced image, that is, the generated target image is equivalent to the to-be-enhanced image. Therefore, when the to-be-enhanced image is a non-natural image, no enhancement is performed, and the risk of color distortion, contrast distortion, detail loss, and the like caused by enhancement of the non-natural image is reduced, to protect the detail features.

[0049] In the embodiments of the present application, the electronic device obtains the probability that the to-be-enhanced image is a natural image and a non-natural image, and obtains the enhanced image corresponding to the to-be-enhanced image. For each pixel point in the enhanced image, a first pixel value of the pixel point and a second pixel value of a pixel point corresponding to the pixel point in the to-be-enhanced image are obtained. The probability that the to-be-enhanced image is a non-natural image and a natural image is taken as the weight corresponding to the second pixel value and the first pixel value, respectively, to determine a target pixel value of a pixel point corresponding to the pixel point in a target image after fusion. Since the electronic device fuses the to-be-enhanced image and the enhanced image according to the probability that the to-be-enhanced image is a non-natural image and a natural image, and generates a target image after fusion, the risk of color distortion, contrast distortion, detail loss, and the like caused by direct image enhancement can be reduced, and the image quality can be improved.

[0050] To accurately obtain the enhanced image, on the basis of the above-mentioned embodiments, in the embodiments of the present application, the obtaining of the enhanced image corresponding to the to-be-enhanced image comprises:

[0051] The image to be enhanced is input into the content enhancement model to obtain the image to be processed with enhanced contrast and color output by the content enhancement model.

[0052] The image to be processed is input into the perception enhancement model to obtain the enhanced image output by the perception enhancement model; wherein, the enhanced image is an image that enhances the scene-related perception information of the image to be enhanced.

[0053] In practical applications, image enhancement models are typically used to enhance images. These models are usually trained using content-based loss. However, models trained using content-based loss have limited training datasets with diverse image scenes and categories, failing to cover the complex enhancement needs of real-world applications. The resulting image enhancement models often exhibit limited enhancement effects and weak adaptability. To accurately acquire enhanced images that more closely resemble what the naked eye observes, the television pre-stores trained content enhancement and perceptual enhancement models. The television can input the image to be enhanced into the pre-trained content enhancement model, obtaining its output image for processing. This image enhances contrast and color, providing initial enhancement to the low-quality input image. After acquiring the image for processing, it can be input into the pre-trained perceptual enhancement model, obtaining its output image, which is the enhanced image. This enhanced image adds perceptual information related to the scene from which the image to be enhanced was acquired. Perceptual enhancement models can extract adaptive features from the input image, thereby adding more image-related details to the initially enhanced image, i.e., the image to be processed.

[0054] In this embodiment, the enhancement of the image to be enhanced is performed using both a content enhancement model and a perceptual enhancement model. Essentially, the enhancement model includes a content enhancement sub-model and a perceptual enhancement sub-model, achieving image enhancement through a multi-stage, progressive approach from content to perception. The content enhancement model is trained based on image content, while the perceptual enhancement model is trained based on multi-dimensional image features. The perceptual enhancement model adds detailed features related to the image scene and category to the image to be processed, improving the image's adaptability and resulting in a higher quality enhanced image.

[0055] In order to obtain the trained content augmentation model, this application embodiment stores a sample set for training. The sample images in the sample set include images collected at different times and different unnatural images. In order to facilitate the training of the content augmentation model, the sample set also stores a corresponding labeled image for each sample image. The labeled image is an image with the same content as the sample image and of high quality. Specifically, the labeled image can be an image re-shot using a more advanced device.

[0056] After obtaining any sample image and its corresponding labeled image from the sample set, the sample image is input into the original content enhancement model, which outputs the corresponding image. To train the original content enhancement model, the television locally stores a loss function. Based on the labeled image, the output image, and the loss function, the loss value corresponding to the sample image is determined. The original content enhancement model is then trained using this loss value, specifically adjusting its internal parameters. When a preset condition is met, the trained content enhancement model is obtained. This preset condition can be that the number of sample images in the sample set whose determined loss values ​​are less than a preset threshold is greater than a set number. The loss function is:

[0057]

[0058]

[0059] Among them, L c This is the loss value. To output the image to be processed, I G To label the image to be processed, Tanh(*) is the hyperbolic tangent function, l ca For color bias loss, ||*||1 represents the L1 norm, and · represents the inner product.

[0060] Figure 6 This application provides a schematic diagram of the training process of a content enhancement model, which includes the following steps:

[0061] S601: Obtain any sample image from the sample set and the labeled image saved for that sample image.

[0062] S602: Input the sample image into the original content enhancement model and obtain the output image of the original content enhancement model.

[0063] S603: Determine the corresponding loss value based on the labeled image, the output image, and the loss function.

[0064] S604: Train the original content enhancement model based on this loss value.

[0065] Because the content enhancement model adopts a relatively lightweight structure and fewer parameters, it first learns from the content itself to allow the network to initially learn the basic principles of image enhancement. Therefore, in this stage, the loss function uses the content error in the sample set as the learning direction, enabling the content enhancement model to output an image with basic enhancements in contrast and color. The pixel deviation between the low-quality input image and its corresponding high-quality image in the sample set can constrain the direction of image brightness and saturation enhancement, while their cosine similarity can constrain the direction of tone enhancement.

[0066] Conventional image enhancement models are trained using the pixel difference between low-quality images and their corresponding high-quality images in the training set as the loss, enabling the model to learn how to modify the data distribution within the training set. However, in real-world scenarios, enhancing the contrast and color of an image is a complex process. Besides considering the data distribution of the image itself, it's also necessary to consider its scene, category, and other information to achieve an enhancement effect that conforms to human visual characteristics. The perceptual enhancement model provided in this application can achieve an enhancement effect that conforms to human visual characteristics.

[0067] The content enhancement model consists of a conditional coding module, a feature mapping module, and a content reconstruction module. The feature mapping module further extracts global image features from the image to be enhanced, mapping these global features onto local features used for image enhancement. The conditional coding module outputs modulation information to modulate the features output by the feature mapping module, resulting in more image-adaptive features. Finally, the content reconstruction module analyzes the features to reconstruct the image to be processed after preliminary enhancement of contrast and color. The modulation operation involves multiplying and adding the two 64-dimensional 1x1 feature maps output by the conditional coding module with the feature maps output by the feature mapping module along the channel dimension.

[0068] Figure 7 This is a schematic diagram of the structure of a content enhancement model provided in an embodiment of this application.

[0069] Depend on Figure 7 As can be seen, the content enhancement model consists of a 3*3 Conv, an activation function, and a global average pooling layer. The activation function used in this content enhancement model is Noisy Rectified Linear Activation (ReLU), where... Figure 7 The first layer consists of 3*3Conv, ReLU, 3*3Conv, ReLU, Global average pooling, 3*3Conv, and 3*3Conv, which are conditional coding modules. Figure 7The second layer, consisting of 3*3Conv, ReLU, 3*3Conv, ReLU, and 3*3Conv, is a feature mapping module. Figure 7 The 3*3Conv, ReLU, and 3*3Conv modules on the right-middle side (shown as left and right in the diagram) are content reconstruction modules. Figure 7 The numbers 16, 32, 64, and 3 appearing in the text refer to the channel number.

[0070] Figure 8 This is a schematic diagram of the internal module structure of a content enhancement model provided in an embodiment of this application.

[0071] Depend on Figure 8 It can be seen that the content enhancement module includes a conditional encoding module, a feature mapping module, and a content reconstruction module. The conditional encoding module and the feature mapping module process the input of the content enhancement model, respectively, and the content reconstruction module processes the output of the conditional encoding module and the feature mapping module.

[0072] To accurately determine the probability that the image to be enhanced is a non-natural image and a natural image, based on the above embodiments, in this embodiment, after obtaining the image to be enhanced and before inputting the image to be enhanced into a pre-trained non-natural image recognition model, the method further includes:

[0073] The image to be enhanced is input into a feature extraction network to obtain the global image features of the image to be enhanced output by the feature extraction network.

[0074] The step of inputting the image to be enhanced into a pre-trained unnatural image recognition model includes:

[0075] The global image features of the image to be enhanced are input into a pre-trained non-natural image recognition model.

[0076] To accurately determine the probability of an image to be enhanced being either unnatural or natural, the television pre-stores a feature extraction network. After receiving the image to be enhanced, the television inputs it into the feature extraction network and obtains the global image features output by the network. The specific details of how the feature extraction network extracts image features are existing technologies and will not be elaborated here. The purpose of the feature extraction network is to extract global image features that can be shared by both the unnatural image recognition model and the content enhancement model.

[0077] Figure 9 This is a schematic diagram of the structure of a feature extraction network provided in an embodiment of this application.

[0078] Depend on Figure 9As can be seen, the feature extraction network consists of a 3*3 Conv and an activation function, specifically ReLU. The network's structure is as follows: 3*3 Conv, ReLU, 3*3 Conv, ReLU, 3*3 Conv, and ReLU. Figure 9 The numbers 3, 8, and 16 appearing in the text refer to the channel number.

[0079] After obtaining the global features of the image to be enhanced, these features can be input into a pre-trained non-natural image recognition model to obtain its output, which represents the probability that the image to be enhanced is either a non-natural or natural image. Furthermore, the television can also input the global features of the image to be enhanced into a content enhancement model to obtain the output image to be processed.

[0080] Figure 10 A detailed schematic diagram of an image enhancement process provided for an embodiment of this application is shown, the process including the following steps:

[0081] Figure 10 The following example illustrates how to first obtain the probability that the image to be enhanced is a non-natural image and a natural image, and then obtain the enhanced image corresponding to the image to be enhanced.

[0082] S1001: Obtain the image to be enhanced.

[0083] S1002: Input the image to be enhanced into the feature extraction network to obtain the global features of the image to be enhanced.

[0084] S1003: Input the global image features of the image to be enhanced into the non-natural image recognition model to obtain the probability that the image to be enhanced is a non-natural image and a natural image.

[0085] S1004: Obtain the enhanced image corresponding to the image to be enhanced based on the content enhancement model and the perception enhancement model.

[0086] S1005: Based on the probability that the image to be enhanced is a non-natural image or a natural image, process the image to be enhanced and the enhanced image to obtain the corresponding fused target image.

[0087] Specifically, how to obtain the target image has been described in the above embodiments and will not be repeated here.

[0088] In order to accurately obtain the non-natural image recognition model, based on the above embodiments, the non-natural image recognition model in this application embodiment is trained in the following way:

[0089] Obtain the annotation probabilities of any first sample image in the first sample set as a non-natural image and a natural image;

[0090] Input the first sample image into the original recognition model, and obtain the output probability of the first sample image being a non-natural image and a natural image output by the original recognition model;

[0091] Based on the labeled probability, the output probability, and the first loss function, the first loss value corresponding to the first sample image is determined;

[0092] The original recognition model is trained based on the first loss value.

[0093] In order to obtain the non-natural image recognition model, a first sample set for training is stored. The first sample images in the first sample set include images collected at different times and different non-natural images. In order to facilitate the training of the original recognition model, the first sample set also stores the annotation probability of the first sample image being a non-natural image and a natural image for each first sample image.

[0094] After obtaining any first sample image from the first sample set and the labeled probabilities of the first sample image being a non-natural image and a natural image, the first sample image is input into the original recognition model. The original recognition model outputs the output probabilities of the first sample image being a non-natural image and a natural image. To train the original recognition model and obtain a trained non-natural image recognition model, the television locally stores a first loss function. Based on the labeled probabilities, output probabilities, and the first loss function, a first loss value corresponding to the first sample image is determined, and the original recognition model is trained based on this first loss value. When a preset condition is met, a trained non-natural image recognition model is obtained. This preset condition may be that the number of first loss values ​​corresponding to the first sample images in the first sample set that are less than a preset threshold is greater than a set number.

[0095] To obtain a trained non-natural image recognition model, based on the above embodiments, in this embodiment, the first loss function is:

[0096] l = -[y i ·log(p i )+(1-y i )·log(1-p i )]

[0097] Where l is the first loss function, y i p is the labeled probability that the first sample image i is a non-natural image. i Let y be the output probability that the first sample image i is a non-natural image, 1-y i Let i be the labeled probability of the first sample image and i be a natural image, 1-pi Let i be the output probability of the first sample image and i be the natural image.

[0098] In this embodiment of the application, the first loss function can be: l = -[y i ·log(p i )+(1-y i )·log(1-p i )]

[0099] Where l is the first loss value, i is the identifier of the first sample image, and y i p is the labeled probability that the first sample image i is a non-natural image. i Let y be the output probability that the first sample image i is a non-natural image, 1-y i Let i be the labeled probability of the first sample image and i be a natural image, 1-p i Let i be the output probability of the first sample image and i be the natural image.

[0100] Because natural and unnatural images differ significantly in texture and color, they are easily distinguishable using unnatural image recognition models. Unnatural images are generally designed by the author according to their own intentions, and their details and colors at different brightness levels are often already quite reasonable. Further enhancement could easily destroy the information the author intended to convey through the image. Therefore, this application proposes an unnatural image recognition model to distinguish unnatural images, thereby improving the user's visual experience. If the image to be enhanced is an unnatural image, the probability of it being an unnatural image is output as 1, and the generated target image is the image to be enhanced; if the image to be enhanced is a natural image, the probability of it being a natural image is output as 1, and the generated target image is the enhanced image.

[0101] Figure 11 This application provides a schematic diagram of a process for training a non-natural image recognition model, which includes the following steps:

[0102] S1101: Obtain the annotation probabilities of any first sample image in the first sample set as a non-natural image and a natural image.

[0103] S1102: Input the first sample image into the original recognition model and obtain the output probability of the first sample image being a non-natural image and a natural image from the original recognition model.

[0104] S1103: Determine the first loss value corresponding to the first sample image based on the labeled probability, output probability, and first loss function.

[0105] S1104: Train the original recognition model based on the first loss value.

[0106] To obtain the trained perception enhancement model, based on the above embodiments, the perception enhancement model in this embodiment is trained in the following manner:

[0107] Obtain any second sample image from the second sample set and the corresponding labeled image to be processed; input the second sample image into the content enhancement model and obtain the output image to be processed from the content enhancement model.

[0108] The output image to be processed is input into the original perception enhancement model to obtain the output enhanced image output by the original perception enhancement model.

[0109] Based on the feature maps obtained by the labeled image to be processed and the output enhanced image through the VGG network, and the second loss function, the second loss value corresponding to the second sample image is determined.

[0110] The original perception enhancement model is trained based on the second loss value determined for each second sample image in the second sample set.

[0111] In order to obtain the perception recognition model, this application embodiment stores a second sample set for training. The second sample images in the second sample set include images collected in different scenes and at different times, as well as different unnatural images. In order to facilitate the training of the original perception enhancement model, the second sample set also stores a labeled image to be processed corresponding to each second sample image. The labeled image to be processed corresponding to the second sample image is an image with enhanced contrast and color collected after adjusting the acquisition device.

[0112] After obtaining any second sample image and its corresponding labeled image from the second sample set, the second sample image is input into the content enhancement model to obtain its output, which is the output image to be processed. The output image to be processed is then input into the original perceptual enhancement model to obtain its output enhanced image. After obtaining the output enhanced image, the labeled image to be processed and the output enhanced image can be input into the VGG network to obtain the feature map output by the VGG network. This feature map can be a level l feature map. Based on the two output feature maps and the second loss function, the second loss value corresponding to the second sample image is determined. The original perceptual enhancement model is then trained based on this first loss value. When a preset condition is met, the trained perceptual enhancement model is obtained. This preset condition can be that the number of second loss values ​​corresponding to the second sample images in the second sample set that are less than a preset threshold is greater than a set number.

[0113] To obtain the trained perception enhancement model, based on the above embodiments, in this embodiment of the application, the second loss function is:

[0114]

[0115] in, and N represents the feature maps obtained by passing the labeled image to be processed and the output enhanced image through the VGG network, respectively. l This represents the total number of elements in the feature map output by the VGG network.

[0116] In this embodiment of the application, the second loss function can be:

[0117]

[0118] in, and N represents the feature maps obtained by passing the labeled image to be processed and the output enhanced image through the VGG network, respectively. l The total number of elements in the feature map output by the VGG network is represented by ||·||1, which represents the L1 norm. The specific method for obtaining the L1 norm is a current technology and will not be elaborated here.

[0119] Image enhancement models based on neural networks are typically trained using only content as the loss function. However, in real-world applications, low-quality images often possess rich scenes and categories, resulting in highly complex image distributions. The image types in the sample set may not fully encompass the needs of practical applications. If only content is used as the training guide, it's impossible to adaptively enhance different input images, leading to a limited range of enhancement effects. This application introduces a perceptual enhancement model, guided by perceptual error, to imbue the initially enhanced image output by the content enhancement model with specific details required for different scenes and categories. This allows the model to learn contrast and color enhancement rules that are closer to the essence of the image, thereby enriching the model's expressive capabilities and resulting in enhanced images with better visual effects to the human eye.

[0120] Figure 12 This application provides a schematic diagram of a process for training a perception enhancement model, which includes the following steps:

[0121] S1201: Obtain any second sample image from the second sample set and the corresponding labeled image to be processed; input the second sample image into the content augmentation model and obtain the output image to be processed from the content augmentation model.

[0122] S1202: Input the output image to be processed into the original perception enhancement model to obtain the output enhanced image of the original perception enhancement model.

[0123] S1203: Based on the feature maps obtained by the VGG network from the labeled image to be processed and the output enhanced image, and the second loss function, determine the second loss value corresponding to the second sample image.

[0124] S1204: Train the original perception enhancement model based on the second loss value determined for each second sample image in the second sample set.

[0125] The perceptual enhancement model in this embodiment is composed of cascaded residual modules. Generally, image enhancement neural models are trained only by loss based on image content. This embodiment introduces a perceptual enhancement model to endow the local features of the modulated image to be processed generated by the content enhancement model with perceptual information, thereby obtaining the scene, category and other features of the current low-quality input image. This enables the perceptual enhancement subnetwork to perform adaptive enhancement according to the characteristics of the input image, improve the overall adaptive ability of the model, and have better performance on complex input images in practical applications.

[0126] Figure 13 This is a schematic diagram of the structure of a perception enhancement model provided in an embodiment of this application.

[0127] Depend on Figure 13 It can be seen that the perceptual enhancement model consists of a 3*3Conv array and an activation function. The activation function used in this perceptual enhancement model is ReLU. Specifically, the structure of this perceptual enhancement model is as follows: 3*3Conv, ReLU, 3*3Conv, ReLU, 3*3Conv, ReLU, and 3*3Conv. Figure 13 The 64 and 3 appearing in the text refer to the channel number.

[0128] To improve image quality, based on the above embodiments, in this embodiment, determining the target pixel value of the corresponding pixel in the fused target image includes:

[0129] Determine the first product of the first pixel value and its corresponding weight, and the second product of the second pixel value and its corresponding weight; determine the sum of the first product and the second product as the target pixel value of the corresponding pixel in the fused target image.

[0130] To improve image quality, the television can determine a first product of a first pixel value and its corresponding weight, where the corresponding weight represents the probability that the image to be enhanced is a natural image. It can also determine a second product of a second pixel value and its corresponding weight, where the corresponding weight represents the probability that the image to be enhanced is an unnatural image. After determining the first and second products, the sum of the first and second products can be determined as the target pixel value of the corresponding pixel in the fused target image.

[0131] In other words, after obtaining the probability p that the image to be enhanced is a non-natural image and the probability 1-p that it is a natural image, the enhanced image and the image to be enhanced are fused according to the ratio of 1-p to p to obtain the target image.

[0132] Figure 14 This is a schematic diagram of the structure of the image enhancement device provided in the embodiments of this application, as shown below. Figure 14 As shown, the device includes:

[0133] The receiving and acquisition module 1401 is used to acquire the image to be enhanced, input the image to be enhanced into a pre-trained non-natural image recognition model, acquire the probability that the image to be enhanced output by the non-natural image recognition model is a non-natural image and a natural image, and acquire the enhanced image corresponding to the image to be enhanced.

[0134] Processing module 1402 is used to obtain, for each pixel in the enhanced image, the first pixel value of the pixel in the enhanced image and the second pixel value of the corresponding pixel in the image to be enhanced, and to use the probabilities of the image to be enhanced being a non-natural image and a natural image as weights corresponding to the second pixel value and the first pixel value, respectively, to determine the target pixel value of the corresponding pixel in the fused target image.

[0135] In one possible implementation, the receiving and acquiring module 1401 is specifically used to input the image to be enhanced into a content enhancement model to acquire the image to be processed with enhanced contrast and color output by the content enhancement model; input the image to be processed into a perception enhancement model to acquire the enhanced image output by the perception enhancement model; wherein, the enhanced image is an image that enhances the scene-related perception information of the acquired image to be enhanced.

[0136] In one possible implementation, the receiving and acquiring module 1401 is further configured to input the image to be enhanced into a feature extraction network and acquire the global image features of the image to be enhanced output by the feature extraction network;

[0137] The receiving and acquiring module 1401 is specifically used to input the global image features of the image to be enhanced into a pre-trained non-natural image recognition model.

[0138] In one possible implementation, the processing module 1402 is further configured to obtain the annotation probabilities of any first sample image in the first sample set being a non-natural image and a natural image.

[0139] The first sample image is input into the original recognition model to obtain the output probabilities of the first sample image being a non-natural image and a natural image, respectively; based on the labeled probability, the output probability, and the first loss function, a first loss value corresponding to the first sample image is determined; and the original recognition model is trained based on the first loss value.

[0140] In one possible implementation, the processing module 1402 is further configured to: acquire any second sample image in the second sample set and the corresponding labeled image to be processed; input the second sample image into a content enhancement model to acquire the output image to be processed output by the content enhancement model; input the output image to be processed into an original perception enhancement model to acquire the output enhanced image output by the original perception enhancement model; determine a second loss value corresponding to the second sample image based on the feature map obtained by the labeled image to be processed and the output enhanced image through a VGG network, and a second loss function; and train the original perception enhancement model based on the second loss value determined for each second sample image in the second sample set.

[0141] In one possible implementation, the processing module 1402 is specifically used to determine a first product of a first pixel value and its corresponding weight, and a second product of a second pixel value and its corresponding weight; and to determine the sum of the first product and the second product as the target pixel value of the corresponding pixel in the fused target image.

[0142] Based on the same inventive concept Figure 15 A schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 15 As shown, it includes: one or more processors 1501 (including two) and a communication interface 1502.

[0143] Optionally, the electronic device also includes a memory 1503, which may include read-only memory and random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0144] In some implementations, such as Figure 15 As shown, memory 1503 stores the following elements: execution modules or data structures, or subsets thereof, or extended sets thereof. For example... Figure 15 As shown, by calling the operation instructions stored in memory 1503, the processor 1501 controls the processing operation of the head-end device and executes the corresponding operation. The processor can also be called a central processing unit (CPU).

[0145] like Figure 15 As shown, memory 1503 may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory 1503 may also include NVRAM. For example, in an application, the communication interface and memory are coupled together via bus system 1504, which may include, in addition to a data bus, a power bus, a control bus, and a status signal bus, etc. However, for clarity, in... Figure 15 The general labeled all buses as Bus System 1504.

[0146] The methods disclosed in some embodiments of this application can be applied to a processor or implemented by a processor. A processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by software instructions.

[0147] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the program is run on the electronic device, the electronic device executes the methods disclosed in some embodiments of this application.

[0148] Since the principle of the computer-readable storage medium in solving the problem is similar to that of the image enhancement method, the implementation of the computer-readable storage medium can be found in the embodiments of the method, and repeated details will not be described again.

[0149] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image enhancement method, characterized in that, The method includes: Obtain the image to be enhanced, input the image to be enhanced into a pre-trained non-natural image recognition model, obtain the probability that the image to be enhanced output by the non-natural image recognition model is a non-natural image and a natural image; and obtain the enhanced image corresponding to the image to be enhanced. For each pixel in the enhanced image, the first pixel value of the pixel in the enhanced image and the second pixel value of the corresponding pixel in the image to be enhanced are obtained. The probabilities of the image to be enhanced being a non-natural image and a natural image are respectively used as weights corresponding to the second pixel value and the first pixel value to determine the target pixel value of the corresponding pixel in the fused target image.

2. The method according to claim 1, characterized in that, The step of obtaining the enhanced image corresponding to the image to be enhanced includes: The image to be enhanced is input into the content enhancement model to obtain the image to be processed with enhanced contrast and color output by the content enhancement model. The image to be processed is input into the perception enhancement model to obtain the enhanced image output by the perception enhancement model; wherein, the enhanced image is an image that enhances the scene-related perception information of the image to be enhanced.

3. The method according to claim 1, characterized in that, After acquiring the image to be enhanced, and before inputting the image to be enhanced into a pre-trained non-natural image recognition model, the method further includes: The image to be enhanced is input into a feature extraction network to obtain the global image features of the image to be enhanced output by the feature extraction network. The step of inputting the image to be enhanced into a pre-trained unnatural image recognition model includes: The global image features of the image to be enhanced are input into a pre-trained non-natural image recognition model.

4. The method according to claim 1, characterized in that, The unnatural image recognition model is trained in the following way: Obtain the annotation probabilities of any first sample image in the first sample set as a non-natural image and a natural image; Input the first sample image into the original recognition model, and obtain the output probability of the first sample image being a non-natural image and a natural image output by the original recognition model; Based on the labeled probability, the output probability, and the first loss function, the first loss value corresponding to the first sample image is determined; The original recognition model is trained based on the first loss value.

5. The method according to claim 4, characterized in that, The first loss function is: l c =-[y i ·log(p i )+(1-y i )·log(1-p i )] Wherein, y i p is the labeled probability that the first sample image i is a non-natural image. i Let y be the output probability that the first sample image i is a non-natural image, 1-y i Let i be the labeled probability of the first sample image and i be a natural image, 1-p i Let i be the output probability of the first sample image and i be the natural image.

6. The method according to claim 2, characterized in that, The perception enhancement model is trained in the following way: Obtain any second sample image from the second sample set and the corresponding labeled image to be processed; input the second sample image into the content enhancement model and obtain the output image to be processed from the content enhancement model; The output image to be processed is input into the original perception enhancement model to obtain the output enhanced image output by the original perception enhancement model. Based on the feature maps obtained by the labeled image to be processed and the output enhanced image through the VGG network, and the second loss function, the second loss value corresponding to the second sample image is determined. The original perception enhancement model is trained based on the second loss value determined for each second sample image in the second sample set.

7. The method according to claim 6, characterized in that, The second loss function is: in, and N represents the feature maps obtained by passing the labeled image to be processed and the output enhanced image through the VGG network, respectively. l This represents the total number of elements in the feature map output by the VGG network.

8. The method according to claim 1, characterized in that, Determining the target pixel value of the corresponding pixel in the fused target image includes: Determine the first product of the first pixel value and its corresponding weight, and the second product of the second pixel value and its corresponding weight; determine the sum of the first product and the second product as the target pixel value of the corresponding pixel in the fused target image.

9. An image enhancement device, characterized in that, The device includes: The receiving and acquisition module is used to acquire the image to be enhanced, input the image to be enhanced into a pre-trained non-natural image recognition model, acquire the probability that the image to be enhanced output by the non-natural image recognition model is a non-natural image and a natural image, and acquire the enhanced image corresponding to the image to be enhanced. The processing module is used to obtain, for each pixel in the enhanced image, the first pixel value of the pixel in the enhanced image and the second pixel value of the corresponding pixel in the image to be enhanced, and to use the probabilities of the image to be enhanced being a non-natural image and a natural image as weights corresponding to the second pixel value and the first pixel value, respectively, to determine the target pixel value of the corresponding pixel in the fused target image.

10. An electronic device, characterized in that, The electronic device includes: Processor and memory; The memory is used to store the processor-executable instructions; The processor is configured to execute the instructions to implement the image enhancement method as described in any one of claims 1-8.

Images