Image enhancement method, image enhancement device, product and storage medium

Abstract

The embodiment of the invention provides an image enhancement method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-enhanced image; performing noise reduction processing on the to-be-enhanced image through at least two noise reduction models to obtain at least two noise reduction images corresponding to the noise reduction models; performing image fusion on the at least two noise reduction images to obtain a target noise reduction image; and performing image enhancement processing on the target noise reduction image through a generative enhancement model to obtain a target enhanced image. According to the method, high-quality noise reduction processing on the to-be-enhanced image is realized through complementary characteristics of a plurality of noise reduction models on fidelity and anti-noise capability. The target noise reduction image is subjected to image enhancement through the generative enhancement model, the texture generation capability of the generative enhancement model is brought into full play, meanwhile, the introduction of forged textures is avoided, and the finally output image has rich and credible texture details while keeping the sense of reality of the overall structure.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to an image enhancement method and device, electronic equipment, computer program product and storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning technology, image restoration technology has become a key means to improve visual quality. In practical applications, a deep image restoration model can effectively restore image degradation problems caused by compression, transmission, occlusion, etc. It shows important value in the fields of video streaming media, digital entertainment, security monitoring, etc. In the field of online streaming media, users have increasingly high requirements for image clarity, detail richness and playback stability. The current users' requirements for video clarity, detail performance and playback smoothness continue to improve, which prompts the industry to continuously explore more efficient image enhancement solutions. Image restoration technology not only helps to improve video playback quality, but also significantly improves the viewing experience in low code rate and weak network environments. However, traditional single-model restoration methods often have difficulty in balancing texture detail restoration and image fidelity, resulting in limited restoration effect. SUMMARY

[0003] The present disclosure provides an image enhancement method, device, electronic equipment, computer program product and storage medium.

[0004] According to a first aspect of some example embodiments of the present disclosure, an image enhancement method is provided, including: obtaining a to-be-enhanced image; performing denoising processing on the to-be-enhanced image through at least two denoising models respectively, to obtain at least two denoised images respectively corresponding to each of the denoising models; performing image fusion on the at least two denoised images to obtain a target denoised image; and performing image enhancement processing on the target denoised image through a generative enhancement model to obtain a target enhanced image.

[0005] In some example embodiments of the present disclosure, the performing denoising processing on the to-be-enhanced image through at least two denoising models respectively, to obtain at least two denoised images respectively corresponding to each of the denoising models, includes: performing denoising processing on the to-be-enhanced image through a local denoising model to obtain a local denoised image; and performing denoising processing on the to-be-enhanced image through a global denoising model to obtain a global denoised image.

[0006] In some example embodiments of the present disclosure, the local denoising model adopts a convolutional neural network; and the global denoising model adopts a neural network based on a self-attention mechanism.

[0007] In some example embodiments of the present disclosure, the image fusion of the at least two noise-reduced images to obtain a target noise-reduced image comprises: obtaining a difference image corresponding to each noise-reduction model based on a comparison between each of the noise-reduced images and the image to be enhanced; each pixel in the difference image is used to represent a difference value between a corresponding pixel in the noise-reduced image and the image to be enhanced; determining a target pixel value of the corresponding pixel by comparing the difference values of the corresponding pixels in each of the difference images; and obtaining the target enhanced image according to the target pixel value of each pixel.

[0008] In some example embodiments of the present disclosure, the determination of the target pixel value of the corresponding pixel by comparing the difference values of the corresponding pixels in each of the difference images comprises: determining a window difference value by sliding a preset sliding window in each of the difference images; the window difference value is a sum of the difference values of the pixels in the corresponding region of the sliding window; determining a target noise-reduced image of the corresponding region of the sliding window by comparing the window difference values of each of the difference images; and determining the target pixel value based on the pixels in the corresponding region of the sliding window in the target noise-reduced image.

[0009] In some example embodiments of the present disclosure, the determination of the target pixel value of the corresponding pixel by comparing the difference values of the corresponding pixels in each of the difference images comprises: determining a fusion weight of the corresponding pixel in each of the difference images according to the difference value of the corresponding pixel in each of the difference images; obtaining a pixel value of the corresponding pixel in each of the difference images; and determining the target pixel value of the corresponding pixel according to the pixel value and the fusion weight of the corresponding pixel in each of the difference images.

[0010] In some example embodiments of the present disclosure, the determination of the target pixel value of the corresponding pixel by comparing the difference values of the corresponding pixels in each of the difference images comprises: determining a target noise-reduced image of the corresponding pixel by comparing the difference values of the corresponding pixels in each of the difference images; the target noise-reduced image is the noise-reduced image corresponding to the minimum difference value; and determining the target pixel value based on the corresponding pixel in the target noise-reduced image.

[0011] In some example embodiments of the present disclosure, the image enhancement processing of the target noise-reduced image by the generative enhancement model to obtain a target enhanced image comprises: obtaining an image description text corresponding to the image to be enhanced; and the generative enhancement model performs image enhancement processing on the target noise-reduced image according to the image description text to obtain the target enhanced image.

[0012] According to a second aspect of the embodiments of the present disclosure, an image enhancement device is provided, comprising: an image acquisition module configured to acquire an image to be enhanced; an image denoising module configured to perform denoising processing on the image to be enhanced by at least two denoising models respectively, to obtain at least two denoised images respectively corresponding to each of the denoising models; an image fusion module configured to perform image fusion on the at least two denoised images, to obtain a target denoised image; and an image enhancement module configured to perform image enhancement processing on the target denoised image by a generative enhancement model, to obtain a target enhanced image.

[0013] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement any of the image enhancement methods.

[0014] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the image enhancement methods.

[0015] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising a computer program, which is executed by a processor to perform any of the image enhancement methods.

[0016] The image enhancement method provided by the embodiments of the present disclosure performs denoising processing on the image to be enhanced by at least two denoising models respectively, to obtain at least two denoised images; performs image fusion on the at least two denoised images, to obtain a target denoised image; and performs image enhancement processing on the target denoised image by a generative enhancement model, to obtain a target enhanced image. The method takes advantage of the complementary characteristics of the multiple denoising models in fidelity and noise resistance, to achieve high-quality denoising processing on the image to be enhanced. Then, the generative enhancement model is used to perform image enhancement on the target denoised image, which fully utilizes the texture generation capability of the generative enhancement model while avoiding introducing fake textures. Finally, the output image retains the overall structural realism while having rich and reliable texture details.

[0017] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure. It is apparent that the accompanying drawings are only some embodiments of the present disclosure, and other drawings can be obtained by those of ordinary skill in the art without creative effort based on the accompanying drawings.

[0019] Figure 1 A schematic diagram showing an exemplary system architecture to which the method of the embodiments of the present disclosure can be applied.

[0020] Figure 2 A flow chart of an image enhancement method according to an exemplary embodiment.

[0021] Figure 3 A schematic diagram of an image enhancement process according to an example.

[0022] Figure 4 A flow chart of a denoising image fusion method according to an example.

[0023] Figure 5 A schematic diagram of a denoising image fusion process according to an example.

[0024] Figure 6 A flow chart of a method for determining a target pixel value according to an exemplary embodiment. Figure 1 .

[0025] Figure 7 A flow chart of a method for determining a target pixel value according to an exemplary embodiment. Figure 2 .

[0026] Figure 8 A block diagram of an image enhancement apparatus according to an exemplary embodiment.

[0027] Figure 9 A structural schematic diagram of an electronic device suitable for implementing the exemplary embodiments of the present disclosure according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the description.

[0029] The features, structures or characteristics described in the present disclosure can be incorporated in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the techniques of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0030] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain principles of the present disclosure. In the drawings:

[0031] The flowcharts shown in the accompanying drawings are only illustrative and do not necessarily include all contents and steps, nor are they necessarily executed in the order described. For example, some steps can be further divided, and some steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0032] In the present specification, the terms "one", "an", "the", "said", and "at least one" are used to indicate that at least one element / component / etc. exists; the terms "comprise", "include" and "have" are used to indicate an open-ended inclusion in such a way that additional elements / components / etc. can be included; the terms "first", "second" and "third" etc. are used only as labels, not as a numerical limitation.

[0033] Figure 1 A schematic diagram showing an exemplary system architecture to which the method of the embodiments of the present disclosure can be applied is shown.

[0034] As Figure 1 shown, the system architecture can include a server 101, a network 102, a terminal device 103, a terminal device 104, and a terminal device 105. The network 102 is used as a medium to provide a communication link between the terminal device 103, the terminal device 104, or the terminal device 105 and the server 101. The network 102 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0035] The server 101 can be a server that provides various services, such as a background management server that provides support for operations performed by a user using the terminal device 103, the terminal device 104, or the terminal device 105. The background management server can analyze and process received request data and the like, and feed back the processing result to the terminal device 103, the terminal device 104, or the terminal device 105.

[0036] The terminal device 103, the terminal device 104, and the terminal device 105 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a wearable smart device, a virtual reality device, an augmented reality device, or the like, but are not limited thereto.

[0037] It should be understood that Figure 1 The number of the terminal device 103, the terminal device 104, the terminal device 105, the network 102, and the server 101 in the above

[0038] In the following, the various steps of the method according to the present disclosure will be described in more detail with reference to the accompanying drawings and embodiments.

[0039] Figure 2 is a flowchart of an image enhancement method according to an example embodiment. Figure 2 The method provided by the embodiments can be executed by any electronic device, such as the terminal device in the above Figure 1 or the server in the above Figure 1 or the terminal device and the server in the above Figure 1 but the present disclosure does not limit this.

[0040] In step S210, an image to be enhanced is obtained.

[0041] With the rapid development of artificial intelligence and deep learning technology, image inpainting has become an important means to improve image quality and visual experience. In practical applications, a deep image inpainting model can effectively solve the problem of image degradation caused by compression, transmission, occlusion, and the like. In the field of streaming media, image inpainting technology not only helps to improve video playback quality, but also significantly improves the viewing experience in low-code rate and weak network environments, and has become one of the key factors to improve the core competitiveness of the platform.

[0042] In the embodiments of the present disclosure, an image to be enhanced is obtained. The image to be enhanced can be image data that needs to be inpainted, or any image frame in streaming media data that needs to be inpainted.

[0043] In step S220, the to-be-enhanced image is denoised by at least two denoising models respectively to obtain at least two denoised images respectively corresponding to the denoising models.

[0044] In the related art, the main challenge of the image enhancement task is how to achieve the balance between texture detail generation and image fidelity. The current mainstream method has obvious performance limitations: on the one hand, a model with strong texture generation capability is easy to produce unrealistic pseudo-texture, and at the same time, it will amplify the noise and compression artifacts in the input image; on the other hand, a model emphasizing fidelity can maintain the authenticity of the image structure, but often cannot effectively restore rich texture details, resulting in unsatisfactory enhancement effect. Based on this, the present disclosure divides the image enhancement process into two stages, first removes the noise in the image through a denoising model to improve the fidelity of the image, and then enhances the image after denoising through an image enhancement model. Through the above two stages of image enhancement processing, the richness of the image details is improved without introducing obvious pseudo-texture, and the structural distortion caused by image compression damage and noise is suppressed, achieving the balance between fidelity and perceptual quality in the image enhancement process.

[0045] In the embodiments of the present disclosure, the denoising models have different denoising effects due to different denoising methods. For example, some denoising models perform denoising based on surrounding image information, have strong local inductive bias, but may cause loss of texture details. Some denoising models focus on the extraction of image context information and have global perception ability, but the local denoising effect is poor. Based on this, if only a single model is used for denoising processing, it is difficult to balance fidelity and noise resistance.

[0046] In the embodiments of the present disclosure, at least two denoising models are introduced in the image denoising stage. The to-be-enhanced image is denoised by the at least two denoising models respectively, so as to obtain denoised images respectively corresponding to the denoising models. The denoised images have complementary characteristics in fidelity and noise resistance based on different denoising methods. Through fusion processing of the denoised images, the advantages of each other can be combined to achieve high-quality processing of the preliminary enhancement result.

[0047] In the example embodiments, the denoising model can use a convolutional neural network (CNN). The image enhancement model based on the CNN architecture can effectively capture the local structure and spatial correlation in the image through its unique convolution operation and hierarchical feature extraction mechanism, and is particularly good at identifying and removing structural distortions such as compression noise and blocking artifacts.

[0048] In an example embodiment, the noise reduction model can employ a neural network based on a self-attention mechanism (Transformer). An image enhancement model based on a Transformer architecture realizes modeling of global context information of an image through a self-attention mechanism. The model uses a multi-head attention layer to establish long-range dependencies between pixels and can simultaneously consider the relevance of all regions of an image, and performs well in processing large-scale structural distortion. This global perception feature makes it perform well in maintaining the structural authenticity of an image, and is particularly suitable for processing content consistency repair in complex scenes.

[0049] In step S230, image fusion is performed on the at least two noise reduction images to obtain a target noise reduction image.

[0050] In the embodiments of the present disclosure, the obtained multiple noise reduction images are subjected to image fusion to obtain a target noise reduction image. Through image fusion, the advantages of each of the multiple noise reduction models are combined to improve the image enhancement quality of the target noise reduction image output in the noise reduction stage.

[0051] It should be noted that there are many ways of image fusion of multiple noise reduction images in combination with the characteristics of each of the multiple noise reduction models, and the present disclosure does not limit the specific image fusion algorithm. Any fusion method that combines the advantages of each noise reduction image through image fusion processing should be considered within the protection scope of the present disclosure.

[0052] In step S240, the target noise reduction image is subjected to image enhancement processing through a generative enhancement model to obtain a target enhanced image.

[0053] In the embodiments of the present disclosure, the target denoised image subjected to the denoising processing is subjected to image enhancement processing by a generative enhancement model to obtain a target enhanced image. The target enhanced image is the target image subjected to image enhancement processing on the to-be-enhanced image. The generative enhancement model is configured to learn data features of high-quality images, repair and generate details of low-quality inputs, and output enhanced high-quality images. The generative enhancement model can be constructed based on a generative adversarial network (GAN) or a diffusion model (Diffusion Model). Since the generative enhancement model not only processes the low-quality image based on the original input but also combines data features of a large number of learned high-quality images, it can generate more abundant texture details (such as hair, skin details, etc.). Although the generative enhancement model has strong texture generation capability, it is sensitive to noise of the input image and is prone to introduce fake textures or amplify noise and compression damage in the image. The present disclosure is directed to the characteristics of the generative enhancement model and introduces multiple denoising models in front of the generative enhancement model to denoise the to-be-enhanced image, and adaptively fuses the output images of the denoising models to fully improve the fidelity and noise resistance of the target denoised image. The generative enhancement model generates a generative image enhancement processing based on the target denoised image, and can generate a cleaner and clearer image.

[0054] In an example embodiment, the generative enhancement model can adopt a text-to-image diffusion model. The model uses a diffusion model to generate highly natural high-frequency texture details in a multi-step iterative manner, and is particularly good at completing missing complex texture structures. At the same time, the model has stronger semantic understanding capability and can generate reasonable details according to the image content and related description text.

[0055] In an example embodiment, in view of the characteristics of the text-to-image diffusion model, the step S240 can further include the following steps:

[0056] An image description text corresponding to the to-be-enhanced image is obtained.

[0057] The generative enhancement model performs image enhancement processing on the target denoised image according to the image description text to obtain the target enhanced image.

[0058] Since the text-to-image model has the ability to enhance images based on semantic understanding combined with image description text, the image description text corresponding to the image to be enhanced can be further introduced in the above steps. The generative enhancement model performs image enhancement processing on the target denoising image according to the image description text, thereby obtaining a target enhanced image. In this way, the semantic understanding ability of the text-to-image model can be further utilized to better perform image enhancement processing. The image description text can be a relevant image description text obtained with the image to be enhanced, or an image description text generated based on the image to be enhanced by other models.

[0059] The image enhancement method provided by the embodiments of the present disclosure can perform denoising processing on the image to be enhanced by at least two denoising models to obtain at least two denoising images, perform image fusion on the at least two denoising images to obtain a target denoising image, and perform image enhancement processing on the target denoising image by a generative enhancement model to obtain a target enhanced image. The method can utilize the complementary characteristics of the multiple denoising models in fidelity and noise resistance to achieve high-quality denoising processing on the image to be enhanced. Then, the generative enhancement model is used to perform image enhancement on the target denoising image, which can fully utilize the texture generation capability of the generative enhancement model while avoiding the introduction of fake textures. Finally, the output image can retain the overall structural realism while having rich and credible texture details.

[0060] Figure 3 is a schematic diagram of an image enhancement processing process according to an example. The image enhancement processing process of the foregoing step S220 can include the following steps.

[0061] performing denoising processing on the image to be enhanced by a local denoising model to obtain a local denoising image;

[0062] performing denoising processing on the image to be enhanced by a global denoising model to obtain a global denoising image.

[0063] In the embodiments of the present disclosure, as shown in Figure 3 In the image enhancement processing process, there are a local denoising model, a global denoising model, an image fusion module, and a generative enhancement model. In the image denoising stage, the local denoising model and the global denoising model perform denoising processing on the image to be enhanced, respectively.

[0064] In the embodiments of the present disclosure, the local denoising model is used to perform denoising processing on the image to be enhanced to obtain a local denoising image. The local denoising model can be a convolutional neural network (CNN). The model can capture local structures and spatial correlations in the image by utilizing the characteristics of local perception and weight sharing, thereby obtaining a local denoising image that focuses on local denoising of the image.

[0065] In this embodiment of the disclosure, a global denoising model is used to denoise the image to be enhanced, resulting in a globally denoised image. This global denoising model can employ a self-attention-based neural network (Transformer). By utilizing the self-attention mechanism of the Transformer architecture, it models the global contextual information of the image, thereby obtaining a globally denoised image that emphasizes the consistency of the global content of the image.

[0066] The image enhancement method provided in this disclosure combines the advantages of different denoising models by employing local denoising models for local denoising and global denoising models for global denoising. This approach preserves both noise suppression and more realistic information, improving the fidelity of the denoised image. Compared to traditional denoising processes, it achieves higher-quality denoised images, providing a data foundation for subsequent image enhancement.

[0067] Figure 4 This is a flowchart illustrating a noise reduction image fusion method, based on an example. Figure 5 This is a schematic diagram illustrating a noise reduction image fusion process, as shown in the example. Figure 4 , 5 As shown in this embodiment of the present disclosure, the noise reduction image fusion process in step S230 may include the following steps.

[0068] In step S410, based on the comparison between each of the denoised images and the image to be enhanced, difference images corresponding to each of the denoising models are obtained; each pixel in the difference image is used to characterize the difference between corresponding pixels between the corresponding denoised image and the image to be enhanced.

[0069] In this embodiment, the main objective of the image denoising stage is to maximize image fidelity while removing noise from the image to be enhanced. To this end, the denoised images obtained from each denoising model are compared with the image to be enhanced, resulting in difference images corresponding to each denoising model. Each pixel in the difference image represents the difference between the corresponding pixel in the denoised image and the image to be enhanced. If the difference is small, it indicates that the difference between the denoised image and the image to be enhanced for that pixel is small, meaning high fidelity. If the difference is large, it indicates that the difference between the denoised image and the image to be enhanced for that pixel is large, meaning low fidelity.

[0070] like Figure 5As shown, the to-be-enhanced image 500 is subjected to noise reduction processing by the noise reduction model 1 to obtain a noise reduction image 510. The to-be-enhanced image 500 is subjected to noise reduction processing by the noise reduction model 2 to obtain a noise reduction image 520. The noise reduction image 510 is compared with the to-be-enhanced image 500 to obtain a corresponding difference image 511. The difference image 511 is used to represent the difference between the noise reduction image 510 and the to-be-enhanced image 500 at each pixel. The noise reduction image 510 is compared with the to-be-enhanced image 500 to obtain a corresponding difference image 521. The difference image 521 is used to represent the difference between the noise reduction image 520 and the to-be-enhanced image 500 at each pixel.

[0071] It should be noted that the above only takes two noise reduction models as an example for illustration. If multiple noise reduction models are introduced for noise reduction processing, the related processing is similar and will not be repeated.

[0072] In step S420, the target pixel value of the corresponding pixel is determined by comparing the difference values of the corresponding pixels of each difference image.

[0073] In the embodiments of the present disclosure, the fidelity between each noise reduction image and the to-be-enhanced image is determined by comparing the difference values of the corresponding pixels between each difference image, and then the pixel value of the corresponding pixel in the final target enhanced image is determined. It should be noted that there are many ways to compare the difference values between related difference images, and many ways to determine the target pixel value of the corresponding pixel based on the comparison results, and the present disclosure does not limit the related embodiments.

[0074] In the exemplary embodiments, the difference values of the corresponding pixels between the difference image 511 and the difference image 521 are compared. The noise reduction image corresponding to the minimum difference value is taken as the target noise reduction image of the pixel. For example, for the pixel point (a1, b1), the difference value corresponding to the difference image 511 is X1, and the difference value corresponding to the difference image 521 is Y1. Wherein, X1 < Y1, it is determined that the target noise reduction image of the pixel point (a1, b1) is the noise reduction image 510. For the pixel point (a2, b2), the difference value corresponding to the difference image 511 is X2, and the difference value corresponding to the difference image 521 is Y2. Wherein, X2 > Y2, it is determined that the target noise reduction image of the pixel point (a2, b2) is the noise reduction image 520. Based on this, by traversing each pixel in the to-be-enhanced image, the target pixel value of the pixel is determined based on the corresponding pixel in the target noise reduction image corresponding to each pixel. As described above, the target pixel value corresponding to the pixel point (a1, b1) in the target enhanced image is the pixel value of the pixel point (a1, b1) in the noise reduction image 510; the target pixel value corresponding to the pixel point (a2, b2) in the target enhanced image is the pixel value of the pixel point (a2, b2) in the noise reduction image 520.

[0075] In step S430, the target enhancement image is obtained according to the target pixel value of each pixel.

[0076] In the embodiment of the present disclosure, the target pixel value corresponding to each pixel in the image to be enhanced is obtained through the above step S420, and then the target enhancement image is determined according to the target pixel value of each pixel.

[0077] In the exemplary embodiment, a mask image corresponding to the noise reduction image 510 and the noise reduction image 520 can be generated based on the comparison of the difference values of the corresponding pixels between the difference image 511 and the difference image 521, and then image fusion is performed based on the related mask image to obtain the target enhancement image 530.

[0078] Figure 6 The flow of the method for determining the target pixel value is shown according to an exemplary embodiment Figure 1 As shown in Figure 6 , in the embodiment of the present disclosure, the foregoing step S420 can include the following steps.

[0079] In step S610, a preset sliding window is used to slide search in each of the difference images to determine a corresponding window difference value; the window difference value is the sum of the difference values of each pixel in the region corresponding to the sliding window.

[0080] In the embodiment of the present disclosure, since the difference values are compared pixel by pixel, it is possible that the picture of the target enhancement image has a local non-smooth problem due to the influence of individual pixel difference values. Therefore, in the present embodiment, the pixel-by-pixel comparison method is changed to a sliding search method based on a sliding window. A preset sliding window is used to slide search in each of the difference images. The size of the sliding window is preset, for example, a 2X2, 3X3 or 5X5 sliding window is used. The sliding window is used to slide search in each of the difference images to determine a corresponding window difference value. The window difference value is the sum of the difference values of each pixel in the region corresponding to the sliding window.

[0081] For example, taking a 3X3 sliding window as an example, the difference values corresponding to nine pixel points (a, b), (a, b+1), (a, b+2), (a+1, b), (a+1, b+1), (a+1, b+2), (a+2, b), (a+2, b+1), (a+2, b+2) of each difference image are obtained based on the sliding window. The window difference value corresponding to the difference image is calculated according to the difference values corresponding to the nine pixel points.

[0082] In step S620, the target noise reduction image of the region corresponding to the sliding window is determined by comparing the window difference values corresponding to each of the difference images.

[0083] In the embodiments of the present disclosure, the target denoising image of the region corresponding to the sliding window is determined by comparing the window difference values corresponding to the difference images.

[0084] In the example embodiments, the window difference values corresponding to the sliding window between the difference image 511 and the difference image 521 are compared. The window difference value corresponding to the difference image 511 is X3, and the window difference value corresponding to the difference image 521 is Y3. Wherein, X3 < Y3, and the target denoising image of the region corresponding to the sliding window is determined as the denoising image 510.

[0085] In step S630, the target pixel value is determined based on the pixels in the region corresponding to the sliding window in the target denoising image.

[0086] In the embodiments of the present disclosure, the target pixel value corresponding to each pixel in the region corresponding to the sliding window is obtained through the above step S520, and then the target enhanced image is determined according to the target pixel value of each pixel.

[0087] In the example embodiments, the mask image corresponding to the denoising image 510 and the denoising image 520 can be generated based on the comparison of the window difference values corresponding to the sliding window between the difference image 511 and the difference image 521, and then the image fusion is performed based on the related mask image to obtain the target enhanced image 530.

[0088] Figure 7 The flow of the method for determining the target pixel value is shown according to an example embodiment Figure 2 . As shown in Figure 7 , in the embodiments of the present disclosure, the foregoing step S420 can include the following steps.

[0089] In step S710, the fusion weight of each pixel corresponding to each difference image is determined according to the difference value of each pixel corresponding to each difference image.

[0090] In the embodiments of the present disclosure, instead of directly determining one of the denoising images as the target denoising image based on the difference value between each difference image, the fusion weight corresponding to the corresponding pixel in each denoising image is determined based on the difference value of the pixel corresponding to each denoising image, and then the image fusion is performed based on the fusion weight.

[0091] In the example embodiments, for the pixel point (a3, b3), the difference value corresponding to the difference image 511 is X4, and the difference value corresponding to the difference image 521 is Y4. According to the difference values X4 and Y4, the corresponding fusion weight is determined. For example, the fusion weight corresponding to the difference image 511 can be determined as Y4 / (X4+Y4), and the fusion weight corresponding to the difference image 521 can be determined as X4 / (X4+Y4).

[0092] In step S720, pixel values of pixels corresponding to each of the difference images are obtained.

[0093] In step S730, the target pixel value of a corresponding pixel is determined according to the pixel value of the corresponding pixel of each of the difference images and the fusion weight.

[0094] In the embodiments of the present disclosure, the target pixel value of the corresponding pixel is determined by weighted calculation according to the pixel value corresponding to each of the difference images and the determined corresponding fusion weight. The target pixel value is the pixel value of the corresponding pixel in the target denoising image.

[0095] The image enhancement method provided by the embodiments of the present disclosure provides a plurality of different denoising image fusion methods. The window difference of each difference image can be searched by sliding the sliding window, so as to solve the problem of local non-smoothness of the enhanced picture. Each pixel in the denoising image can be weighted by comparing the corresponding pixel difference, so as to realize image fusion between each denoising image. Different image fusion methods can meet the needs of different application scenarios.

[0096] The following is an apparatus embodiment of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0097] Figure 8 is a block diagram of an image enhancement apparatus according to an exemplary embodiment. Referring to Figure 8 The apparatus 800 can include an image acquisition module 810, an image denoising module 820, an image fusion module 830, and an image enhancement module 840.

[0098] The image acquisition module 810 is configured to acquire a to-be-enhanced image.

[0099] The image denoising module 820 is configured to perform denoising processing on the to-be-enhanced image by at least two denoising models, to obtain at least two denoising images respectively corresponding to each of the denoising models.

[0100] The image fusion module 830 is configured to perform image fusion on the at least two denoising images, to obtain a target denoising image.

[0101] The image enhancement module 840 is configured to perform image enhancement processing on the target denoising image by a generative enhancement model, to obtain a target enhanced image.

[0102] In some exemplary embodiments of the present disclosure, the image denoising module 820 is further configured to perform denoising processing on the to-be-enhanced image by a local denoising model, to obtain a local denoising image; and perform denoising processing on the to-be-enhanced image by a global denoising model, to obtain a global denoising image.

[0103] In some example embodiments of the present disclosure, the local noise reduction model adopts a convolutional neural network; and the global noise reduction model adopts a neural network based on a self-attention mechanism.

[0104] In some example embodiments of the present disclosure, the image fusion module 830 is further configured to obtain a difference image corresponding to each of the noise reduction models based on a comparison between each of the noise reduction images and the image to be enhanced; each pixel in the difference image is used to represent a difference value between a corresponding pixel in the noise reduction image and the image to be enhanced; a target pixel value of the corresponding pixel is determined by comparing the difference values of the corresponding pixels in each of the difference images; and the target enhanced image is obtained according to the target pixel value of each pixel.

[0105] In some example embodiments of the present disclosure, the image fusion module 830 is further configured to determine a corresponding window difference value by performing a sliding search on each of the difference images through a preset sliding window; the window difference value is a sum of the difference values of the pixels in the corresponding region of the sliding window; a target noise reduction image of the corresponding region of the sliding window is determined by comparing the corresponding window difference values of each of the difference images; and the target pixel value is determined based on the pixels in the corresponding region of the sliding window in the target noise reduction image.

[0106] In some example embodiments of the present disclosure, the image fusion module 830 is further configured to determine a fusion weight of each pixel in each of the difference images according to the difference value of the corresponding pixel in each of the difference images; obtain a pixel value of each pixel in each of the difference images; and determine the target pixel value of the corresponding pixel according to the pixel value and the fusion weight of each pixel in each of the difference images.

[0107] In some example embodiments of the present disclosure, the image fusion module 830 is further configured to determine a target noise reduction image of the corresponding pixel by comparing the difference values of the corresponding pixels in each of the difference images; the target noise reduction image is the noise reduction image corresponding to the minimum difference value; and the target pixel value is determined based on the corresponding pixel in the target noise reduction image.

[0108] In some example embodiments of the present disclosure, the image enhancement module 840 is further configured to obtain an image description text corresponding to the image to be enhanced; and the generative enhancement model performs image enhancement processing on the target noise reduction image according to the image description text to obtain the target enhanced image.

[0109] As to the apparatus in the above-mentioned embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0110] The electronic device 900 according to this embodiment of the present disclosure will be described below with reference to Figure 9 FIG. 9. Figure 9 The electronic device 900 shown is merely an example and should not limit the scope of functionality or use of embodiments of the present disclosure.

[0111] As Figure 9 shown, the electronic device 900 is in the form of a general computing device. Components of the electronic device 900 can include, but are not limited to, the at least one processing unit 910 described above, the at least one storage unit 920 described above, a bus 930 that connects different system components, including the storage unit 920 and the processing unit 910, a display unit 940.

[0112] The storage unit stores program code that can be executed by the processing unit 910, so that the processing unit 910 performs the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of the present disclosure. For example, the processing unit 910 can perform various steps as shown in Figure 2

[0113] For another example, the electronic device can implement various steps as shown in Figure 2

[0114] The storage unit 920 can include a readable medium in the form of volatile storage such as a random access memory (RAM) 921 and / or cache memory 922, and can further include a read-only memory (ROM) 923.

[0115] The storage unit 920 can further include program / utility 924 having a set of the program modules 925, including but not limited to, an operating system, one or more application programs, other program modules, and program data, and each of these examples or some combination thereof, can include implementation of a network environment.

[0116] The bus 930 can be representative of one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit bus, or a local bus using any of a variety of bus architectures.

[0117] ​​The electronic device 900 can also communicate with one or more external devices 970 such as a keyboard or pointing device, a Bluetooth device, or a device for reading media. Communication with one or more devices can enable a user to interact with the electronic device 900 in order to use it or perform methods described herein. In some embodiments, the communication can be facilitated by an I / O interface 950. The electronic device 900 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the Internet, through a network adapter 960. As depicted, the network adapter 960 can communicate with the other components of the electronic device 900 through the bus 930. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with the electronic device 900. These components, as well as the software components of the electronic device 900, are meant to be illustrative only and the scope of the application is not limited to any particular software configuration or hardware configuration.

[0118] From the above description of the embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the disclosure.

[0119] In the example embodiments, a computer readable storage medium including instructions, such as a memory including instructions, is also provided, and the instructions can be executed by a processor of a device to complete the above method. Alternatively, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0120] In the example embodiments, a computer program product including a computer program is also provided, and the computer program is executed by a processor to implement the method in the above embodiments.

[0121] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0122] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image enhancement method characterized by, The method comprises: obtaining an image to be enhanced; performing noise reduction processing on the image to be enhanced by at least two noise reduction models respectively to obtain at least two noise reduction images respectively corresponding to each of the noise reduction models; performing image fusion on the at least two noise reduction images to obtain a target noise reduction image; performing image enhancement processing on the target noise reduction image by using a generative enhancement model to obtain a target enhanced image.

2. The method of claim 1, wherein, The method comprises: performing noise reduction processing on the image to be enhanced by using a local noise reduction model to obtain a local noise reduction image; performing noise reduction processing on the image to be enhanced by using a global noise reduction model to obtain a global noise reduction image.

3. The method of claim 2, wherein, The method comprises: The local noise reduction model uses a convolutional neural network. The global noise reduction model uses a neural network based on a self-attention mechanism.

4. The method of claim 1, wherein, The method comprises: comparing each of the noise reduction images with the image to be enhanced to obtain difference images respectively corresponding to each of the noise reduction models; each pixel in the difference image is used to represent the difference between the corresponding pixels of the noise reduction image and the image to be enhanced; determining a target pixel value of the corresponding pixel by comparing the difference values of the corresponding pixels of each of the difference images; obtaining the target enhanced image according to the target pixel value of each pixel.

5. The method of claim 4, wherein, The method comprises: determining a corresponding window difference value by sliding a preset sliding window in each of the difference images; the window difference value is the sum of the difference values of the pixels in the corresponding region of the sliding window; determining a target noise reduction image of the corresponding region of the sliding window by comparing the window difference values of each of the difference images; determining the target pixel value based on the pixels in the corresponding region of the sliding window in the target noise reduction image.

6. The method of claim 4, wherein, The method comprises: determining a fusion weight of the corresponding pixel of each of the difference images according to the difference value of the corresponding pixel of each of the difference images; obtaining the pixel value of the corresponding pixel of each of the difference images; determining the target pixel value of the corresponding pixel according to the pixel value and the fusion weight of the corresponding pixel of each of the difference images.

7. The method of claim 4, wherein, The method comprises: determining a target noise reduction image of the corresponding pixel by comparing the difference values of the corresponding pixels of each of the difference images; the target noise reduction image is the noise reduction image corresponding to the minimum difference value; determining the target pixel value based on the corresponding pixel in the target noise reduction image.

8. The method of claim 1, wherein, The method comprises: obtaining an image description text corresponding to the image to be enhanced; The generative enhancement model performs image enhancement processing on the target denoised image according to the image description text, to obtain the target enhanced image.

9. An image enhancement device, characterized by The method comprises: an image acquisition module configured to acquire a to-be-enhanced image; an image denoising module configured to perform denoising processing on the to-be-enhanced image by at least two denoising models respectively, to obtain at least two denoised images respectively corresponding to each of the denoising models; an image fusion module configured to perform image fusion on the at least two denoised images, to obtain a target denoised image; an image enhancement module configured to perform image enhancement processing on the target denoised image by a generative enhancement model, to obtain a target enhanced image.

10. An electronic device, comprising: The method comprises: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the image enhancement method according to any one of claims 1 to 8. 11.A computer readable storage medium, when instructions in the computer readable storage medium are executed by a processor of an electronic device, enabling the electronic device to perform the image enhancement method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the image enhancement method according to any one of claims 1 to 8.

Images