Image enhancement method and device, electronic equipment and storage medium

Abstract

The embodiment of the present application discloses a kind of image enhancement method and device, the method includes: obtaining image to be enhanced, and the illumination enhancement sequence of image to be enhanced is obtained by image enhancement algorithm learning, the illumination enhancement sequence includes continuous multiple illumination enhancement components;The denoising of image to be enhanced is carried out, and the first denoising image is obtained, and the first illumination enhancement component in illumination enhancement sequence is used, and the image enhancement of first denoising image is carried out, and the first enhanced image is obtained;The denoising of first enhanced image is carried out, and the second denoising image is obtained, and the second illumination enhancement component in illumination enhancement sequence is used, and the image enhancement of second denoising image is carried out, and the second enhanced image is obtained;Until all illumination enhancement components in illumination enhancement sequence complete the image enhancement about denoising image, and low light enhanced image is obtained.The present application solves the problem of related art in low light enhancement, simultaneously enhanced noise leads to the problem of poor image enhancement effect.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to an image enhancement method, apparatus, electronic device, and storage medium. Background Technology

[0002] Due to unavoidable environmental or technical limitations (such as insufficient lighting and limited exposure time), the aesthetic quality of photos taken under backlight, uneven lighting, and dim lighting is severely compromised. Therefore, in real shooting conditions, noise always occurs simultaneously with low light. Image acquisition in low-light environments will result in decreased image quality and information loss.

[0003] In recent years, low-light image enhancement has been widely applied in various fields, including visual surveillance, autonomous driving, and computational photography. In particular, smartphone photography has become ubiquitous and prominent. Taking photos with a smartphone camera in dimly lit environments is especially challenging due to limitations in camera aperture, real-time processing requirements, and memory constraints. In these applications, enhancing low-light images is an interesting and practical area of ​​research.

[0004] However, while these technologies enhance the brightness of low-light images, they inevitably amplify noise as well, resulting in poor image enhancement effects.

[0005] Therefore, there is an urgent need for an image enhancement method that can simultaneously achieve low-light enhancement and noise reduction. Summary of the Invention

[0006] The embodiments of the present invention provide an image enhancement method, apparatus, electronic device, and storage medium to solve the problem in the related art where the image enhancement effect is poor due to the simultaneous enhancement of noise during low-light enhancement.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] According to one aspect of the present invention, an image enhancement method is provided, the method comprising: acquiring an image to be enhanced, and learning an illumination enhancement sequence of the image to be enhanced through an image enhancement algorithm, the illumination enhancement sequence comprising a plurality of consecutive illumination enhancement components; denoising the image to be enhanced to obtain a first denoised image, and enhancing the first denoised image using a first illumination enhancement component in the illumination enhancement sequence to obtain a first enhanced image; denoising the first enhanced image to obtain a second denoised image, and enhancing the second denoised image using a second illumination enhancement component in the illumination enhancement sequence to obtain a second enhanced image; until all illumination enhancement components in the illumination enhancement sequence have completed image enhancement of the denoised image, a low-light enhanced image is obtained.

[0009] In an exemplary embodiment, the illumination enhancement sequence of the image to be enhanced is learned by an image enhancement algorithm, including performing vector mapping on the image to be enhanced to obtain a feature map of the image to be enhanced, and converting the feature map into a token; learning the illumination components in the image to be enhanced based on the token to obtain the illumination enhancement sequence of the image to be enhanced.

[0010] In one exemplary embodiment, converting the feature map into a token includes performing multiple vector convolution operations on the feature map to obtain a convolution result; and flattening the convolution result to obtain the token.

[0011] In one exemplary embodiment, the low-light enhanced image is obtained through an enhancement-denoising joint network; the enhancement-denoising joint network is a trained machine learning model capable of denoising and enhancing the image to be enhanced; wherein the enhancement-denoising joint network includes a low-light enhancement module and a denoising module.

[0012] In an exemplary embodiment, the training process of the enhancement-denoising joint network includes acquiring a training set, which includes multiple training image pairs; the training image pairs include an image to be enhanced after denoising and image enhancement, and a low-light enhancement image; inputting the current training image pair in the training set into the machine learning model for training, obtaining the loss value of the low-light enhancement module and the loss value of the denoising module; obtaining a total loss value based on the loss value of the low-light enhancement module and the loss value of the denoising module; if the total loss value satisfies the convergence condition, the training is completed, and the enhancement-denoising joint network is obtained; otherwise, the model parameters of the machine learning model are updated, and the next training image pair is acquired and input into the machine learning model for continued training until the total loss value satisfies the convergence condition.

[0013] In an exemplary embodiment, the denoising module includes a downsampler; the calculation process of the loss value of the denoising module includes downsampling the image to be enhanced by the downsampler in the denoising module by the first adjacent pixel to obtain a first downsampled image pair; using one downsampled image in the first downsampled image pair as an input signal and the other downsampled image as a supervision signal; performing self-supervision on the denoising module through the input signal and the supervision signal to obtain the loss value of the denoising module; when the loss value of the denoising module satisfies the convergence condition, the denoising module that has completed training has denoising capability.

[0014] In an exemplary embodiment, the loss value of the denoising module is obtained by self-supervising the denoising module through the input signal and the supervision signal, including denoising the image to be enhanced and the input signal respectively; downsampling the denoised image to be enhanced a second time by adjacent pixels to obtain a second downsampled image pair; constructing a reconstruction loss function and a regularization loss function based on the supervision signal, the denoised input signal, and the obtained second downsampled image pair respectively; calculating the loss value of the reconstruction loss function and the loss value of the regularization loss function, and calculating the loss value of the denoising module based on the loss value of the reconstruction loss function and the loss value of the regularization loss function.

[0015] According to one aspect of the present invention, an image enhancement apparatus includes: an illumination enhancement sequence acquisition module for acquiring an image to be enhanced and learning an illumination enhancement sequence of the image to be enhanced through an image enhancement algorithm; a first enhancement module for denoising the image to be enhanced to obtain a first denoised image, and enhancing the first denoised image using a first illumination enhancement component in the illumination enhancement sequence to obtain a first enhanced image; a second enhancement module for denoising the first enhanced image to obtain a second denoised image, and enhancing the second denoised image using a second illumination enhancement component in the illumination enhancement sequence to obtain a second enhanced image; and a result acquisition module for obtaining a low-light enhanced image after all illumination enhancement components in the illumination enhancement sequence have completed image enhancement of the denoised image.

[0016] According to one aspect of the present invention, an electronic device includes a processor and a memory, wherein the memory stores computer-readable instructions that, when executed by the processor, implement the image enhancement method as described above.

[0017] According to one aspect of the present invention, a storage medium having a computer program stored thereon, which, when executed by a processor, implements the image enhancement method as described above.

[0018] According to one aspect of the present invention, a computer program product includes a computer program stored in a storage medium, a processor of a computer device reads the computer program from the storage medium, and the processor executes the computer program such that the computer device, when executed, implements the image enhancement method as described above.

[0019] In the above technical solution, the present invention solves the problem that the related technology has poor image enhancement effect due to the simultaneous enhancement of noise when enhancing low light.

[0020] Specifically, this invention first acquires the image to be enhanced and learns an illumination enhancement sequence for the image using an image enhancement algorithm. This illumination enhancement sequence includes multiple consecutive illumination enhancement components. Then, the image to be enhanced is denoised to obtain a first denoised image. The first illumination enhancement component in the illumination enhancement sequence is then used to enhance the first denoised image, resulting in a first enhanced image. This process is repeated until all illumination enhancement components in the illumination enhancement sequence have completed image enhancement for the denoised images, ultimately yielding a low-light enhanced image. In this process, this invention introduces a denoising task during image enhancement, ensuring that brightness enhancement does not simultaneously enhance noise, significantly improving the image enhancement effect.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0023] Figure 1 This is a schematic diagram of a low-quality image under real shooting conditions in the background technology;

[0024] Figure 2 This is a schematic diagram illustrating the processing results of existing related technologies on real datasets;

[0025] Figure 3 This is a schematic diagram based on the implementation environment involved in this application;

[0026] Figure 4 This is a flowchart illustrating an image enhancement method according to an exemplary embodiment;

[0027] Figure 5 yes Figure 4 A schematic diagram corresponding to the embodiment;

[0028] Figure 6 yes Figure 4 A schematic diagram of the low-light enhancement module STAR in the corresponding embodiment;

[0029] Figure 7 yes Figure 4 A schematic diagram of the process for obtaining a token based on a feature map in a corresponding embodiment;

[0030] Figure 8 This is a training schematic diagram of a noise reduction module according to an exemplary embodiment;

[0031] Figure 9 This is a comparison diagram of the effects of an exemplary embodiment in an application scenario with existing technologies;

[0032] Figure 10 This is a schematic diagram of multiple application scenarios according to an exemplary embodiment;

[0033] Figure 11 This is a block diagram illustrating an image enhancement apparatus according to an exemplary embodiment;

[0034] Figure 12 This is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0035] Figure 13 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0036] The accompanying drawings have illustrated specific embodiments of the present invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the inventive concept in any way, but rather to illustrate the concept of the invention to those skilled in the art by referring to specific embodiments. Detailed Implementation

[0037] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0038] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0039] like Figure 1As shown, due to unavoidable environmental or technical limitations (such as insufficient lighting and limited exposure time), the aesthetic quality of photographs taken under backlight, uneven lighting, and dim lighting is severely compromised, resulting in some degradation caused by suboptimal lighting conditions, such as backlight, uneven lighting, dim light, extremely low brightness, and noise.

[0040] Therefore, existing low-light enhancement algorithms can be divided into two main categories: reconstruction-based low-light enhancement algorithms and mapping-based low-light enhancement algorithms. Reconstruction-based algorithms lose detail during image reconstruction and, due to the fixed input scale, are difficult to transfer to high-resolution images. They also suffer from high computational complexity, are prone to meshing, have poor real-time performance, and a small dynamic range for brightness adjustment. Mapping-based low-light enhancement algorithms directly learn the mapping curve from the low-light image to the highlight image. This method is suitable for high-resolution low-light images, but the curve still maps noise into the highlight image during the mapping process, inevitably enhancing noise while brightening the image. Furthermore, existing low-light enhancement algorithms rely on synthetic datasets and have poor generalization ability on real data because there is a significant inter-domain difference between real and synthetic data.

[0041] Figure 2 The results of existing mapping-based low-light enhancement algorithms on real datasets are shown, and it is clear that there is a lot of noise in the low-light enhanced images.

[0042] As can be seen from the above, the relevant technologies still have the drawback that the image enhancement effect is not good when enhancing low light because noise is enhanced at the same time.

[0043] Therefore, the image enhancement method provided in this application first acquires the image to be enhanced, and then learns an illumination enhancement sequence for the image to be enhanced through an image enhancement algorithm. This illumination enhancement sequence includes multiple consecutive illumination enhancement components. Next, the image to be enhanced is denoised to obtain a first denoised image. Then, the first illumination enhancement component in the illumination enhancement sequence is used to enhance the first denoised image to obtain a first enhanced image. The first enhanced image is then denoised again to obtain a second denoised image. The second illumination enhancement component in the illumination enhancement sequence is used to enhance the second denoised image to obtain a second enhanced image. This process continues until all illumination enhancement components in the illumination enhancement sequence have completed image enhancement for the denoised image, resulting in a low-light enhanced image. The image enhancement method provided by this invention significantly improves the image enhancement effect. This image enhancement method is applicable to image enhancement devices, which can be deployed in electronic devices. These electronic devices can be computer devices configured with a von Neumann architecture, such as desktop computers, laptops, servers, etc. They can also be devices with image acquisition capabilities.

[0044] Figure 3 This is a schematic diagram of an implementation environment for an image enhancement method. The implementation environment includes a data acquisition terminal 110 and a server terminal 130.

[0045] Specifically, the acquisition terminal 110 acquires facial images. The acquisition terminal 110 can be a smartphone, tablet, laptop, desktop computer, or other electronic device with image acquisition capabilities (such as a smart camera), and there is no limitation on it here.

[0046] The data acquisition terminal 110 and the server terminal 130 can establish a communication connection through wired or wireless means to realize data transmission between them. For example, the transmitted data can be facial images, etc.

[0047] The server-side 130 can also be considered as the cloud, cloud platform, platform-side, server-side, etc. This server-side 130 can be a single server, a server cluster consisting of multiple servers, or a cloud computing center consisting of multiple servers, in order to better provide backend services to the massive data acquisition terminal 110. For example, backend services include image enhancement services.

[0048] With the interaction between the acquisition terminal 110 and the server terminal 130, in one application scenario, taking the image enhancement service provided by the server terminal 130 as an example, after the acquisition terminal 110 acquires the image to be enhanced, it sends the image to be enhanced to the server terminal 130. Then, the server terminal 130 can receive the image to be enhanced sent by the acquisition terminal 110 and provide image enhancement services based on the image to be enhanced. Specifically, after the server terminal 130 acquires the image to be enhanced, it can learn the illumination enhancement sequence of the image to be enhanced through an image enhancement algorithm. Then, it denoises the image to be enhanced to obtain a first denoised image, and uses the first illumination enhancement component in the illumination enhancement sequence to enhance the first denoised image to obtain a first enhanced image. Then, it denoises the first enhanced image to obtain a second denoised image, and uses the second illumination enhancement component in the illumination enhancement sequence to enhance the second denoised image to obtain a second enhanced image, until all illumination enhancement components in the illumination enhancement sequence have completed image enhancement of the denoised image, resulting in a low-light enhanced image.

[0049] Of course, in another application scenario, the acquisition terminal 110 can also simultaneously acquire the image to be enhanced and enhance the image, which is not limited here.

[0050] Please see Figure 4This application provides an image enhancement method applicable to electronic devices, such as desktop computers, laptops, servers, etc., or electronic devices with image acquisition capabilities.

[0051] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0052] like Figure 4 As shown, the method may include the following steps:

[0053] Step 110: Obtain the image to be enhanced, and learn the illumination enhancement sequence of the image to be enhanced through an image enhancement algorithm.

[0054] The image to be enhanced can be obtained by capturing images through a capture device. The capture device can be an electronic device with image capture capabilities, such as a camera, a smartphone with a camera, etc. It can be understood that capturing can be a single shot or multiple shots, resulting in either a video (in which case the image to be enhanced can be any frame from the video) or multiple photographs (in which case the image to be enhanced can be any one of the photographs). In other words, in this embodiment, the image enhancement method is implemented on a frame-by-frame basis.

[0055] Regarding image acquisition, images can originate from real-time captures by the acquisition device, or from images captured by the acquisition device within a historical time period pre-stored on the electronic device. Therefore, for the electronic device, after capturing images, it can process them in real-time or pre-store them for later processing. For example, it can process images when the electronic device's CPU is low, or process images according to staff instructions. Thus, the image enhancement in this embodiment can target both real-time acquired images and images acquired within a historical time period; no specific limitation is made here.

[0056] Secondly, the illumination enhancement sequence includes multiple illumination enhancement vectors. These vectors are used to simulate and enhance the external lighting environment of the image to be enhanced, thereby achieving the image enhancement effect.

[0057] In one possible implementation, the image enhancement algorithm includes the Retinex algorithm. Specifically, based on the Retinex algorithm, the image to be enhanced is divided into illumination components and reflection components. Therefore, by learning an illumination enhancement sequence from the illumination components in the image to be enhanced, the illumination components in the image to be enhanced are corrected, thereby obtaining a low-light enhanced image.

[0058] Step 130: Denoise the image to be enhanced to obtain the first denoised image, and use the first illumination enhancement component in the illumination enhancement sequence to enhance the first denoised image to obtain the first enhanced image.

[0059] Step 150: Denoise the first enhanced image to obtain the second denoised image, and use the second illumination enhancement component in the illumination enhancement sequence to enhance the second denoised image to obtain the second enhanced image.

[0060] Step 170 continues until all illumination enhancement components in the illumination enhancement sequence have completed image enhancement of the denoised image, resulting in a low-light enhanced image.

[0061] Figure 5 A schematic diagram of an embodiment of the present invention is shown. Figure 5 In this embodiment of the invention, the image enhancement algorithm provided by the STAR module first learns multiple consecutive illumination enhancement components, and then performs multiple image enhancements on the low-light image to be enhanced. In each enhanced image, the self-supervised denoising module DN with adjacent downsampling is used to denoise the enhanced image. Through multiple loops, the low-light enhanced image is finally output.

[0062] Through the above process, the present invention introduces a denoising task into the image enhancement process, ensuring that brightness enhancement does not simultaneously enhance noise, thus significantly improving the image enhancement effect.

[0063] Furthermore, the parameter sharing of the denoising module in this embodiment of the invention greatly reduces the number of parameters, thereby improving the efficiency of image enhancement.

[0064] In one possible implementation, the low-light enhanced image in this embodiment of the invention is obtained through a joint enhancement and denoising network. This joint enhancement and denoising network is a trained machine learning model capable of denoising and enhancing the image to be enhanced.

[0065] In one possible implementation, the enhancement and denoising joint network includes a low-light enhancement module and a denoising module.

[0066] In one possible implementation, the training process for the enhanced denoising joint network may specifically include the following steps:

[0067] The first step is to obtain the training set. This training set includes multiple training image pairs, which consist of the image to be enhanced after denoising and image enhancement, and the low-light enhancement image.

[0068] The second step is to train the machine learning model with the current training image in the training set to obtain the loss values ​​of the low-light enhancement module and the denoising module.

[0069] The third step is to obtain the total loss value based on the loss value of the low-light enhancement module and the loss value of the noise reduction module.

[0070] If the total loss value meets the convergence condition, the training is complete, and the enhanced denoising joint network is obtained.

[0071] Otherwise, update the model parameters of the machine learning model and obtain the next training image to continue training the machine learning model until the total loss value meets the convergence condition.

[0072] Through the above training process, a joint enhancement and denoising network with the ability to denoise and enhance the image to be enhanced is obtained.

[0073] Please refer back to Figure 5 As shown, the low-light enhancement module is the STAR module, and the noise reduction enhancement module is the DN module. Figure 6 The diagram illustrates the structure of the low-light enhancement module STAR in this embodiment of the invention. The low-light enhancement module STAR includes a structure in parallel with a single-layer ViT-encoder and a CNN structure, as well as an embedding layer for vector mapping and tokens for generating tokens.

[0074] Specifically, the single-layer ViT-encoder structure includes two Nor normalization layers, a Multihead Attention layer, and an MLP Block multilayer perceptron. The MLP Block multilayer perceptron includes fully connected layers, a GELU activation function, and Dropout regularization. The CNN structure includes Nor normalization layers, Conv convolutional layers, and ReLU activation function layers.

[0075] With the above structure, the enhancement and denoising joint network provided by the present invention significantly reduces the number of parameters, which in turn helps to further reduce the computational load in the image enhancement process, enabling the enhancement and denoising joint network to be deployed on acquisition terminals with limited computing resources, such as drones, smart cameras, etc.

[0076] In one possible implementation, embodiments of the present invention obtain an illumination enhancement sequence for the image to be enhanced through a low-light enhancement module STAR. Specifically, as... Figure 6 As shown, the embedding layer is used to perform vector mapping on the image to be enhanced to obtain the feature map of the image to be enhanced. The feature map is then converted into tokens using tokens. Then, the illumination component in the image to be enhanced is learned based on the tokens using a single-layer ViT-encoder and CNN structure, thus obtaining the illumination enhancement sequence of the image to be enhanced.

[0077] The inventors discovered that existing token-switching methods consume excessive memory by directly flattening the image to be enhanced. Therefore, in one possible implementation, the token-switching process is implemented using a CNN. Specifically, as... Figure 7 As shown, H×W represents the feature map, and Conv represents the convolutional layer in the CNN structure. Multiple vector convolution operations are performed on the feature map using multiple Conv layers to obtain the convolution result. Simultaneously, the convolution result is flattened using the flattening function, ultimately resulting in the token H×W×C. in / p 2 This enables the conversion of feature maps into tokens.

[0078] Through the combination of the above embodiments, the present invention learns the illumination enhancement sequence of the image to be enhanced by the lightweight low-light enhancement module STAR, which reduces the number of parameters for image enhancement and helps to reduce the computational load of image enhancement. At the same time, the illumination components of the image to be enhanced are specifically corrected by the learned illumination enhancement sequence, thereby greatly improving the effect and efficiency of image enhancement.

[0079] Figure 8 A training schematic diagram of the denoising module in an embodiment of the present invention is shown. In one possible implementation, the denoising module includes a downsampler.

[0080] The calculation process of the loss value of the denoising module may include the following steps:

[0081] The downsampler G in the denoising module performs the first downsampling of adjacent pixels in the image y to be enhanced, resulting in the first downsampled image pair g1(y) and g2(y).

[0082] One of the downsampled images g1(y) in the first downsampled image pair is used as the input signal, and the other downsampled image g2(y) is used as the supervision signal.

[0083] The denoising module undergoes self-supervision using both input and supervision signals to obtain its loss value. The trained denoising module is considered to have denoising capability when its loss value satisfies the convergence condition. Specifically, denoising is performed on the image to be enhanced y and the input signal g1(y) respectively; the denoised image to be enhanced f is then... θ (y) performs a second downsampling of adjacent pixels to obtain the second downsampled image pair g1(f θ (y)) and g2(f θ (y)); based on the supervision signal g2(y) and the denoised input signal f θ (g1(y)) and the obtained second downsampled image pair g1(f) θ (y)) and g2(f θ(y)) , construct the reconstruction loss function and the regularization loss function respectively; calculate the loss value L of the reconstruction loss function. rec The loss value L of the regularized loss function reg And based on the loss value L of the reconstruction loss function rec The loss value L of the regularized loss function reg The loss value L of the denoising module is calculated, and then the loss value L of the denoising module is reduced to a set threshold so that the loss value L of the denoising module satisfies the convergence condition, thus obtaining the denoising module that has completed training.

[0084] Under the above embodiments, the denoising part of the present invention achieves self-supervised denoising, avoiding the shortcomings of the prior art that relies too much on synthetic datasets, and improving the generalization on real data.

[0085] Based on the trained enhancement and denoising joint network, this network can be used to enhance various images that have poor enhancement results in low light enhancement due to the simultaneous enhancement of noise.

[0086] Figure 9 The diagram shows a comparison of the effects of embodiments of this application with existing technologies in application scenarios. The first column of the diagram represents the input image, the second column represents the effect diagram of the existing technology, and the third column represents the effect diagram of the embodiment of this invention. It can be observed that the present invention significantly reduces noise while brightening low-light photos, avoiding the defect of amplifying noise while enhancing low light, and significantly improving the image enhancement effect.

[0087] Figure 10 The illustration shows application scenarios of embodiments of this application. Image acquisition in low-light environments can lead to decreased video image quality and information loss. Therefore, a clear and effective low-light enhancement system for video images is particularly important in fields such as drone aerial photography, traffic monitoring, industrial production, and military security. This invention can be adapted to various imaging devices for low-light enhancement processing and has broad application prospects.

[0088] The following are embodiments of the apparatus described in this application, which can be used to execute the image enhancement method involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the image enhancement method involved in this application.

[0089] Please see Figure 11 This application provides an image enhancement device 700. The device 700 includes, but is not limited to: an illumination enhancement sequence acquisition module 710, a first enhancement module 730, a second enhancement module 750, and a result acquisition module 770.

[0090] The illumination enhancement sequence acquisition module 710 is used to acquire the image to be enhanced and learn the illumination enhancement sequence of the image to be enhanced through an image enhancement algorithm. The illumination enhancement sequence includes multiple consecutive illumination enhancement components.

[0091] The first enhancement module 730 is used to denoise the image to be enhanced to obtain the first denoised image, and to enhance the first denoised image using the first illumination enhancement component in the illumination enhancement sequence to obtain the first enhanced image.

[0092] The second enhancement module 750 is used to denoise the first enhanced image to obtain the second denoised image, and to enhance the second denoised image using the second illumination enhancement component in the illumination enhancement sequence to obtain the second enhanced image.

[0093] The result acquisition module 770 is used to obtain a low-light enhanced image after all illumination enhancement components in the illumination enhancement sequence have completed image enhancement of the denoised image.

[0094] It should be noted that the image enhancement device provided in the above embodiments is only illustrated by the division of the above functional modules when performing image enhancement. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the image enhancement device will be divided into different functional modules to complete all or part of the functions described above.

[0095] Furthermore, the image enhancement apparatus and image enhancement method embodiments provided in the above embodiments belong to the same concept, and the specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0096] Figure 12 A schematic diagram of the structure of a server is shown according to an exemplary embodiment. This server is suitable for... Figure 3 The server-side 130 in the implementation environment is shown.

[0097] It should be noted that this server is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Nor should this server be interpreted as requiring or depending on any specific feature. Figure 12 One or more components of the exemplary server 2000.

[0098] The hardware architecture of Server 2000 can vary significantly due to differences in configuration or performance, such as... Figure 12As shown, the server 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0099] Specifically, power supply 210 is used to provide operating voltage for the various hardware devices on server 2000.

[0100] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. For example, in Figure 4 The implementation environment is shown, consisting of server 130 and acquisition terminal 110.

[0101] Of course, in other examples adapted in this application, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 12 As shown, this does not constitute a specific limitation.

[0102] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0103] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the server 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0104] Application 253 is a computer program that performs at least one specific task based on operating system 251, and may include at least one module ( Figure 12 (Not shown), each module may contain a computer program for server 2000. For example, the image enhancement device may be considered as application 253 deployed on server 2000.

[0105] Data 255 can be photos, pictures, etc. stored on a disk, or it can be input image data, etc., stored in memory 250.

[0106] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer programs stored in the memory 250, thereby performing operations and processing on massive amounts of data 255 stored in the memory 250. For example, an image enhancement method may be implemented by the central processing unit 270 reading a series of computer programs stored in the memory 250.

[0107] Furthermore, this application can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or combination thereof.

[0108] Please see Figure 13 This application provides an electronic device 4000, which may include: desktop computer, laptop computer, server, smart camera, drone, etc.

[0109] exist Figure 13 The electronic device 4000 includes at least one processor 4001, at least one communication bus 4002, and at least one memory 4003.

[0110] The processor 4001 and memory 4003 are connected, for example, via a communication bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0111] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0112] The communication bus 4002 may include a path for transmitting information between the aforementioned components. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0113] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0114] The memory 4003 stores a computer program, and the processor 4001 reads the computer program stored in the memory 4003 through the communication bus 4002.

[0115] When the computer program is executed by the processor 4001, it implements the image enhancement methods described in the above embodiments.

[0116] Furthermore, this application provides a storage medium storing a computer program that, when executed by a processor, implements the image enhancement methods described in the above embodiments.

[0117] This application provides a computer program product including a computer program stored in a storage medium. A processor of a computer device reads the computer program from the storage medium and executes the computer program, causing the computer device to perform the image enhancement methods described in the above embodiments.

[0118] Compared with related technologies, the beneficial effects of the present invention are:

[0119] 1. This invention proposes a novel image enhancement method. The low-light enhancement part of this invention achieves efficient low-light enhancement based on Retinex theory. The denoising part of this invention implements self-supervised denoising, avoiding the shortcomings of existing technologies that rely too much on synthetic datasets and improving generalization on real data. This invention connects the low-light enhancement module and the denoising module to handle both low-light enhancement and denoising tasks simultaneously, significantly reducing the computational load and avoiding the defect of amplifying noise while enhancing low-light, thus significantly improving the image enhancement effect.

[0120] 2. This invention adopts a lightweight network structure and shares parameters in the denoising module, which greatly reduces the number of model parameters and achieves efficient image enhancement.

[0121] 3. This invention proposes a novel end-to-end structure that simultaneously handles low-light enhancement and denoising tasks, suitable for high-resolution night images, and outputs low-light enhanced images with minimal noise.

[0122] 4. This invention can be applied to various imaging devices for low-light enhancement processing, and has a wide range of applications and prospects.

[0123] 5. This invention has low computational cost, is less prone to meshing, is suitable for high-resolution images, does not rely on synthetic datasets, and has good generalization ability on real data. Because there is a large inter-domain difference between real data and synthetic data, it avoids amplifying noise while performing low-light enhancement.

[0124] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0125] The above description is merely a preferred exemplary embodiment of the present invention and is not intended to limit the implementation of the present invention. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of protection claimed in the claims.

Claims

1. An image enhancement method, characterized in that, The method includes: An image to be enhanced is acquired, and an illumination enhancement sequence of the image to be enhanced is learned through an image enhancement algorithm. The illumination enhancement sequence includes multiple consecutive illumination enhancement components. The image to be enhanced is denoised to obtain a first denoised image, and the first illumination enhancement component in the illumination enhancement sequence is used to enhance the first denoised image to obtain a first enhanced image. The first enhanced image is denoised to obtain a second denoised image, and the second illumination enhancement component in the illumination enhancement sequence is used to enhance the second denoised image to obtain a second enhanced image. The process continues until all illumination enhancement components in the illumination enhancement sequence have completed image enhancement of the denoised image, resulting in a low-light enhanced image.

2. The method as described in claim 1, characterized in that, The process of learning the illumination enhancement sequence of the image to be enhanced through an image enhancement algorithm includes: The image to be enhanced is vector-mapped to obtain a feature map of the image to be enhanced, and the feature map is converted into a token; The illumination components in the image to be enhanced are learned based on the token to obtain the illumination enhancement sequence of the image to be enhanced.

3. The method as described in claim 2, characterized in that, The step of converting the feature map into a token includes: Perform multiple vector convolution operations on the feature map to obtain the convolution result; The result of the convolution operation is flattened to obtain the token.

4. The method according to any one of claims 1 to 3, characterized in that, The low-light enhanced image is obtained through an enhancement-denoising joint network; the enhancement-denoising joint network is a trained machine learning model capable of denoising and enhancing the image to be enhanced. The enhanced denoising joint network includes a low-light enhancement module and a denoising module.

5. The method as described in claim 4, characterized in that, The training process of the enhanced denoising joint network includes: Obtain a training set, which includes multiple training image pairs; the training image pairs include an image to be enhanced after denoising and image enhancement, and a low-light enhancement image. The current training image in the training set is used to train the machine learning model to obtain the loss value of the low-light enhancement module and the loss value of the denoising module. The total loss value is obtained based on the loss value of the low-light enhancement module and the loss value of the noise reduction module; If the total loss value satisfies the convergence condition, the training is completed, and the enhanced denoising joint network is obtained; otherwise, the model parameters of the machine learning model are updated, and the next training image is obtained to input the machine learning model for continued training until the total loss value satisfies the convergence condition.

6. The method as described in claim 5, characterized in that, The noise reduction module includes a downsampler; The calculation process of the loss value of the denoising module includes: using the downsampler in the denoising module to perform the first downsampling of adjacent pixels of the image to be enhanced, and obtaining the first downsampled image pair; Use one of the downsampled images in the first downsampled image pair as the input signal and the other downsampled image as the supervision signal; The denoising module is self-supervised by the input signal and the supervision signal to obtain the loss value of the denoising module. When the loss value of the denoising module satisfies the convergence condition, the denoising module that has completed training has denoising capability.

7. The method as described in claim 6, characterized in that, The self-supervision of the denoising module using the input signal and the supervision signal to obtain the loss value of the denoising module includes: Denoise is denoised both the image to be enhanced and the input signal. The denoised image to be enhanced is then downsampled a second time by adjacent pixels to obtain a second downsampled image pair; Based on the supervision signal, the denoised input signal, and the obtained second downsampled image pair, a reconstruction loss function and a regularization loss function are constructed respectively; Calculate the loss value of the reconstruction loss function and the loss value of the regularization loss function, and calculate the loss value of the denoising module based on the loss values ​​of the reconstruction loss function and the regularization loss function.

8. An image enhancement device, characterized in that, The device includes: An illumination enhancement sequence acquisition module is used to acquire an image to be enhanced and learn an illumination enhancement sequence of the image to be enhanced through an image enhancement algorithm. The illumination enhancement sequence includes multiple consecutive illumination enhancement components. The first enhancement module is used to denoise the image to be enhanced, to obtain the first denoised image, and then use the first illumination enhancement component in the illumination enhancement sequence to enhance the first denoised image, to obtain the first enhanced image. The second enhancement module is used to denoise the first enhanced image to obtain the second denoised image, and then use the second illumination enhancement component in the illumination enhancement sequence to enhance the second denoised image to obtain the second enhanced image. The result acquisition module is used to obtain a low-light enhanced image after all the illumination enhancement components in the illumination enhancement sequence have completed image enhancement of the denoised image.

9. An electronic device, characterized in that, include: At least one processor, at least one memory, and at least one communication bus, wherein, The memory stores a computer program, and the processor reads the computer program from the memory via the communication bus; When the computer program is executed by the processor, it implements the image enhancement method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image enhancement method as described in any one of claims 1 to 7.

Images