Collaborative unsupervised image enhancement and denoising method, system and device and medium

Abstract

The invention provides a collaborative unsupervised image enhancement and denoising method, system and device and a medium, and belongs to the technical field of image processing. The method comprises the following steps: constructing a data set of a low-light image based on an image low-light enhancement data set; constructing an illuminance-driven iterative enhancement network to improve the image brightness and adjust the exposure by using the illuminance characteristics; constructing a regional denoising network to remove dense noise and retain semantic information of a bright region by using a mixed denoising module; generating a collaborative unsupervised framework by using an illumination-driven iterative enhancement network and a regional denoising network; training a collaborative unsupervised framework by using a comprehensive loss function and an Adam optimizer; and obtaining a final image enhancement and denoising result through the collaborative unsupervised framework so as to complete the testing of the collaborative unsupervised framework. According to the method, the balance between the brightness and the exposure can be realized in the enhancement process, the visual recovery effect of the bright area is improved on the basis of global image denoising, and the image enhancement and denoising performance of the network is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and more specifically relates to a collaborative unsupervised image enhancement and denoising method, system, device and medium. Background Art

[0002] In low-light or backlit conditions, due to the inherent limitations of optical devices and the variability of external imaging conditions, it is difficult for camera sensors to capture enough light to fully resolve the details and colors in the image. The resulting low-light images are noisy and lack detail and color accuracy. These degradation factors significantly damage the visual aesthetics of the image and have an adverse impact on a wide range of downstream computer vision and multimedia tasks. Therefore, low-light enhancement methods have been proposed to improve the quality and details of images captured under low-light conditions and provide good explanations for subsequent downstream tasks. Currently, algorithms used to solve image denoising tasks can be divided into two categories: traditional optimization methods and deep learning-based methods.

[0003] Early traditional methods include histogram equalization and gamma correction methods. The histogram equalization method changes the contrast of the image by adjusting the original histogram to a uniform histogram. For example, the contrast-constrained adaptive histogram equalization method sets a threshold to determine the pixel value in the histogram. If it exceeds the threshold, the pixels are cropped and then evenly distributed to each grayscale level. In addition, gamma correction edits the gamma curve of the image to achieve nonlinear editing and detect the dark and bright parts of the image signal. Although the contrast can be enhanced in this way, the performance is poor in areas with high grayscale values ​​and under different lighting conditions. In general, these methods do not consider the correlation between pixels and illumination components, resulting in artifacts and dense noise.

[0004] Other traditional methods based on Retinex theory can simulate the color stability of human visual perception and use different priors to constrain the solution space of the reflection component and the illumination component. For example, the single-scale Retinex method first filters the RGB channels through a constructed Gaussian surround function to obtain the illumination component, and then obtains the reflection component by enhancing the difference between the original image and the illumination component in the logarithmic domain. However, these Retinex-based methods assume that the image is noise-free, which is inconsistent with the image characteristics obtained in real low-light scenes, and therefore introduces severe noise or color distortion locally.

[0005] With the rapid development and widespread application of deep learning, low-light image enhancement methods based on deep learning can be divided into two categories: data-driven supervised methods and unsupervised enhancement methods. Data-driven deep learning methods use pairs of low-light / normal-light images for training. These methods directly learn part of the mapping between image pairs. However, these methods lack interpretable theories and ignore human visual perception. Deep learning methods based on Retinex theory obtain reflection components and illumination components by designing a decomposition network, and process them separately, and finally combine them through formulas to obtain enhanced images. However, these methods often require a large number of parameters and a long training time. In addition, since the large number of low-light / normal-light image pairs required by supervised methods are difficult to collect in real scenes, the generalization and applicability of the model will be reduced.

[0006] Therefore, in order to solve the lack of enhancement caused by the lack of absolute labels, unsupervised methods that can achieve single image training are proposed. For example, the SGZ method proposes an unsupervised framework with semantic guidance, which is the first method to integrate high-level semantic information with the low-light enhancement network. The RRDNet method proposes a three-branch network that decomposes the input into illumination, reflection, and noise components, and designs a loss function to evaluate the current image decomposition and guide the estimation of noise. The RUAS method designs an optimization model based on the Retinex rule to describe the inherent underexposure structure of low-light images, and establishes the overall framework of the enhancement network by extending the corresponding optimization process. The SCI method develops a new self-calibrated illumination learning framework for fast, flexible, and robust image brightening in real-world low-light scenes. Although these unsupervised methods can effectively improve image brightness and promote the development of enhancement methods, they also have some shortcomings, such as restricting the use of image prior knowledge in the unsupervised framework and ignoring subsequent image refinement enhancement tasks such as image denoising and color correction. Summary of the invention

[0007] In view of the above problems, the purpose of the present invention is to provide a collaborative unsupervised image enhancement and denoising method, system, device and medium, which can improve the brightness and contrast of low-light images while alleviating overexposure and eliminating residual noise, retaining the structural details of the bright areas of the image, improving the visual effect, and improving the enhancement and denoising performance of the network.

[0008] In order to achieve the above object, the present invention is implemented through the following technical solutions: In a first aspect, an embodiment of the present application provides a collaborative unsupervised image enhancement and denoising method, comprising: Constructing a dataset of low-light images based on an image low-light enhancement dataset, wherein the dataset includes a training dataset and a test dataset; Construct an illumination-driven iterative enhancement network to utilize illumination features to increase image brightness and adjust exposure to generate enhanced images. Constructing a regional denoising network to remove dense noise and retain semantic information of bright areas by using a hybrid denoising module; the hybrid denoising module includes an illumination mask branch and a center mask branch; A collaborative unsupervised framework is generated using an illumination-driven iterative enhancement network and a region-based denoising network. Based on the training dataset, the collaborative unsupervised framework is trained using a comprehensive loss function containing prior information and the Adam optimizer; Using the test dataset, the collaborative unsupervised framework is used to obtain the final image enhancement and denoising results to complete the testing of the collaborative unsupervised framework. Low-light images are collected and enhanced and denoised using a tested collaborative unsupervised framework.

[0009] In an optional implementation, constructing a low-light image dataset based on the image low-light enhancement dataset includes: Construct a dataset of low-light images based on the LOL dataset; In the low-light image dataset, 485 sRGB images from the LOL dataset are selected and cropped to a size of 120×120 as the training dataset; The remaining 15 images of the LOL dataset are selected from the low-light image dataset, and the original image size of 400×600 is used as the test dataset.

[0010] In an optional embodiment, constructing an illumination-driven iterative enhancement network to improve image brightness and adjust exposure using illumination features includes: Construct an illumination-driven iterative enhancement network, and embed an illumination optimization module, an illumination prediction module, and an illumination calibration module into the illumination-driven iterative enhancement network; Enhance the illumination component of low-light images through the illumination optimization module; The illumination prediction module is based on the multi-scale illumination features of the captured low-light image; The illumination components are adjusted according to the multi-scale illumination features by an illumination calibration module to generate an illumination map.

[0011] In an optional embodiment, the construction of a regional denoising network to remove dense noise and retain semantic information of bright areas by using a hybrid denoising module includes: A regional denoising network is constructed, wherein the regional denoising network is provided with a hybrid denoising module, wherein the hybrid denoising module is provided with an illumination mask branch and a center mask branch; The input end of the sub-region denoising network is used to convert the illumination map into an illumination mask map, and input the illumination mask map and the enhanced image into a hybrid denoising module; The hybrid denoising module is used to: In the illumination mask branch, the illumination mask map and the enhanced image are fused, and the dilation rate is adjusted through the symmetrical dilated convolution block to expand the range of capturing local features and aggregate features; In the center mask branch, the blind spot is set by the center mask convolution, and the global features are obtained by using the residual dilation convolution block to remove the dense noise in the dark area; The outputs of the illumination mask branch and the two center mask branches are fused to output a denoised image that retains texture details in bright areas.

[0012] In an optional embodiment, the method of generating a collaborative unsupervised framework using an illumination-driven iterative enhancement network and a region-based denoising network includes: The illumination-driven iterative enhancement network and the region-based denoising network are integrated to generate a collaborative unsupervised framework for performing image enhancement and image denoising processes. The image enhancement process includes: Based on the iterative framework of Retinex theory, the illumination optimization module is used to improve the image brightness, the illumination estimation module is used to generate a linear enhanced image, and the illumination calibration module is used to adjust the overexposure through the illumination characteristics to generate an enhanced image; The image denoising process includes: The illumination map of the enhanced image is converted into an illumination mask map to protect the pixels in the bright area, and a denoised image that retains the texture details of the bright area is output on the basis of removing the global noise.

[0013] In an optional implementation, the iterative process of the illumination driven iterative enhancement network is expressed by the following formula:

[0014] in, Represents low-light images, Represents the illumination component generated by the illumination optimization module, Represents the reflection component obtained according to the Retinex theory, Indicates that the illumination feature is obtained through the illumination prediction module; Represents the illumination calibration module, Indicates that the illumination calibration module outputs the calibration component; is the input for the next iteration. is the number of iterations.

[0015] In an optional embodiment, the collaborative unsupervised framework is trained based on the training data set using a comprehensive loss function containing prior information and an Adam optimizer, including: Import the training dataset and use the Adam optimizer to train the collaborative unsupervised framework; The initial learning rate is set to 0.0001, and a comprehensive loss function combining fidelity loss, smoothness loss and L1 regularization is used as the output of the training loss constraint iterative process. After the training is completed, the illumination-driven iterative enhancement network is frozen; Use L1 regularization as the training loss to constrain the denoising output to train the sub-region denoising network. When the loss is stable, save the weights of the illumination-driven iterative enhancement network and the sub-region denoising network and complete the training. The comprehensive loss function L is as follows:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] in, is the fidelity loss function, is the smoothness loss function, is the weight of the fidelity loss function, is the weight of the slip loss function, i and j represent the i-th pixel and the j-th pixel respectively. express The neighboring pixels within its 5×5 window, represents the associated weight; is the constraint function of lighting; represents the output function after optimization of the illumination calibration module, is its weight; is the loss function of linear enhanced image and refinement enhanced image, is its weight; Represents a linearly enhanced image.

[0022] In a second aspect, the embodiment of the present application further provides a collaborative unsupervised image enhancement and denoising system, comprising: A data set construction module is configured to construct a data set of low-light images based on an image low-light enhancement data set, wherein the data set includes a training data set and a test data set; An enhancement network architecture construction module configured to construct an illumination-driven iterative enhancement network to utilize illumination features to increase image brightness and adjust exposure to generate an enhanced image; A denoising network architecture construction module configured to construct a sub-region denoising network to remove dense noise and retain semantic information of bright areas using a hybrid denoising module; the hybrid denoising module includes an illumination mask branch and a center mask branch; A network generation module configured to generate a collaborative unsupervised framework using an illumination-driven iterative enhancement network and a region-based denoising network; A network training module configured to train a collaborative unsupervised framework based on a training dataset using a comprehensive loss function containing prior information and an Adam optimizer; A network testing module, configured to use a test data set to obtain final image enhancement and denoising results through the collaborative unsupervised framework to complete the testing of the collaborative unsupervised framework; The enhancement and denoising processing module is configured to collect low-light images and perform image enhancement and denoising using a tested collaborative unsupervised framework.

[0023] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the steps of the collaborative unsupervised image enhancement and denoising method as described in any one of the above items are implemented.

[0024] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the collaborative unsupervised image enhancement and denoising method as described in any one of the above items are implemented.

[0025] It can be seen from the above technical solutions that the present invention has the following advantages: This application designs a two-level collaborative unsupervised image enhancement and denoising framework to balance image brightness, exposure and noise in the restoration process, focusing on multiple visual effects, and defines a comprehensive unsupervised loss function that can fully utilize prior information. The collaborative unsupervised framework includes an illumination-driven iterative enhancement network based on Retinex theory and a region-based denoising network to refine the image structure while enhancing the image brightness.

[0026] This application proposes an illumination-driven iterative enhancement network, embedding three learning-based modules that are responsible for enhancing the lighting component, fine-tuning the exposure level, and achieving efficient linear optimization to improve image brightness while preventing overexposure.

[0027] This application refines the enhancement results by establishing a regional denoising network, and uses a hybrid denoising module including an illumination mask branch and a center mask branch. In the illumination mask branch, the illumination map is converted into an illumination mask map and matched with a symmetrical dilated convolution block to protect the pixels in the bright area. In the center mask branch, a center mask convolution is used to set a blind spot and matched with a residual dilated convolution block with the same dilation rate to eliminate global dense noise. Finally, the network can remove global noise while retaining the structural details of the bright area and achieving accurate semantic information output.

[0028] It can be seen that compared with the prior art, the present invention has outstanding substantive features and significant progress, and the beneficial effects of its implementation are also obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0030] Figure 1 A schematic diagram of the process flow of the collaborative unsupervised image enhancement and denoising method provided in this application.

[0031] Figure 2 This is a schematic diagram of the structure of the illumination-driven iterative enhancement network provided in this application.

[0032] Figure 3 It is a structural principle diagram of the regional denoising network provided in this application.

[0033] Figure 4 It is a structural principle diagram of the collaborative unsupervised framework provided in this application.

[0034] Figure 5 This is a comparison chart of the visual effects of this application and the existing enhancement algorithm.

[0035] Figure 6 This is a comparison chart of the PSNR, SSIM and LPIPS indicators of this application and the existing enhancement algorithm.

[0036] Figure 7 A schematic diagram of the structure of the collaborative unsupervised image enhancement and denoising system provided in this application.

[0037] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0038] In the specific steps of the collaborative unsupervised image enhancement and denoising method, which will be described in detail below, various embodiments of the present disclosure will be described more fully. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but rather the present disclosure should be understood to cover all adjustments, equivalents and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.

[0039] Hereinafter, the terms "include" or "may include" used in various embodiments of the present disclosure indicate the presence of disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present disclosure, the terms "include", "have", and their cognates are intended only to indicate specific features, numbers, steps, operations, elements, components, or a combination of the foregoing, and should not be understood as first excluding the presence of one or more other features, numbers, steps, operations, elements, components, or a combination of the foregoing or the possibility of adding one or more features, numbers, steps, operations, elements, components, or a combination of the foregoing.

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 1 FIG. 1 is a flowchart of a method for collaborative unsupervised image enhancement and denoising in a specific embodiment, the method comprising: S1: A dataset of low-light images is constructed based on an image low-light enhancement dataset, wherein the dataset includes a training dataset and a test dataset.

[0042] In a specific implementation, a low-light image dataset is constructed based on the LOL dataset. 485 sRGB images in the LOL dataset are selected from the low-light image dataset and cropped to a size of 120×120 as a training dataset, and the remaining 15 images of the LOL dataset are selected from the low-light image dataset, with the original image size of 400×600, as a test dataset.

[0043] S2: Construct an illumination-driven iterative enhancement network to utilize illumination features to increase image brightness and adjust exposure to generate enhanced images.

[0044] In a specific implementation, an illumination drive iterative enhancement network is constructed, and an illumination optimization module, an illumination prediction module, and an illumination calibration module are embedded in the illumination drive iterative enhancement network. Figure 2 As shown, by embedding three learning-based modules in the illumination-driven iterative enhancement network, they are responsible for enhancing the lighting component, fine-tuning the exposure level, and achieving efficient linear optimization to improve image brightness while preventing overexposure.

[0045] Specifically, the illumination optimization module is first used to enhance the illumination component of the low-light image. At the same time, the illumination prediction module captures multi-scale illumination features. Since the illumination characteristics contain clearer information about bright areas, they can promote information prediction in dark areas. Embedding the illumination characteristics into the illumination calibration module can better ensure that the iterative input and low-light observations are at the same level, indirectly constrain each iterative process, and explore exposure correction. Therefore, the multi-scale illumination features are combined with the illumination calibration module to guide the illumination calibration module to adjust the enhanced illumination component obtained by the illumination optimization module to ensure the correct transmission of illumination information during the iteration process.

[0046] It should be noted that the illumination-driven iterative enhancement network is built based on the iterative framework of Retinex theory.

[0047] The basic unit of Retinex theory is , Based on the Retinex theory, the iterative process of the proposed illumination-driven iterative enhancement network can be described as:

[0048] in, Represents low-light images, Represents the illumination component generated by the illumination optimization module, Represents the reflection component obtained according to the Retinex theory, It means that the illumination features are obtained through the multi-scale illumination prediction module, and the reflection component and lighting features are used as the illumination guidance calibration module The output of the calibration component Finally, the low-light image and the calibration component are added together as the input for the next iteration. , repeat the above steps, the number of iterations is .

[0049] S3: construct a regional denoising network to remove dense noise and retain semantic information of bright areas by using a hybrid denoising module; the hybrid denoising module includes an illumination mask branch and a center mask branch.

[0050] In a specific embodiment, Figure 3As shown, a regional denoising network is constructed, and a hybrid denoising module is provided in the regional denoising network, and an illumination mask branch and a center mask branch are provided in the hybrid denoising module. In combination with the above structure, the regional denoising network is configured so that the regional denoising network can realize image denoising by executing the following process.

[0051] First, the illumination map obtained from the enhancement stage is converted into an illumination mask map, which is divided into bright and dark areas according to the criterion of setting the pixel mean as a threshold. Pixels whose values ​​exceed the threshold are classified as bright areas and retain features, while those below the threshold are set as blind spots. Subsequently, the illumination mask map and the enhanced image are used as inputs to the region-based denoising network, and denoising is performed using a hybrid denoising module consisting of an illumination mask branch and a center mask branch.

[0052] In the illumination mask branch, the illumination mask map and the enhanced image are fused through a symmetrical dilated convolution block, where the dilation rate of the dilated convolution is first adjusted from 1 to 4 to expand the range of capturing local features, and then the dilation rate is reduced from 4 to 1 to aggregate features and alleviate the grid phenomenon caused by excessive dilation rate. At the same time, in the center mask branch, a blind spot is set for the entire image through the center mask convolution, and the residual dilated convolution block with the same dilation rate is used to obtain global features and remove dense noise in dark areas. Finally, the outputs of the two branches are fused so that the network can make full use of the information in bright and dark areas. The output result can retain the texture details in bright areas and improve the quality of visual perception.

[0053] S4: Generate a collaborative unsupervised framework using an illumination-driven iterative enhancement network and a region-based denoising network.

[0054] In a specific embodiment, Figure 4 As shown in the figure, the illumination-driven iterative enhancement network and the regional denoising network are combined to generate a collaborative unsupervised framework to process low-light images in stages. Specifically, in the first stage, the image enhancement process is performed through the illumination-driven iterative enhancement network; in the second stage, the image denoising process is performed through the regional denoising network.

[0055] In the first stage, an iterative framework based on Retinex theory is designed to improve the image brightness through the illumination optimization module, generate a linear enhanced image with rich prior information using the illumination estimation module, and adjust the overexposure through the illumination characteristics using the illumination calibration module to ensure the accurate transmission of feature information.

[0056] In the second stage, the illumination map is converted into an illumination mask map to protect the pixels in the bright area and achieve accurate semantic information output based on removing global noise.

[0057] S5: Based on the training dataset, the collaborative unsupervised framework is trained using a comprehensive loss function containing prior information and the Adam optimizer.

[0058] In a specific implementation, the linear enhanced image generated by the illumination prediction module is introduced as a reference for the final enhancement, which can avoid falling into a local optimal solution and ensure the correct direction of the iterative process. And setting an appropriate loss function weight can effectively control the degree of each module to avoid generating low brightness and low contrast restoration results.

[0059] In the loss function of the illumination-driven iterative enhancement network, fidelity loss is used To ensure pixel-level consistency between the observed illumination and the input in multiple iterations, it is expressed as:

[0060] Additionally, a smoothness loss is used to constrain the illumination component:

[0061] where the definition For the pixels, express Neighboring pixels within its 5×5 window. Symbol represents the associated weight. and smoothness loss Combined with the common constraint Len as lighting:

[0062] in, and Set to 1.5 and 1. At the same time, the linear enhancement image and Regularization is used to optimize the output of the calibration module to ensure the correct transmission of iterative information. The formula is as follows:

[0063] in, Represents information related to the calibration component, represents the linear enhanced image. By using the rich prior information in the linear enhanced image, the enhancement performance of the illumination guided calibration module can be improved. Finally, in order to further optimize the final enhanced output and prevent the interference of iterative errors, Regularization establishes the loss function for linear enhanced images and thin enhanced images. The formula is as follows:

[0064] in, Represents the final refined enhanced image. In summary, the overall loss function of the iterative enhancement framework is as follows:

[0065] in, and is set to 2.

[0066] In addition, As the loss function for training the region denoising network, represents the input of the denoising network, Represents the output denoised image.

[0067] When training, first import the training data set, and then use the Adam optimizer to train the collaborative unsupervised framework. Set the initial learning rate to 0.0001, and use the comprehensive loss function of fidelity loss, smoothness loss and L1 regularization as the output of the training loss constraint iterative process. After training, freeze the illumination-driven iterative enhancement network, and use L1 regularization as the training loss constraint denoising output to train the sub-region denoising network. When the loss is stable, save the network weights and complete the training.

[0068] S6: Using the test dataset, the final image enhancement and denoising results are obtained through the collaborative unsupervised framework to complete the testing of the collaborative unsupervised framework.

[0069] In a specific implementation, in a test data set, test data is extracted and passed through an illumination-driven iterative enhancement network to obtain an enhanced image and illumination map, which is then transmitted to a regional denoising network. After two stages of collaborative processing, the final enhanced and denoised image is obtained.

[0070] S7: Collect low-light images and use the tested collaborative unsupervised framework to enhance and denoise the images.

[0071] After using this method for image enhancement and denoising, it can be found that this method greatly reduces the noise by using a collaborative unsupervised framework. Figure 5 A comparison chart of the enhanced visual effects of this method and the prior art is provided. From the perspective of visual effects, the collaborative unsupervised framework of this method can improve the brightness and contrast of low-light images while alleviating overexposure and eliminating residual noise, and retain the structural details of the bright area of ​​the image, improve the visual effects, and enhance the enhancement and denoising performance of the network.

[0072] Furthermore, in order to better reflect the enhancement and denoising effects of this method, the following comparative experiments can be used for verification: Comparative experimental plan: This method is compared with the unsupervised learning-based denoising network SCI and PairLIE methods in terms of PSNR, SSIM and LPIPS on the LOL dataset. The higher the PSNR value, the smaller the difference between the pixel value of the denoised image and the pixel value of the original image. The higher the SSIM value, the more similar the structure of the denoised image and the original image, which means the stronger the denoising performance. The lower the LPIPS value, the better the visual perception effect of the image.

[0073] Experimental conditions: The test images are the real low-light image dataset LOL. The test indicators are PSNR, SSIM and LPIPS. The present invention does not involve any normal light images in the training stage.

[0074] Experimental results: Figure 6 The enhanced performance of different methods on the LOL dataset is provided. It can be seen that this method is significantly ahead of the SCI and PairLIE methods.

[0075] In summary, the present invention discloses a collaborative unsupervised image enhancement and denoising method that combines an illumination-driven iterative enhancement network and a regional denoising network into a collaborative unsupervised framework, and coordinates an illumination optimization module, a multi-scale illumination prediction module, and an illumination-guided calibration module in the illumination-driven iterative enhancement network to promote iterative convergence, thereby improving brightness and contrast and alleviating overexposure. In the regional denoising network, the illumination map is converted into an illumination mask map, retaining the detailed structure of the bright area, and using a symmetrically expanded convolution block to expand the receptive field to capture the bright area features and eliminate the grid phenomenon, which greatly improves the visual perception quality of the enhanced image.

[0076] like Figure 7 As shown, the following is an embodiment of the collaborative unsupervised image enhancement and denoising system provided by the embodiments of the present disclosure. The system and the collaborative unsupervised image enhancement and denoising methods of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the collaborative unsupervised image enhancement and denoising system, please refer to the embodiments of the above-mentioned collaborative unsupervised image enhancement and denoising method.

[0077] A collaborative unsupervised image enhancement and denoising system comprises: a data set construction module 1, an enhancement network architecture construction module 2, a denoising network architecture construction module 3, a network generation module 4, a network training module 5, a network testing module 6, and an enhancement and denoising processing module 7.

[0078] The data set construction module 1 is configured to construct a data set of low-light images based on the image low-light enhancement data set, wherein the data set includes a training data set and a test data set.

[0079] The enhancement network architecture construction module 2 is configured to construct an illumination-driven iterative enhancement network to utilize illumination features to increase image brightness and adjust exposure to generate an enhanced image.

[0080] The denoising network architecture construction module 3 is configured to construct a regional denoising network to remove dense noise and retain the semantic information of the bright area by using a hybrid denoising module; the hybrid denoising module includes an illumination mask branch and a center mask branch.

[0081] The network generation module 4 is configured to generate a collaborative unsupervised framework using an illumination-driven iterative enhancement network and a regional denoising network.

[0082] The network training module 5 is configured to train the collaborative unsupervised framework based on the training data set using a comprehensive loss function containing prior information and an Adam optimizer.

[0083] The network testing module 6 is configured to use the test data set to obtain the final image enhancement and denoising results through the collaborative unsupervised framework to complete the testing of the collaborative unsupervised framework.

[0084] The enhancement and denoising processing module 7 is configured to collect low-light images and perform image enhancement and denoising processing using the tested collaborative unsupervised framework.

[0085] The collaborative unsupervised image enhancement and denoising system provided in this embodiment realizes effective enhancement and denoising of low-light images by constructing an illumination-driven iterative enhancement network and a regional denoising network, and uses a collaborative unsupervised framework for training and testing, thereby improving image brightness and clarity while retaining the semantic information of bright areas, effectively improving image quality.

[0086] Figure 8 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0087] The collaborative unsupervised image enhancement and denoising method provided in the embodiment of the present application can be applied to electronic devices. It will be appreciated by those skilled in the art that the electronic device structure involved in the embodiment of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or less components than shown, or combine certain components, or arrange different components. In an embodiment of the present invention, the electronic device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0088] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, buttons, a camera, a display, and a SIM card interface, etc.

[0089] The processor may include one or more processing units, for example, the processor may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors.

[0090] The processor can be the nerve center and command center of the electronic device. The controller can generate an operation control signal according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0091] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may store instructions or data that the processor has just used or is cyclically used. If the processor needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor, and thus improves system efficiency.

[0092] The external memory interface can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor through the external memory interface to implement data storage functions. For example, files such as music and videos can be saved in the external memory card.

[0093] The internal memory can be used to store computer executable program codes, which include instructions. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. The internal memory may include a program storage area and a data storage area. The internal memory may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0094] The wireless communication function of an electronic device can be realized through an antenna, a wireless communication module, a modem processor, and a baseband processor.

[0095] The wireless communication module can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0096] Electronic devices can implement audio functions, etc. through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0097] Electronic devices can achieve shooting functions through ISP, camera, video codec, GPU, display and application processor.

[0098] Electronic devices can achieve display functions through GPU, display screen and application processor.

[0099] The GPU is a microprocessor for image processing that connects the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more GPUs that execute program instructions to generate or change display information.

[0100] The display screen is used to display images, videos, etc. The display screen includes a display panel.

[0101] The above-mentioned electronic device implements the collaborative unsupervised image enhancement and denoising method of the present application by constructing a specific data set, building an illumination-driven iterative enhancement network and a regional denoising network, and generating a collaborative unsupervised framework for training and testing, thereby realizing brightness enhancement and noise removal of low-light images, effectively improving image quality, while retaining the semantic information of bright areas, and bringing significant beneficial effects to the field of image processing.

[0102] The storage medium provided in the present application stores a program product capable of implementing a collaborative unsupervised image enhancement and denoising method.

[0103] The collaborative unsupervised image enhancement and denoising method includes: constructing a low-light image dataset based on an image low-light enhancement dataset, the dataset including a training dataset and a test dataset; constructing an illumination-driven iterative enhancement network to utilize illumination features to improve image brightness and adjust exposure to generate an enhanced image; constructing a regional denoising network to utilize a hybrid denoising module to remove dense noise and retain semantic information in bright areas; the hybrid denoising module includes an illumination mask branch and a center mask branch; generating a collaborative unsupervised framework using the illumination-driven iterative enhancement network and the regional denoising network; based on the training dataset, training the collaborative unsupervised framework using a comprehensive loss function containing prior information and an Adam optimizer; using the test dataset, obtaining the final image enhancement and denoising results through the collaborative unsupervised framework to complete the testing of the collaborative unsupervised framework; collecting low-light images, and using the tested collaborative unsupervised framework to perform image enhancement and denoising processing.

[0104] In some possible embodiments, the collaborative unsupervised image enhancement and denoising method of the present disclosure can be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0105] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0106] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A collaborative unsupervised image enhancement and denoising method, characterized in that: include: Constructing a dataset of low-light images based on an image low-light enhancement dataset, wherein the dataset includes a training dataset and a test dataset; Construct an illumination-driven iterative enhancement network to utilize illumination features to improve image brightness and adjust exposure to generate enhanced images. Constructing a regional denoising network to remove dense noise and retain semantic information of bright areas by using a hybrid denoising module; the hybrid denoising module includes an illumination mask branch and a center mask branch; A collaborative unsupervised framework is generated using an illumination-driven iterative enhancement network and a region-based denoising network. Based on the training dataset, the collaborative unsupervised framework is trained using a comprehensive loss function containing prior information and the Adam optimizer; Using the test dataset, the collaborative unsupervised framework is used to obtain the final image enhancement and denoising results to complete the testing of the collaborative unsupervised framework. Low-light images are collected and enhanced and denoised using a tested collaborative unsupervised framework.

2. The collaborative unsupervised image enhancement and denoising method according to claim 1, characterized in that: The low-light image dataset is constructed based on the image low-light enhancement dataset, including: Construct a dataset of low-light images based on the LOL dataset; In the low-light image dataset, 485 sRGB images from the LOL dataset are selected and cropped to a size of 120×120 as the training dataset; The remaining 15 images of the LOL dataset are selected from the low-light image dataset, and the original image size of 400×600 is used as the test dataset.

3. The collaborative unsupervised image enhancement and denoising method according to claim 1, characterized in that: The illumination driven iterative enhancement network is constructed to improve the image brightness and adjust the exposure by using the illumination feature, including: Construct an illumination-driven iterative enhancement network, and embed an illumination optimization module, an illumination prediction module, and an illumination calibration module into the illumination-driven iterative enhancement network; Enhance the illumination component of low-light images through the illumination optimization module; The illumination prediction module is based on the multi-scale illumination features of the captured low-light image; The illumination components are adjusted according to the multi-scale illumination features through an illumination calibration module to generate an illumination map.

4. The collaborative unsupervised image enhancement and denoising method according to claim 3, characterized in that: The construction of a regional denoising network to remove dense noise and retain semantic information of bright areas by using a hybrid denoising module includes: A regional denoising network is constructed, wherein the regional denoising network is provided with a hybrid denoising module, wherein the hybrid denoising module is provided with an illumination mask branch and a center mask branch; The input end of the sub-region denoising network is used to convert the illumination map into an illumination mask map, and input the illumination mask map and the enhanced image into a hybrid denoising module; The hybrid denoising module is used to: In the illumination mask branch, the illumination mask map and the enhanced image are fused, and the dilation rate is adjusted through the symmetrical dilated convolution block to expand the range of capturing local features and aggregate features; In the center mask branch, the blind spot is set by the center mask convolution, and the global features are obtained by using the residual dilation convolution block to remove the dense noise in the dark area; The outputs of the illumination mask branch and the two center mask branches are fused to output a denoised image that retains texture details in bright areas.

5. The collaborative unsupervised image enhancement and denoising method according to claim 4, characterized in that: The method of using the illumination driven iterative enhancement network and the region-based denoising network to generate a collaborative unsupervised framework includes: The illumination-driven iterative enhancement network and the region-based denoising network are integrated to generate a collaborative unsupervised framework for performing image enhancement and image denoising processes. The image enhancement process includes: Based on the iterative framework of Retinex theory, the illumination optimization module is used to improve the image brightness, the illumination estimation module is used to generate a linear enhanced image, and the illumination calibration module is used to adjust the overexposure through the illumination characteristics to generate an enhanced image; The image denoising process includes: The illumination map of the enhanced image is converted into an illumination mask map to protect the pixels in the bright area, and a denoised image that retains the texture details of the bright area is output on the basis of removing the global noise.

6. The collaborative unsupervised image enhancement and denoising method according to claim 5, characterized in that: The iterative process of the illumination-driven iterative enhancement network is expressed by the following formula: in, Represents low-light images, Represents the illumination component generated by the illumination optimization module, Represents the reflection component obtained according to the Retinex theory, Indicates that the illumination feature is obtained through the illumination prediction module; Represents the illumination calibration module, Indicates that the illumination calibration module outputs the calibration component; is the input for the next iteration. is the number of iterations.

7. The collaborative unsupervised image enhancement and denoising method according to claim 6, characterized in that: The collaborative unsupervised framework is trained based on the training data set using a comprehensive loss function containing prior information and an Adam optimizer, including: Import the training dataset and use the Adam optimizer to train the collaborative unsupervised framework; The initial learning rate is set to 0.0001, and a comprehensive loss function combining fidelity loss, smoothness loss and L1 regularization is used as the output of the training loss constraint iterative process. After the training is completed, the illumination-driven iterative enhancement network is frozen; Use L1 regularization as the training loss to constrain the denoising output to train the sub-region denoising network. When the loss is stable, save the weights of the illumination-driven iterative enhancement network and the sub-region denoising network and complete the training. The comprehensive loss function L is as follows: in, is the fidelity loss function, is the smoothness loss function, is the weight of the fidelity loss function, is the weight of the slip loss function, i and j represent the i-th pixel and the j-th pixel respectively. express The neighboring pixels within its 5×5 window, represents the associated weight; is the constraint function of lighting; represents the output function after optimization of the illumination calibration module, is its weight; is the loss function of linear enhanced image and refinement enhanced image, is its weight; Represents a linearly enhanced image.

8. A collaborative unsupervised image enhancement and denoising system, characterized in that: The system adopts the collaborative unsupervised image enhancement and denoising method as claimed in any one of claims 1 to 7; The system comprises: A data set construction module is configured to construct a data set of low-light images based on an image low-light enhancement data set, wherein the data set includes a training data set and a test data set; An enhancement network architecture construction module configured to construct an illumination-driven iterative enhancement network to utilize illumination features to increase image brightness and adjust exposure to generate an enhanced image; A denoising network architecture construction module configured to construct a sub-region denoising network to remove dense noise and retain semantic information of bright areas using a hybrid denoising module; the hybrid denoising module includes an illumination mask branch and a center mask branch; A network generation module configured to generate a collaborative unsupervised framework using an illumination-driven iterative enhancement network and a region-based denoising network; A network training module configured to train a collaborative unsupervised framework based on a training dataset using a comprehensive loss function containing prior information and an Adam optimizer; A network testing module, configured to use a test data set to obtain final image enhancement and denoising results through the collaborative unsupervised framework to complete the testing of the collaborative unsupervised framework; The enhancement and denoising processing module is configured to collect low-light images and perform image enhancement and denoising using a tested collaborative unsupervised framework.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the collaborative unsupervised image enhancement and denoising method as described in any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the collaborative unsupervised image enhancement and denoising method according to any one of claims 1 to 7 are implemented.

Images