Lightweight low-light image enhancement method and device based on deep learning

Abstract

The invention discloses a lightweight low-light image enhancement method and device based on deep learning, and the method comprises the steps: inputting a low-light image into a trained lightweight low-light image enhancement network, and obtaining an enhanced low-light image; wherein the lightweight low-light image enhancement network comprises a feature extraction module and an illumination enhancement module; the feature extraction module comprises a multi-dimensional feature extraction module and a multi-scale convolution module; the multi-dimensional feature extraction module is used for extracting features of the low-light image in multiple dimensions to obtain a first fusion feature of the low-light image, and the multi-scale convolution module is used for extracting a multi-scale feature of the low-light image; and the illumination enhancement module is used for performing illumination enhancement on the low-light image according to a second fusion feature to obtain an enhanced low-light image, wherein the second fusion feature is obtained by fusing the first fusion feature and the multi-scale feature. The network structure used by the method is simpler, and the calculation amount and the cost during enhancement are smaller.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a lightweight low-light image enhancement method and device based on deep learning. Background Art

[0002] With the continuous development of technology, images have been widely used in fields such as object detection, medical image processing, and monitoring systems. However, due to the limitations of hardware devices or lighting environments, the acquired images often have problems such as loss of detail information, dark brightness, high noise, and color distortion. These low-quality illuminated images not only have an adverse impact on the human visual experience, but may also transmit some incorrect information to many advanced computer vision tasks, reducing their performance.

[0003] Low-light image enhancement technology can effectively improve the brightness and clarity of images in low-light environments, improve image details, maintain color balance, and improve image quality. It is usually used in the data preprocessing stage of some advanced vision tasks to help downstream advanced vision tasks achieve better performance. Therefore, low-light image enhancement has important research value and commercial value.

[0004] The currently most popular low-light image enhancement method is the low-light image enhancement method based on deep learning. With the continuous development and innovation of deep learning networks, this method has achieved remarkable results. However, with the improvement of the enhancement effect, the network structure has become more and more complex, resulting in a huge number of parameters and high computational complexity. The high computational cost makes it difficult to deploy these models on devices with limited resources, thus restricting the application of low-light image enhancement technology.

[0005] Therefore, the model structures used in current low-light image enhancement methods are relatively complex and costly. Summary of the Invention

[0006] Embodiments of the present invention provide a lightweight low-light image enhancement method and device based on deep learning, which can solve the problems that the model structures used in current low-light image enhancement methods are relatively complex and costly.

[0007] In a first aspect, a lightweight low-light image enhancement method based on deep learning provided by an embodiment of the present invention includes:

[0008] Inputting a low-light image into a trained lightweight low-light image enhancement network to obtain an enhanced low-light image;

[0009] Among them, the lightweight low-light image enhancement network includes a feature extraction module and a lighting enhancement module; the feature extraction module includes a multi-dimensional feature extraction module and a multi-scale convolution module; the multi-dimensional feature extraction module is used to extract features of multiple dimensions of the low-light image to obtain the first fusion feature of the low-light image, and the multi-scale convolution module is used to extract multi-scale features of the low-light image; the lighting enhancement module is used to perform lighting enhancement on the low-light image according to the second fusion feature to obtain the enhanced low-light image, and the second fusion feature is obtained by fusing the first fusion feature and the multi-scale feature.

[0010] In a second aspect, an embodiment of the present invention provides a lightweight low-light image enhancement device based on deep learning. The device includes a processing unit; the processing unit is configured to:

[0011] Input the low-light image into the trained lightweight low-light image enhancement network to obtain the enhanced low-light image;

[0012] Among them, the lightweight low-light image enhancement network includes a feature extraction module and a lighting enhancement module; the feature extraction module includes a multi-dimensional feature extraction module and a multi-scale convolution module; the multi-dimensional feature extraction module is used to extract features of multiple dimensions of the low-light image to obtain the first fusion feature of the low-light image, and the multi-scale convolution module is used to extract multi-scale features of the low-light image; the lighting enhancement module is used to perform lighting enhancement on the low-light image according to the second fusion feature to obtain the enhanced low-light image, and the second fusion feature is obtained by fusing the first fusion feature and the multi-scale feature.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory. Among them, the memory is used to store a computer program; the processor can be used to execute the calculator program (instructions) stored in the memory to implement the method of the first aspect above.

[0014] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, and a computer program is stored in the computer-readable storage medium. When the computer program is executed, the method of the first aspect above can be implemented.

[0015] It can be understood that the beneficial effects of the second to fourth aspects above can refer to the relevant descriptions of the first aspect above, and will not be elaborated here.

[0016] The beneficial effect of the embodiment of the present invention compared with the prior art is that: according to the image enhancement method provided by the present invention, by using a lightweight low-light image enhancement network with a simpler structure to perform enhancement processing on the low-light image, the amount of calculation during image enhancement can be reduced, thereby reducing the usage cost. Description of the Drawings

[0017] Figure 1Schematic diagram of the structure of a lightweight low-light image enhancement network provided by an embodiment of the present invention;

[0018] Figure 2 Scene diagram of enhancing an input image by a lightweight low-light image enhancement network provided by an embodiment of the present invention;

[0019] Figure 3 Schematic diagram of the structure of a multi-scale convolution module provided by an embodiment of the present invention;

[0020] Figure 4 Flowchart of a training method for a lightweight low-light image enhancement network provided by an embodiment of the present invention;

[0021] Figure 5 Implementation flowchart of a lightweight low-light image enhancement method based on deep learning provided by an embodiment of the present invention;

[0022] Figure 6 Schematic diagram of the structure of a lightweight low-light image enhancement device based on deep learning provided by an embodiment of the present invention;

[0023] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0024] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present invention. However, those skilled in the art should clearly understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present invention.

[0025] It should be understood that when used in the specification and the appended claims of the present invention, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0026] It should also be understood that the term " / and" as used in the specification and the appended claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] As used in the specification of the present invention and the appended claims, the term "if" may be construed, depending on the context, as "when", "once", "in response to determining", or "in response to detecting". Similarly, the phrases "if determined" or "if [the described condition or event] is detected" may be construed, depending on the context, to mean "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".

[0028] In addition, in the description of the specification of the present invention and the appended claims, the terms "first", "second", "third", etc. are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0029] Reference to "one embodiment" or "some embodiments" or the like described in the specification of the present invention means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present invention. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0030] The present invention will be further described in detail below in conjunction with specific embodiments, but the implementation manners of the present invention are not limited thereto.

[0031] Embodiment 1

[0032] Figure 1 The following shows a schematic structural diagram of a lightweight low-light image enhancement network provided by an embodiment of the present invention. By way of example and not limitation, referring to Figure 1 , the network 100 may include a feature extraction module 110 and a lighting enhancement module 120. The feature extraction module 110 may include a multi-dimensional feature extraction module 111 and a multi-scale convolution module 112.

[0033] In some embodiments, the network 100 may input an input image into the feature extraction module 110 to make it extract the second fusion feature of the input image, and then input the second fusion feature and the input image into the lighting enhancement module 120, so that the lighting enhancement module 120 enhances the lighting of the input image according to the second fusion feature to obtain the enhanced input image.

[0034] Exemplarily, the input image may be a sample low-light image or a low-light image to be enhanced in actual use.

[0035] In a possible implementation, refer to Figure 1 , the feature extraction module 110 may input the input image into the multi-dimensional feature extraction module 111 and the multi-scale convolution module 112 respectively, and let the multi-dimensional feature extraction module 111 extract the features of multiple dimensions of the input image to obtain the first fusion feature of the input image; let the multi-scale convolution module 112 extract the features of multiple scales of the input image to obtain the multi-scale feature of the input image, and then fuse the first fusion feature and the multi-scale feature to obtain the second fusion feature of the input image.

[0036] In an example, refer to Figure 2 , the multi-dimensional feature extraction module 111 may include two feature extraction channels, one is the spatial channel reconstruction convolution module 1111, and the other is the attention module 1112. Before the two feature extraction channels, multiple first convolutional layers and at least one pooling layer may also be provided, and at least one second convolutional layer and multiple interpolation layers may also be provided after the two feature extraction channels.

[0037] For example, refer to Figure 2 , the multi-dimensional feature extraction module 111 may be provided with 2 first convolutional layers, 1 pooling layer, 1 second convolutional layer and 2 interpolation layers.

[0038] Exemplarily, the multi-dimensional feature extraction module 111 may perform preliminary compression processing on the input image through the first convolutional layer, and perform further compression through the pooling layer to obtain the reduced input image. Then the reduced input image is respectively input into the spatial channel reconstruction convolution module 1111 and the attention module 1112. The spatial channel reconstruction convolution module 1111 extracts the features of the spatial dimension and the channel dimension of the input image to obtain the two-dimensional feature of the input image, and the attention module 1112 extracts the global feature of the input image. Then the two-dimensional features of the input image are fused and added, and the channel number of the fused features is adjusted through the second convolutional layer, and the fused features are enlarged to the original size of the input image through the interpolation layer to obtain the first fusion feature of the input image.

[0039] For example, the sizes of the first convolutional layer and the second convolutional layer may both be 3×3, and the size of the input image may be 256×256. The first convolutional layer may compress the size of the input image to 64×64, and the pooling layer further compresses the input image to 32×32. The interpolation layer finally adjusts the size of the first fusion feature to 256×256.

[0040] Compressing the input image before processing can reduce the computational complexity of the feature extraction process and reduce the network parameters required by the spatial channel reconstruction convolution module 1111 and the attention module 1112.

[0041] Optionally, the spatial channel reconstruction convolution module 1111 can be an SCConv (Spatial and Channel reconstruction Convolution) network, and the attention module 1112 can be a Transformer network. The pooling layer can be an adaptive average pooling layer.

[0042] Exemplarily, the process of the multi-dimensional feature extraction module 111 extracting the first fusion feature can be represented by the following formula:

[0043] F1 = ReLU(Conv1(F)) (1.1)

[0044] F2 = ReLU(Conv1(F1)) (1.2)

[0045] G = AdaptiveAvgPool2d(F2) (1.3)

[0046] T = MLP(LN(MSA(LN(G) + G)) + MSA(LN(G)) + G (1.4)

[0047] F out1 = Bilinear(Bilinear(Conv1(Conv2(G) + T))) (1.5)

[0048] Among them, F1 is the output of the first first convolution layer. Conv1 in formulas (1.1) and (1.2) represents the first convolution layer, F is the input image, ReLU is the activation function; F2 is the output of the second first convolution layer; G is the reduced input image, AdaptiveAvgPool2d represents the adaptive average pooling operation; T is the global feature, MLP represents the multi-layer perceptron, LN represents the normalization layer within the attention module, MSA represents the multi-head attention mechanism; F out1 is the first fusion feature of the input image, Bilinear represents the bilinear interpolation operation, Conv2 is the convolution layer in the spatial channel reconstruction convolution module 1111, and Conv1 in formula (1.5) represents the second convolution layer.

[0049] In one example, refer to Figure 3, the multi-scale convolution module 112 may include multiple convolution channels (e.g., 3), and a depthwise separable convolution layer and a regular convolution layer are sequentially arranged in each convolution channel. The sizes of the depthwise separable convolution layers in each channel are different. By combining the depthwise separable convolution layers with different scales in each channel and the ReLU activation function, features of different scales of the input image can be extracted. Through the regular convolution layer combined with the ReLU activation function, the features output by the depthwise separable convolution layer can be integrated and output. Finally, the multi-scale convolution module 112 performs a splicing process on the multiple features with different scales output by each convolution channel to obtain the multi-scale features of the input image.

[0050] For example, see Figure 3 , the sizes of the depthwise separable convolution layers in the 3 convolution channels in the multi-scale convolution module 112 can be 3×3, 5×5, and 7×7 respectively. The sizes of the regular convolution layers can be the same, all being 1×1.

[0051] Extracting the multi-scale features of the input image through the multi-scale convolution module 112 can enrich the information extracted by the network 100, so that the features lost during the sampling process can be supplemented to a certain extent. Using the multi-scale convolution module 112 to extract the multi-scale features of the input image can not only enable the network 100 to obtain richer and more diverse feature expressions; compared with the traditional CNN convolution module, the structure of the multi-scale convolution module 112 is simpler, and the number of parameters and the amount of calculation during feature extraction are less.

[0052] In a possible implementation manner, the illumination enhancement module 120 can segment the second fused feature to obtain multiple segmented features of the input image; then, based on the illumination enhancement curve, the input image is iteratively enhanced according to the multiple segmented features to obtain the enhanced input image.

[0053] In one example, the illumination enhancement curve can adjust the image within the dynamic range and automatically map the input image to its enhanced version.

[0054] Exemplarily, taking the input image as the low-light image to be enhanced in the actual process, the enhancement process of the illumination enhancement curve can be expressed by the following formula:

[0055] LE n (x) = LE n-1 (x) + S n LE n-1 (x)[1 - LE n-1 (x)](1.6)

[0056] Where LE n (x) is the low-light image after the nth illumination enhancement, n is a non-negative integer less than or equal to N, LE0(x) is the original unenhanced low-light image, and Sn is the nth segmented feature of the low-light image.

[0057] For example, if the number of channels of the low-light image is 24, the second fusion feature of it can be divided into 8 parts, that is, N = 8.

[0058] Exemplarily, the nth segmented feature of the low-light image can satisfy the following formula:

[0059] S n = Split(F out1 + F out2 )(1.7)

[0060] where Split represents the splitting operation, F out1 is the first fusion feature of the low-light image here, and F out2 is the multi-scale feature of the low-light image.

[0061] According to the lightweight low-light image enhancement network provided by the present invention, the multi-dimensional feature extraction module is used to simultaneously extract features of multiple dimensions of the low-light image to obtain the first fusion feature, and the multi-scale convolution module is used to extract the multi-scale feature of the low-light image. Finally, these two features are fused to enhance the low-light image. Compared with the current deep network that uses multiple modules to separately extract different types of features, the lightweight low-light image enhancement network provided by the present invention has a simpler structure, can reduce the computational amount and complexity during feature extraction, and thus reduce the usage cost.

[0062] Embodiment 2

[0063] The training method of the lightweight low-light image enhancement network provided by the embodiment of the present invention can be applied to electronic devices such as mobile terminals, personal laptop computers, supercomputers, etc. The embodiment of the present invention does not impose any restrictions on the specific type of the electronic device.

[0064] Figure 4 The flowchart of a training method of a lightweight low-light image enhancement network provided by the embodiment of the present invention is shown. By way of example and not limitation, this training method can be applied to the above-mentioned electronic devices, and this training method can be used to train the above-mentioned network 100. It may include steps S401 - S405, which will be described below.

[0065] S401, input the kth group of sample low-light images into the lightweight low-light image enhancement network after the (k - 1)th round of training to obtain the enhanced kth group of sample low-light images.

[0066] Exemplarily, k is a positive integer, and the network parameters of the lightweight low-light image enhancement network after the 0th round of training are preset initial parameters.

[0067] S402. Based on the total loss function, determine the k-th total loss of the lightweight low-light image enhancement network according to the enhanced k-th group of sample low-light images and the real enhanced images of the k-th group of sample low-light images.

[0068] In a possible implementation, the total loss function can be a weighted sum of the L1 loss function and the frequency reconstruction loss function.

[0069] Exemplarily, the total loss function can satisfy the following formula:

[0070] L total = L1 + αL FR (1.8)

[0071] where, L total represents the total loss function, L1 represents the L1 loss function, L FR represents the frequency reconstruction loss function, and α is its weight, generally taking 0.009.

[0072] In an example, the L1 loss function is also known as the Mean Absolute Error (MAE), and can be used to measure the difference between the predicted value and the true value of the network 100, and can accurately reflect the size of the actual prediction error.

[0073] Exemplarily, the L1 loss function can satisfy the following formula:

[0074]

[0075] where, T is the total number of the k-th group of sample low-light images, X t is the t-th enhanced sample low-light image in the k-th group, and I t is the real enhanced image corresponding to the t-th sample low-light image.

[0076] In an example, in the low-light image enhancement task, using frequency domain information is very important. Frequency domain information can help capture the details and textures of the image. Through frequency domain processing, the image details can be effectively restored and enhanced to achieve a clearer and more natural enhancement effect. However, the L1 loss function mainly operates in the spatial domain, focusing on the error at each pixel position in the spatial domain and ignoring the information operation in the frequency domain. To better capture the frequency characteristics of the image and reduce the difference in the frequency space, a frequency reconstruction loss function (frequency reconstruction, MSFR) can be introduced. The frequency reconstruction loss function can measure the L1 distance between the multi-scale real image and the enhanced low-light image in the frequency domain.

[0077] Exemplarily, the frequency reconstruction loss function can satisfy the following formula:

[0078]

[0079] Among them, FFT represents the Fast Fourier Transform.

[0080] S403, determine whether the training stop condition is reached.

[0081] In one example, if the training stop condition is reached, step S405 can be performed.

[0082] Exemplarily, the training stop condition can be that k is greater than or equal to the maximum number of training times, or that the k-th total loss is less than or equal to the convergence threshold.

[0083] In another example, if the training stop condition is not reached, step S404 can be performed, then let k = k + 1, and continue the iteration.

[0084] S404, based on the backpropagation algorithm, perform the k-th round of training on the lightweight low-light image enhancement network according to the k-th total loss to obtain the lightweight low-light image enhancement network after the k-th round of training.

[0085] Exemplarily, the network 100 can be trained on a graphics card of model NVIDIA GeForce RTX 4090, based on the PyTorch framework, using the Adam optimizer.

[0086] Specifically, the initial learning rate can be set to 0.001, the weight decay parameter to 0.0001, and the network parameters of the network 100 can be gradually adjusted through the cosine annealing strategy. And the sample low-light images are uniformly cropped to 256×256 pixels, the batch size of each round of training is set to 8, and the maximum number of training times is 100.

[0087] S405, output the lightweight low-light image enhancement network after the (k - 1)-th round of training as the trained lightweight low-light image enhancement network.

[0088] The training method provided by the present invention uses the total loss function composed of the L1 loss function and the frequency reconstruction loss function to train the lightweight low-light image enhancement network, which can enable the trained lightweight low-light image enhancement network to have better ability to restore and enhance image details; at the same time, training based on such a total loss function does not increase the number of parameters and the amount of calculation of the lightweight low-light image enhancement network.

[0089] Embodiment 3

[0090] Figure 5The figure shows a flowchart of the implementation of a lightweight low-light image enhancement method based on deep learning provided by an embodiment of the present invention. By way of example and not limitation, this image enhancement method can also be applied to the above-mentioned electronic device. It may include steps S501 - S502, and the following is an explanation of each step.

[0091] S501, obtain a low-light image.

[0092] S502, input the low-light image into the trained lightweight low-light image enhancement network to obtain the enhanced low-light image.

[0093] Exemplarily, the trained lightweight low-light image enhancement network can be network 100 trained based on the above training method.

[0094] According to the image enhancement method provided by the present invention, by using a lightweight low-light image enhancement network with a simpler structure to enhance the low-light image, the amount of computation during image enhancement can be reduced, thereby reducing the usage cost.

[0095] Embodiment 4

[0096] Figure 6 The figure shows a schematic structural diagram of a lightweight low-light image enhancement device provided by an embodiment of the present invention. By way of example and not limitation, device 600 may include a processing unit 610, and a lightweight low-light image enhancement network 100 may be provided in the processing unit 610.

[0097] Exemplarily, the processing unit may be used to input the low-light image into the trained lightweight low-light image enhancement network to obtain the enhanced low-light image.

[0098] According to the image enhancement device provided by the present invention, by using a lightweight low-light image enhancement network with a simpler structure to enhance the low-light image, the amount of computation during image enhancement can be reduced, thereby reducing the usage cost.

[0099] Embodiment 5

[0100] Figure 7 The figure shows a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As Figure 7 shown, the electronic device 700 may include: at least one processor 710 ( Figure 7 only one processor is shown herein), a memory 720, and a computer program 730 stored in the memory 720 and executable on the at least one processor 710. When the processor 710 executes the computer program 730, the steps in any of the above method embodiments are implemented.

[0101] The electronic device 700 may be a processing device such as a robot that can implement the above method. The embodiments of the present invention do not impose any restrictions on the specific type of the electronic device.

[0102] Those skilled in the art can understand that Figure 7 merely examples of the electronic device 700, which do not constitute a limitation on the electronic device, may include more or fewer components than shown in the figure, or combine some components, or different components. For example, the electronic device 700 may further include an input / output interface.

[0103] The so-called processor 710 may be a central processing unit (CPU), and the processor 710 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASTCs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0104] The memory 720 may be an internal storage unit in some embodiments, such as a hard disk or memory. The memory 720 may also be an external storage device in other embodiments, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 720 may also include both an internal storage unit and an external storage device. The memory 720 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program, etc. The memory 720 may also be used to temporarily store data that has been output or will be output.

[0105] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution is prior or posterior. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present invention. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0107] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.

[0108] An embodiment of the present invention provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the foregoing method embodiments when executed.

[0109] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of the present invention, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium that can carry the computer program code to the photographing device / terminal device. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0110] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not described in detail or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0111] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

Claims

1. A lightweight low-light image enhancement method based on deep learning, characterized in that: include: Input the low-light image into the trained lightweight low-light image enhancement network to obtain an enhanced low-light image; Among them, the lightweight low-light image enhancement network includes a feature extraction module and an illumination enhancement module; the feature extraction module includes a multi-dimensional feature extraction module and a multi-scale convolution module; the multi-dimensional feature extraction module is used to extract the features of multiple dimensions of the low-light image to obtain a first fusion feature, and the multi-scale convolution module is used to extract the multi-scale features of the low-light image; the illumination enhancement module is used to perform illumination enhancement on the low-light image according to a second fusion feature to obtain the enhanced low-light image, and the second fusion feature is obtained by fusing the first fusion feature and the multi-scale feature.

2. The method according to claim 1, characterized in that The multi-dimensional feature extraction module includes a spatial channel reconstruction convolution module and an attention module; the multi-dimensional feature extraction module is used to: compressing the low-light image to obtain a reduced low-light image; Inputting the reduced low-light image into the spatial channel reconstruction convolution module to extract features of the spatial dimension and channel dimension of the low-light image to obtain two-dimensional features; inputting the reduced low-light image into the attention module to extract global features of the low-light image; The two-dimensional feature and the global feature are fused to obtain the first fused feature.

3. The method according to claim 1, characterized in that: The multi-scale convolution module includes multiple convolution channels, each of which includes a depth-separable convolution layer and a common convolution layer, and the sizes of the depth-separable convolution layers in different channels are different.

4. The method according to claim 3, characterized in that The multi-scale convolution module is used to: Inputting the low-light image into different convolution channels to extract features of the low-light image at multiple scales; The multiple features at different scales are spliced ​​to obtain the multi-scale feature.

5. The method according to claim 1, characterized in that The illumination enhancement module is specifically used for: Segmenting the second fused feature to obtain a plurality of segmented features; Based on the illumination enhancement curve, the low-light image is iteratively enhanced according to the multiple segmented features to obtain the enhanced low-light image.

6. The method according to claim 5, characterized in that The illumination enhancement curve satisfies the following formula: THE n (x)=THE n-1 (x)+S n THE n-1 (x)[1-THE n-1 (x)] Among them, LE n (x) is the low-light image after the nth illumination enhancement, n is a non-negative integer, LE0(x) is the low-light image, S n is the feature after the nth segmentation.

7. The method according to claim 1, characterized in that The total loss function used for training the lightweight low-light image enhancement network is a weighted sum of the L1 loss function and the frequency reconstruction loss function.

8. A lightweight low-light image enhancement device based on deep learning, characterized in that: The apparatus comprises a processing unit, wherein the processing unit is configured to: Input the low-light image into the trained lightweight low-light image enhancement network to obtain an enhanced low-light image; Among them, the lightweight low-light image enhancement network includes a feature extraction module and an illumination enhancement module; the feature extraction module includes a multi-dimensional feature extraction module and a multi-scale convolution module; the multi-dimensional feature extraction module is used to extract the features of multiple dimensions of the low-light image to obtain the first fusion feature of the low-light image, and the multi-scale convolution module is used to extract the multi-scale features of the low-light image; the illumination enhancement module is used to perform illumination enhancement on the low-light image according to the second fusion feature to obtain the enhanced low-light image, and the second fusion feature is obtained by fusing the first fusion feature and the multi-scale feature.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the electronic device, the method according to any one of claims 1 to 7 is implemented.

Images