Lightweight high-resolution dark-light image enhancement method, system, device and medium

Abstract

The present invention discloses a lightweight high-resolution dark-light image enhancement method, system, device and medium, which sequentially performs point-line conversion, waveform reconstruction, spatiotemporal transformation optimization processing, edge texture feature extraction and image reconstruction on each pixel point data of a low-resolution dark-light image obtained by downsampling a high-resolution dark-light image to obtain a low-resolution enhanced image; performs mean filtering processing and local linear processing on the low-resolution dark-light image and the low-resolution enhanced image to obtain the first linear coefficient and the first bias item of each pixel point; performs upsampling on the low-resolution parameter feature map obtained by the imaging processing of the first linear coefficient and the first bias item to obtain a high-resolution parameter feature map; performs global linear processing on the high-resolution dark-light image according to the high-resolution parameter feature map to obtain a high-resolution enhanced image. The method provided by the present invention can perform dark-light enhancement processing on the entire image without brightness splitting and better visualization effect.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a lightweight high-resolution dark-light image enhancement method, system, device and medium. Background Art

[0002] Dark-light image enhancement aims to solve the problem of low image visibility and poor contrast caused by factors such as insufficient shooting light and limited exposure time. Traditional dark-light image enhancement algorithms, such as histogram equalization and Retinex algorithm, can simply enhance the brightness and contrast of images, but they are not effective in suppressing noise and color correction.

[0003] In addition, when processing ultra-high-resolution images, existing low-light image enhancement algorithms first apply a cutting strategy to divide the high-resolution image into a group of sub-images, process the sub-images one by one, and then stitch them into a complete enhanced image in a specific spatial order. However, applying the cutting strategy for image enhancement will result in obvious brightness splitting at the boundaries of the cutting blocks in the enhanced image, which is not a satisfactory subjective perception effect for the human eye.

[0004] It can be seen that how to solve the poor human eye perception effect of the existing dark light image enhancement processing has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] The present invention provides a lightweight high-resolution dark-light image enhancement method, system, device and medium to solve the technical problem of poor human eye perception effect in the existing dark-light image enhancement processing, and to improve the human eye perception effect of the image after the dark-light image enhancement processing.

[0006] In a first aspect, the present invention provides a lightweight high-resolution dark-light image enhancement method, the method comprising: downsampling the high-resolution dark-light image to be enhanced to obtain a low-resolution dark-light image;

[0007] Performing point-line conversion on each pixel point data of the low-resolution dark light image, and reconstructing a waveform graph according to the point-line converted data;

[0008] Performing time-space domain transformation optimization processing on the reconstructed waveform diagram to obtain optimized waveform data;

[0009] Reconstructing the edge texture features extracted from the waveform data to obtain a low-resolution enhanced image;

[0010] Performing mean filtering on the low-resolution dark-light image and the low-resolution enhanced image to obtain a variance feature map and a covariance feature map between the low-resolution dark-light image and the low-resolution enhanced image;

[0011] Performing splicing processing on the variance feature map and the covariance feature map, wherein the splicing process is designed to merge the variance feature map and the covariance feature map on the corresponding channel dimension in the convolution layer of the convolutional neural network;

[0012] Performing local linear processing on the splicing processing result to obtain a first linear coefficient and a first bias term of each pixel point;

[0013] Performing image processing on the first linear coefficient and the first bias item of each pixel point to obtain a low-resolution parameter feature map;

[0014] Upsampling the low-resolution parameter feature map to obtain a high-resolution parameter feature map;

[0015] The high-resolution dark-light image is globally linearly processed according to the high-resolution parameter feature map to obtain a high-resolution enhanced image.

[0016] Preferably, performing point-to-line conversion on each pixel point data of the low-resolution dark light image, and reconstructing a waveform according to the point-to-line converted data, comprises:

[0017] Extracting shallow features of the low-resolution dark-light image to obtain a shallow feature map;

[0018] Extracting deep features of the shallow feature map to obtain a first deep feature map;

[0019] Performing nonlinear mapping on the first deep feature map to obtain a second deep feature map;

[0020] Performing a fast Fourier transform on the second deep feature map to perform point-to-line conversion on each pixel point data of the second deep feature map to obtain the amplitude and phase of the second deep feature map;

[0021] A waveform diagram is reconstructed for the low-resolution dark-light image according to the amplitude and the phase.

[0022] Preferably, the step of performing time-space domain transformation optimization processing on the reconstructed waveform graph to obtain optimized waveform data includes:

[0023] Performing nonlinear transformation on the amplitude and the phase respectively to obtain amplitude characteristics and phase characteristics;

[0024] Performing inverse Fourier transform on the amplitude feature and the phase feature to obtain a Fourier transform feature graph;

[0025] Performing a nonlinear transformation on the second deep feature map to obtain a spatial transformation feature map;

[0026] The Fourier transform feature map and the spatial transform feature map are fused in channel dimension to obtain optimized waveform data.

[0027] Preferably, the image reconstruction of the edge texture features extracted from the waveform data to obtain a low-resolution enhanced image includes:

[0028] Reconstructing the brightness image residual on the edge texture features extracted from the waveform data to obtain the brightness image residual;

[0029] The brightness image residual is added to the low-resolution dark light image to obtain a low-resolution enhanced image.

[0030] Preferably, the low-resolution dark-light image and the low-resolution enhanced image are subjected to mean filtering to obtain a variance feature map and a covariance feature map between the low-resolution dark-light image and the low-resolution enhanced image, including:

[0031] Performing a first mean filtering process on the low-resolution dark-light image and the low-resolution enhanced image to obtain a low-resolution dark-light image mean feature map and a low-resolution enhanced image mean feature map;

[0032] A second mean filtering process is performed on the low-resolution dark-light image, the low-resolution enhanced image, the low-resolution dark-light image mean feature map and the low-resolution enhanced image mean feature map to obtain the covariance feature map and the variance feature map.

[0033] Preferably, the step of performing local linear processing on the splicing processing result to obtain a first linear coefficient and a first bias term includes:

[0034] Perform two nonlinear transformations on the splicing results to obtain the feature map to be compressed;

[0035] Compressing the feature map to be compressed to obtain a 3-channel feature map in RGB format;

[0036] Perform a linear operation on the 3-channel feature map to obtain the first linear coefficient and the first bias term.

[0037] Preferably, performing global linear processing on the high-resolution dark-light image according to the high-resolution parameter feature map to obtain a high-resolution enhanced image comprises:

[0038] Performing global linear processing on the high-resolution dark-light image based on the high-resolution parameter feature map to obtain the high-resolution enhanced image;

[0039] The expression of the high-resolution enhanced image is:

[0040]

[0041] in, represents a high-resolution enhanced image, represents a high-resolution dark-light image, Represents the second linear coefficient corresponding to the high-resolution parameter feature map, Represents the second bias term corresponding to the high-resolution parameter feature map.

[0042] In a second aspect, the present invention further provides a lightweight high-resolution dark light image enhancement system, the system comprising: a downsampling unit, a waveform reconstruction unit, a spatiotemporal transformation optimization processing unit, an image reconstruction unit, a mean filter processing unit, a splicing processing unit, a local linear processing unit, a graphical processing unit, an upsampling unit and a global linear processing unit;

[0043] The down-sampling unit is used to down-sample the high-resolution dark-light image to be enhanced to obtain a low-resolution dark-light image;

[0044] The waveform reconstruction unit is used to perform point-to-line conversion on each pixel point data of the low-resolution dark light image, and to reconstruct the waveform according to the point-to-line converted data;

[0045] The time-space domain transformation optimization processing unit is used to perform time-space domain transformation optimization processing on the reconstructed waveform image to obtain optimized waveform data;

[0046] The image reconstruction unit is used to reconstruct the edge texture features extracted from the waveform data to obtain a low-resolution enhanced image;

[0047] The mean filtering processing unit is used to perform mean filtering on the low-resolution dark-light image and the low-resolution enhanced image to obtain a variance feature map and a covariance feature map between the low-resolution dark-light image and the low-resolution enhanced image;

[0048] The splicing processing unit is used for splicing the variance feature map and the covariance feature map, wherein the splicing process is designed to merge the variance feature map and the covariance feature map on the corresponding channel dimension in the convolution layer of the convolutional neural network;

[0049] The local linear processing unit is used to perform local linear processing on the splicing processing result to obtain a first linear coefficient and a first bias term of each pixel point;

[0050] The graphical processing unit is used to perform graphical processing on the first linear coefficient and the first bias item of each pixel point to obtain a low-resolution parameter feature map;

[0051] The upsampling unit is used to upsample the low-resolution parameter feature map to obtain a high-resolution parameter feature map;

[0052] The global linear processing unit is used to perform global linear processing on the high-resolution dark light image according to the high-resolution parameter feature map to obtain a high-resolution enhanced image.

[0053] In a third aspect, the present invention also provides a computer device, comprising a memory, a processor and a transceiver, which are connected via a bus; the memory is used to store a set of computer program instructions and data, and transmit the stored data to the processor, and the processor executes the program instructions stored in the memory to execute the lightweight high-resolution dark-light image enhancement method described above.

[0054] In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed, the above-mentioned lightweight high-resolution dark-light image enhancement method is implemented.

[0055] The present invention provides a lightweight high-resolution dark light image enhancement method, system, device and medium. Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following:

[0056] (1) There is no bright light splitting phenomenon and the visual effect is good. The lightweight high-resolution dark light image enhancement method provided by the present invention can perform dark light enhancement processing on the entire high-resolution dark light image, solving the problem of brightness splitting and achieving better visualization effect.

[0057] (2) High processing efficiency. There is no need to cut or splice high-resolution dark-light images, which improves the efficiency of dark-light image enhancement processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of the steps of a lightweight high-resolution dark-light image enhancement method provided by a preferred embodiment of the present invention;

[0059] Figure 2 It is a schematic diagram of the structure of a high-resolution dark-light image enhancement model provided by a preferred embodiment of the present invention;

[0060] Figure 3 It is a schematic diagram of the steps of a waveform reconstruction method provided by a preferred embodiment of the present invention;

[0061] Figure 4 is a structural schematic diagram of a dark light enhancement module provided by a preferred embodiment of the present invention;

[0062] Figure 5It is a schematic diagram of the steps of a method for optimizing time-space domain transformation provided by a preferred embodiment of the present invention;

[0063] Figure 6 It is a schematic diagram of the steps of a mean filtering method provided by a preferred embodiment of the present invention;

[0064] Figure 7 It is a schematic diagram of the steps of a method for local linear processing provided by a preferred embodiment of the present invention;

[0065] Figure 8 It is a qualitative analysis comparison diagram provided by a preferred embodiment of the present invention;

[0066] Fig. 9 It is a schematic structural diagram of a lightweight high-resolution dark-light image enhancement system provided by a preferred embodiment of the present invention;

[0067] Fig.10 It is a schematic diagram of a computer device provided by a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following is a detailed explanation of the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided only for illustrative purposes and cannot be understood as limiting the present invention. The accompanying drawings are only for reference and illustration purposes and do not constitute a limitation on the scope of patent protection of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In the description of the present invention, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more.

[0069] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. The terms "vertical", "horizontal", "left", "right", "upper", "lower" and similar expressions used herein are only for illustrative purposes, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0070] In the description of the present invention, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0071] See also Figure 1 In an embodiment of the present invention, a lightweight high-resolution dark-light image enhancement method is provided, the method comprising:

[0072] S1. Down-sample the high-resolution dark-light image to be enhanced to obtain a low-resolution dark-light image.

[0073] S2. Performing point-line conversion on each pixel point data of the low-resolution dark light image, and reconstructing a waveform graph according to the point-line converted data.

[0074] S3, performing time-space domain transformation optimization processing on the reconstructed waveform diagram to obtain optimized waveform data.

[0075] S4. Reconstruct the edge texture features extracted from the waveform data to obtain a low-resolution enhanced image.

[0076] S5. Perform mean filtering on the low-resolution dark-light image and the low-resolution enhanced image to obtain a variance feature map and a covariance feature map between the low-resolution dark-light image and the low-resolution enhanced image.

[0077] S6. Perform splicing processing on the variance feature map and the covariance feature map, wherein the splicing process is designed to merge the variance feature map and the covariance feature map on the corresponding channel dimension in the convolution layer of the convolutional neural network.

[0078] S7. Perform local linear processing on the splicing processing result to obtain a first linear coefficient and a first bias term.

[0079] S8. Performing image processing on the first linear coefficient and the first bias term to obtain a low-resolution parameter feature map.

[0080] S9. Upsample the low-resolution parameter feature map to obtain a high-resolution parameter feature map.

[0081] S10. Perform global linear processing on the high-resolution dark-light image according to the high-resolution parameter feature map to obtain a high-resolution enhanced image.

[0082] In a preferred embodiment of the present invention, Figure 2 As shown, an end-to-end high-resolution dark-light image enhancement model is designed, which includes a dark-light image enhancement module and an upsampling module based on guided filtering. An image dataset for training the high-resolution dark-light image enhancement model is collected, and the image dataset should include high-resolution dark-light images and corresponding high-resolution bright-light images.

[0083] In this application, an image dataset is collected by downloading the public dataset UHD-4K. The UHD-4K dataset is a synthetic dataset consisting of 5999 pairs of 4K images with a resolution of 3840×2160, each pair containing a high-resolution dark-light image and a corresponding high-resolution bright-light image. The high-resolution dark-light image is used as the input image for training the high-resolution dark-light image enhancement model, while the high-resolution bright-light image is used as the reference image for comparison with the high-resolution enhanced image predicted by the high-resolution dark-light image enhancement model during the training process.

[0084] In the embodiment of the present application, the dark light image enhancement module and the upsampling module based on guided filtering of the high-resolution dark light image enhancement model are both provided with a loss function, and the loss function of the high-resolution dark light image enhancement model is formed by adding the loss functions of the two modules. The loss functions of the two modules are both L1 loss functions, and the output image of the dark light image enhancement module is reduced by using the back propagation of deep learning. With low resolution bright light images The gap between the output image and the upsampling module based on guided filtering With high resolution bright light images The gap between the low-resolution and bright-light images can be optimized. High-resolution bright light images The loss function of the high-resolution dark light image enhancement model is expressed as:

[0085] L=L1( , )+L1( , )

[0086] in, represents a low-resolution enhanced image, Represents a high-resolution enhanced image.

[0087] The image data set is used to train the high-resolution dark-light image enhancement model to obtain a trained high-resolution dark-light image enhancement model. The trained high-resolution dark-light image enhancement model is used to perform enhancement processing on the high-resolution dark-light image to be enhanced.

[0088] The process of high-resolution dark-light image enhancement model processing high-resolution dark-light images is as follows:

[0089] The high-resolution dark-light image to be enhanced is scaled to a low-resolution image of a fixed size using a bilinear interpolation algorithm to obtain a low-resolution dark-light image. The low-resolution dark-light image has less data volume and lower computational complexity, making the subsequent image processing process faster and more efficient.

[0090] The low-resolution dark light image is used as the input of the dark light enhancement module, and the dark light enhancement module is used to perform brightness restoration on the low-resolution dark light image to obtain a low-resolution enhanced image. The dark light enhancement image can realize the processing of the complete image, ensuring that the final high-resolution enhanced image has uniform brightness.

[0091] In a preferred embodiment of the present invention, first, the dark light enhancement module performs point-to-line conversion on each pixel point data of the low-resolution dark light image, and reconstructs a waveform according to the point-to-line converted data, such as Figure 3 As shown, the following steps are included:

[0092] S201. Extract shallow features of the low-resolution dark-light image to obtain a shallow feature map.

[0093] S202: Extract deep features of the shallow feature map to obtain a first deep feature map.

[0094] S203: Perform nonlinear mapping on the first deep feature map to obtain a second deep feature map.

[0095] S204: Perform fast Fourier transform on the second deep feature map to perform point-to-line conversion on each pixel point data of the second deep feature map to obtain the amplitude and phase of the second deep feature map.

[0096] S205 , reconstructing a waveform of the low-resolution dark-light image according to the amplitude and the phase.

[0097] like Figure 4 As shown, the dark light enhancement module includes a first convolution layer, the number of convolution kernels of the first convolution layer is C, and the convolution kernel size is 3*3. The first convolution layer is used to extract shallow features of the low-resolution dark light image to obtain a shallow feature map.

[0098] Shallow feature maps usually have a higher resolution and contain a large number of fine-grained features, such as color, texture, edges, and corners, which can more finely represent the details of the image. Since shallow feature maps mainly focus on the details and local information of the image, they have a certain tolerance for noise and interference in the image. In the case of low image quality or noise, shallow feature maps can still extract useful information and are suitable for extracting features of low-resolution dark light images.

[0099] The dark light enhancement module also includes a three-layer U-net, which contains two encoders (E0 and E1) and three decoders (D2, D1 and D0), and each encoder and decoder contains a convolution block with the same structure, namely the second convolution block, the third convolution block, the fourth convolution block, the fifth convolution block and the sixth convolution block. A pixel unshuffle layer is inserted at the output of each encoder to downsample the feature map, and the resolution of the shallow feature map is reduced in turn without losing the input information, thereby reducing the computational cost. A pixel shuffle layer is set at the output of the D2 and D1 decoders for upsampling the feature map, so that the shallow feature map is gradually restored to the original resolution. In order to retain the original structural information of the image as much as possible, the output of the decoder is upsampled, and then channel-joined with the output of the corresponding encoder with the same resolution size, and then sent to the next decoder for processing, so as to take into account the decoder features and the encoder features. Specifically, the shallow feature map is input into the second convolution block for encoding processing to obtain the first feature map to be processed. After downsampling the first feature map to be processed, it is input into the third convolution block for encoding processing to obtain a second feature map to be processed. After downsampling the second feature map to be processed, it is input into the fourth convolution block for decoding processing to obtain a third feature map to be processed. After upsampling the third feature map to be processed, it is channel-joined with the second feature map to be processed, and the result of channel-joined is input into the fifth convolution block for decoding processing to obtain a fourth feature map to be processed. After upsampling the fourth feature map to be processed, it is channel-joined with the first feature map to be processed, and the result of channel-joined is input into the sixth convolution block for decoding processing to obtain a first deep feature map.

[0100] The dark light enhancement module also includes a first convolution block, which further refines the first deep feature map. The refinement process is specifically as follows: nonlinear mapping is performed on the first deep feature map to obtain a second deep feature map. A dual-channel neural network is used to process the second deep feature map. The two channels of the dual-channel neural network are a Fourier branch and a spatial branch. For the Fourier branch, a fast Fourier transform is first performed on the second deep feature map to perform point-to-line conversion on each pixel point data of the second deep feature map to obtain the amplitude and phase of the second deep feature map. Furthermore, the waveform of the low-resolution dark light image is reconstructed according to the amplitude and phase.

[0101] Shallow feature maps assist in the extraction of deep feature maps. Shallow feature maps provide rich detail information, while deep feature maps provide higher-level semantic information. By combining shallow feature maps and deep feature maps, a more comprehensive and accurate image representation can be obtained.

[0102] Furthermore, the reconstructed waveform is optimized by time-space transformation to obtain optimized waveform data, such as Figure 5 As shown, the following steps are included:

[0103] S301 , performing nonlinear transformation on the amplitude and the phase respectively to obtain amplitude characteristics and phase characteristics.

[0104] S302: Perform inverse Fourier transform on the amplitude feature and the phase feature to obtain a Fourier transform feature graph.

[0105] S303: Perform a nonlinear transformation on the second deep feature map to obtain a spatial transformation feature map.

[0106] S304: Fusing the Fourier transform feature map and the spatial transform feature map in terms of channel dimension to obtain optimized waveform data.

[0107] The amplitude and phase are respectively transformed nonlinearly to obtain amplitude characteristics and phase characteristics, and the amplitude characteristics and phase characteristics are inversely Fourier transformed to obtain a Fourier transform characteristic map.

[0108] For the spatial branch, it is used to perform nonlinear transformation on the second deep feature map to obtain a spatial transformation feature map. The above-mentioned nonlinear transformations of amplitude, phase and space are all realized by the convolutional neural network with the same structure. The convolutional neural network includes a second convolutional layer with C convolution kernels and a convolution kernel size of 1*1 and a third convolutional layer with C convolution kernels and a convolution kernel size of 3*3. The ReLU activation function is used between the two convolutional layers.

[0109] The Fourier transform feature map and the spatial transform feature map are fused in the channel dimension to obtain the optimized waveform data. Specifically, the channel dimension is fused through a fourth convolution layer with a convolution kernel number of C and a convolution kernel size of 3*3 to obtain the optimized waveform data.

[0110] The dark light enhancement module also includes a fifth convolutional layer. The number of convolution kernels in the fifth convolutional layer is 3, and the convolution kernel size is 3*3. The fifth convolutional layer is used to reconstruct the brightness image residual of the edge texture features extracted from the waveform data to obtain the brightness image residual, and add the brightness image residual to the low-resolution dark light image to obtain a low-resolution enhanced image.

[0111] In a preferred embodiment of the present application, after obtaining a low-resolution enhanced image, the low-resolution enhanced image, the low-resolution dark-light image and the high-resolution dark-light image are used as inputs of an upsampling module of a guided filter to restore the low-resolution enhanced image to its original resolution and obtain a final high-resolution enhanced image.

[0112] In the upsampling module of the guided filtering, the guided filtering assumes that the guided graph and the output graph present a linear relationship within the neighborhood of each pixel. In actual operation, the sliding window size of the filter can be modified to adjust the neighborhood range. In a preferred embodiment of the present invention, the guided graph is the input high-resolution dark light image, and the output graph is the high-resolution enhanced image. Therefore, the guided filtering assumption can be expressed by the following formula:

[0113]

[0114] in, represents the kth filter sliding window, Represents a sliding window The pixel index within Represents the high-resolution enhanced image in the sliding window The first pixels, Represents a high-resolution dark-light image in a sliding window The first pixels, and They represent the linear coefficients and bias terms of the high-resolution dark-light image and high-resolution enhanced image in the sliding window, respectively.

[0115] In a preferred embodiment of the present invention, the upsampling module of the guided filtering includes a mean filter, a local linear module and a linear layer. First, the mean filter performs mean filtering on the low-resolution dark light image and the low-resolution enhanced image to obtain a variance feature map and a covariance feature map between the low-resolution dark light image and the low-resolution enhanced image, such as Figure 6 As shown, including:

[0116] S501. Perform a first mean filtering process on the low-resolution dark-light image and the low-resolution enhanced image to obtain a low-resolution dark-light image mean feature map and a low-resolution enhanced image mean feature map.

[0117] S502, performing a second mean filtering process on the low-resolution dark-light image, the low-resolution enhanced image, the low-resolution dark-light image mean feature map and the low-resolution enhanced image mean feature map to obtain a covariance feature map and a variance feature map.

[0118] The mean filter is defined as a box filter with 3 convolution kernels, a convolution kernel size of 3*3, and convolution kernel parameters of 1. The box filter is used to perform the first mean filtering process on the low-resolution dark light image and the low-resolution enhanced image to obtain the mean feature map of the low-resolution dark light image. And low-resolution enhanced image mean feature map The specific operation steps are as follows:

[0119]

[0120]

[0121] in, represents the mean feature map of low-resolution dark-light images, Represents the process of filtering the input feature map using a box filter, represents a low-resolution dark-light image, represents the mean feature map of the low-resolution enhanced image, Represents a low-resolution enhanced image.

[0122] Furthermore, a second mean filtering process is performed on the low-resolution dark light image, the low-resolution enhanced image, the low-resolution dark light image mean feature map and the low-resolution enhanced image mean feature map using a box filter to obtain a covariance feature map and a variance feature map. Pixel-by-pixel multiplication Then it is fed into the box filter and subtracted * Get the covariance feature map , and then Pixel-by-pixel multiplication Then it is fed into the box filter and subtracted * Get the variance feature map The operation steps are as follows:

[0123]

[0124]

[0125] in, represents the covariance feature map, Represents the variance feature map.

[0126] The variance feature map reflects the similarity between the low-resolution dark light image and the low-resolution enhanced image, and the covariance feature map reflects the correlation between the low-resolution dark light image and the low-resolution enhanced image. The variance feature map and the covariance feature map are spliced, wherein the splicing process is designed to merge the variance feature map and the covariance feature map on the corresponding channel dimension in the convolution layer of the convolutional neural network. The splicing result of the variance feature map and the covariance feature map is used to reflect the correlation between the low-resolution dark light image and the low-resolution enhanced image, which can effectively remove the interference of other information.

[0127] In a preferred embodiment of the present invention, local linear processing is performed on the splicing processing result to obtain a first linear coefficient and a first bias term, such as Figure 7 As shown, including:

[0128] S701, performing two nonlinear transformations on the splicing processing result to obtain a feature map to be compressed.

[0129] S702: compress the feature map to be compressed to obtain a 3-channel feature map in RGB format.

[0130] S703: Perform a linear operation on the three-channel feature map to obtain the first linear coefficient and the first bias term.

[0131] The local linear module is used to perform local linear processing on the splicing processing result to obtain a first linear coefficient and a first bias term. Specifically, the local linear module includes a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer and a ninth convolutional layer, the number of convolution kernels of the sixth convolutional layer and the seventh convolutional layer is C, the convolution kernel size is 1*1, and a ReLU activation function is used between the sixth convolutional layer and the seventh convolutional layer, all of which are used to perform nonlinear transformation on the splicing processing result to obtain a feature map to be compressed.

[0132] The number of convolution kernels in the eighth convolution layer is 3, and the convolution kernel size is 1*1. The eighth convolution layer is used to compress the feature map to be compressed into a 3-channel feature map in RGB format.

[0133] The number of convolution kernels of the ninth convolution layer is 3, and the convolution kernel size is 1*1. The ninth convolution layer is used to perform linear operations on the 3-channel feature map to obtain the first linear coefficient and the first bias term of each pixel.

[0134] Furthermore, the first linear coefficient and the first bias term of each pixel are processed into images to obtain a low-resolution parameter feature map.

[0135] After obtaining the low-resolution feature map, the low-resolution parameter feature map is upsampled using a bilinear difference algorithm to obtain a high-resolution parameter feature map. Correspondingly, the high-resolution parameter feature map includes a second linear coefficient and a second bias term.

[0136] The linear layer performs global linear processing on the high-resolution dark-light image according to the high-resolution parameter feature map to obtain a high-resolution enhanced image.

[0137] The expression of high-resolution enhanced image is:

[0138]

[0139] in, represents a high-resolution dark-light image, Represents the second linear coefficient corresponding to the high-resolution parameter feature map, Represents the second bias term corresponding to the high-resolution parameter feature map.

[0140] The lightweight high-resolution dark-light image enhancement method of the present invention is compared with the existing dark-light image enhancement processing methods. The existing dark-light image enhancement processing methods respectively use Uformer, Restormer and MPRNet models, all of which require block processing. Table 1 shows the quantitative comparison results.

[0141] Table 1

[0142]

[0143] like Figure 8 The results are shown as qualitative analysis.

[0144] From the results of quantitative and qualitative analysis, it can be seen that the lightweight high-resolution dark-light image enhancement method provided by the present invention solves the problem of brightness fragmentation and achieves a better visualization effect.

[0145] In an embodiment of the present application, the high-resolution dark-light image to be enhanced is downsampled to obtain a low-resolution dark-light image; each pixel point data of the low-resolution dark-light image is converted into a point-line, and a waveform diagram of the low-resolution dark-light image is reconstructed according to the data after the point-line conversion; the reconstructed waveform diagram is optimized by time-space domain transformation to obtain optimized waveform data; the edge texture features extracted from the waveform data are reconstructed to obtain a low-resolution enhanced image; the low-resolution dark-light image and the low-resolution enhanced image are mean filtered to obtain a variance feature diagram and a covariance between the low-resolution dark-light image and the low-resolution enhanced image Feature map; splicing the variance feature map and the covariance feature map, the splicing process is designed to merge the variance feature map and the covariance feature map on the corresponding channel dimension in the convolution layer of the convolutional neural network; performing local linear processing on the splicing result to obtain the first linear coefficient and the first bias term of each pixel; performing image processing on the first linear coefficient and the first bias term of each pixel to obtain a low-resolution parameter feature map; upsampling the low-resolution parameter feature map to obtain a high-resolution parameter feature map; performing global linear processing on the high-resolution dark-light image according to the high-resolution parameter feature map to obtain a high-resolution enhanced image. The lightweight high-resolution dark-light image enhancement method provided by the present invention can perform dark-light enhancement processing on the entire high-resolution dark-light image without brightness splitting, thereby achieving a better visualization effect.

[0146] Accordingly, if Fig. 9As shown, based on a lightweight high-resolution dark-light image enhancement method, an embodiment of the present invention further provides a lightweight high-resolution dark-light image enhancement system, including: a downsampling unit 1, a waveform reconstruction unit 2, a spatiotemporal transformation optimization processing unit 3, an image reconstruction unit 4, a mean filtering processing unit 5, a splicing processing unit 6, a local linear processing unit 7, a graphical processing unit 8, an upsampling unit 9 and a global linear processing unit 10;

[0147] The down-sampling unit 1 is used to down-sample the high-resolution dark-light image to be enhanced to obtain a low-resolution dark-light image.

[0148] The waveform reconstruction unit 2 is used to perform point-to-line conversion on each pixel point data of the low-resolution dark light image, and to reconstruct the waveform according to the point-to-line converted data.

[0149] The time-space domain transformation optimization processing unit 3 is used to perform time-space domain transformation optimization processing on the reconstructed waveform diagram to obtain optimized waveform data.

[0150] The image reconstruction unit 4 is used to reconstruct the edge texture features extracted from the waveform data to obtain a low-resolution enhanced image.

[0151] The mean filtering processing unit 5 is used to perform mean filtering on the low-resolution dark-light image and the low-resolution enhanced image to obtain a variance feature map and a covariance feature map between the low-resolution dark-light image and the low-resolution enhanced image.

[0152] The splicing processing unit 6 is used for splicing the variance feature map and the covariance feature map, wherein the splicing processing process is designed to merge the variance feature map and the covariance feature map on the corresponding channel dimension in the convolution layer of the convolutional neural network.

[0153] The local linear processing unit 7 is used to perform local linear processing on the splicing processing result to obtain the first linear coefficient and the first bias term of each pixel point.

[0154] The graphical processing unit 8 is used to perform graphical processing on the first linear coefficient and the first bias item of each pixel point to obtain a low-resolution parameter feature map.

[0155] The upsampling unit 9 is used to upsample the low-resolution parameter feature map to obtain a high-resolution parameter feature map.

[0156] The global linear processing unit 10 is used to perform global linear processing on the high-resolution dark light image according to the high-resolution parameter feature map to obtain a high-resolution enhanced image.

[0157] For the specific definition of a lightweight high-resolution dark-light image enhancement system, please refer to the above-mentioned definition of a lightweight high-resolution dark-light image enhancement method, which will not be repeated here. A person of ordinary skill in the art will appreciate that the various modules and steps described in conjunction with the embodiments disclosed in the present invention can be implemented in hardware, software, or a combination of both. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0158] like Fig.10 As shown, an embodiment of the present invention provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the steps in the above-mentioned lightweight high-resolution dark-light image enhancement method embodiment are implemented, for example Figure 1 Steps S1 to S10 described in .

[0159] Those skilled in the art will understand that the Fig.10 It is only an example of a computer device and does not constitute a limitation of the computer device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0160] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0161] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0162] Wherein, if the module integrated in the computer device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0163] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0164] Accordingly, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to perform the steps in the lightweight high-resolution dark light image enhancement method of the above embodiment, for example Figure 1 Steps S1 to S10 described in .

[0165] A lightweight high-resolution dark-light image enhancement method, system, computer device, and storage medium provided in this embodiment are aimed at the technical problem of poor human eye perception effect in existing dark-light image enhancement processing. The present invention downsamples the high-resolution dark-light image to be enhanced to obtain a low-resolution dark-light image; performs point-to-line conversion on each pixel point data of the low-resolution dark-light image, and reconstructs a waveform graph of the low-resolution dark-light image based on the point-to-line converted data; performs spatiotemporal domain transformation optimization processing on the reconstructed waveform graph to obtain optimized waveform data; performs image reconstruction on the edge texture features extracted from the waveform data to obtain a low-resolution enhanced image; performs mean filtering processing on the low-resolution dark-light image and the low-resolution enhanced image to obtain a variance feature graph and a covariance feature graph between the low-resolution dark-light image and the low-resolution enhanced image. ; The variance feature map and the covariance feature map are spliced, and the splicing process is designed to merge the variance feature map and the covariance feature map on the corresponding channel dimension in the convolution layer of the convolutional neural network; the splicing result is locally linearly processed to obtain the first linear coefficient and the first bias term of each pixel; the first linear coefficient and the first bias term of each pixel are imaged to obtain a low-resolution parameter feature map; the low-resolution parameter feature map is upsampled to obtain a high-resolution parameter feature map; the high-resolution dark-light image is globally linearly processed according to the high-resolution parameter feature map to obtain a high-resolution enhanced image. The lightweight high-resolution dark-light image enhancement method provided by the present invention can perform dark-light enhancement processing on the entire high-resolution dark-light image without the problem of brightness splitting, thereby achieving a better visualization effect.

[0166] The above-mentioned embodiments only express several preferred implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principle of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the protection scope of the claims.

Claims

1. A lightweight high-resolution dark-light image enhancement method, characterized in that: The method comprises: Down-sampling the high-resolution dark-light image to be enhanced to obtain a low-resolution dark-light image; Performing point-line conversion on each pixel point data of the low-resolution dark light image, and reconstructing a waveform graph according to the point-line converted data; Performing time-space domain transformation optimization processing on the reconstructed waveform diagram to obtain optimized waveform data; Reconstructing the edge texture features extracted from the waveform data to obtain a low-resolution enhanced image; Performing mean filtering on the low-resolution dark-light image and the low-resolution enhanced image to obtain a variance feature map and a covariance feature map between the low-resolution dark-light image and the low-resolution enhanced image; Performing splicing processing on the variance feature map and the covariance feature map, wherein the splicing process is designed to merge the variance feature map and the covariance feature map on the corresponding channel dimension in the convolution layer of the convolutional neural network; Performing local linear processing on the splicing processing result to obtain a first linear coefficient and a first bias term of each pixel point; Performing image processing on the first linear coefficient and the first bias item of each pixel point to obtain a low-resolution parameter feature map; Upsampling the low-resolution parameter feature map to obtain a high-resolution parameter feature map; Performing global linear processing on the high-resolution dark-light image according to the high-resolution parameter feature map to obtain a high-resolution enhanced image; The locally linear processing is performed on the splicing processing result to obtain a first linear coefficient and a first bias term, including: Perform two nonlinear transformations on the splicing results to obtain the feature map to be compressed; Compressing the feature map to be compressed to obtain a 3-channel feature map in RGB format; Perform a linear operation on the 3-channel feature map to obtain the first linear coefficient and the first bias term.

2. The lightweight high-resolution dark-light image enhancement method according to claim 1, characterized in that: The step of performing point-to-line conversion on the pixel data of the low-resolution dark light image and reconstructing a waveform according to the point-to-line converted data includes: Extracting shallow features of the low-resolution dark-light image to obtain a shallow feature map; Extracting deep features of the shallow feature map to obtain a first deep feature map; Performing nonlinear mapping on the first deep feature map to obtain a second deep feature map; Performing a fast Fourier transform on the second deep feature map to perform point-to-line conversion on each pixel point data of the second deep feature map to obtain the amplitude and phase of the second deep feature map; A waveform diagram is reconstructed for the low-resolution dark-light image according to the amplitude and the phase.

3. The lightweight high-resolution dark-light image enhancement method according to claim 2, characterized in that: The step of performing time-space domain transformation optimization processing on the reconstructed waveform graph to obtain optimized waveform data includes: Performing nonlinear transformation on the amplitude and the phase respectively to obtain amplitude characteristics and phase characteristics; Performing inverse Fourier transform on the amplitude feature and the phase feature to obtain a Fourier transform feature graph; Performing a nonlinear transformation on the second deep feature map to obtain a spatial transformation feature map; The Fourier transform feature map and the spatial transform feature map are fused in channel dimension to obtain optimized waveform data.

4. The lightweight high-resolution dark-light image enhancement method according to claim 1, characterized in that: The step of reconstructing the edge texture features extracted from the waveform data to obtain a low-resolution enhanced image includes: Reconstructing the brightness image residual on the edge texture features extracted from the waveform data to obtain the brightness image residual; The brightness image residual is added to the low-resolution dark light image to obtain a low-resolution enhanced image.

5. The lightweight high-resolution dark-light image enhancement method according to claim 1, characterized in that: The low-resolution dark-light image and the low-resolution enhanced image are subjected to mean filtering to obtain a variance feature map and a covariance feature map between the low-resolution dark-light image and the low-resolution enhanced image, including: Performing a first mean filtering process on the low-resolution dark-light image and the low-resolution enhanced image to obtain a low-resolution dark-light image mean feature map and a low-resolution enhanced image mean feature map; A second mean filtering process is performed on the low-resolution dark-light image, the low-resolution enhanced image, the low-resolution dark-light image mean feature map and the low-resolution enhanced image mean feature map to obtain the covariance feature map and the variance feature map.

6. The lightweight high-resolution dark-light image enhancement method according to claim 1, characterized in that: Performing global linear processing on the high-resolution dark-light image according to the high-resolution parameter feature map to obtain a high-resolution enhanced image includes: Performing global linear processing on the high-resolution dark-light image based on the high-resolution parameter feature map to obtain the high-resolution enhanced image; The expression of the high-resolution enhanced image is: in, represents a high-resolution enhanced image, represents a high-resolution dark-light image, Represents the second linear coefficient corresponding to the high-resolution parameter feature map, Represents the second bias term corresponding to the high-resolution parameter feature map.

7. A lightweight high-resolution dark light image enhancement system, characterized in that: The system comprises: a downsampling unit, a waveform reconstruction unit, a time-space domain transformation optimization processing unit, an image reconstruction unit, a mean filter processing unit, a splicing processing unit, a local linear processing unit, a graphical processing unit, an upsampling unit and a global linear processing unit; The down-sampling unit is used to down-sample the high-resolution dark-light image to be enhanced to obtain a low-resolution dark-light image; The waveform reconstruction unit is used to perform point-to-line conversion on each pixel point data of the low-resolution dark light image, and to reconstruct the waveform according to the point-to-line converted data; The time-space domain transformation optimization processing unit is used to perform time-space domain transformation optimization processing on the reconstructed waveform image to obtain optimized waveform data; The image reconstruction unit is used to reconstruct the edge texture features extracted from the waveform data to obtain a low-resolution enhanced image; The mean filtering processing unit is used to perform mean filtering on the low-resolution dark-light image and the low-resolution enhanced image to obtain a variance feature map and a covariance feature map between the low-resolution dark-light image and the low-resolution enhanced image; The splicing processing unit is used for splicing the variance feature map and the covariance feature map, wherein the splicing process is designed to merge the variance feature map and the covariance feature map on the corresponding channel dimension in the convolution layer of the convolutional neural network; The local linear processing unit is used to perform local linear processing on the splicing processing result to obtain a first linear coefficient and a first bias term of each pixel point; The graphical processing unit is used to perform graphical processing on the first linear coefficient and the first bias item of each pixel point to obtain a low-resolution parameter feature map; The upsampling unit is used to upsample the low-resolution parameter feature map to obtain a high-resolution parameter feature map; The global linear processing unit is used to perform global linear processing on the high-resolution dark-light image according to the high-resolution parameter feature map to obtain a high-resolution enhanced image; The locally linear processing is performed on the splicing processing result to obtain a first linear coefficient and a first bias term, including: Perform two nonlinear transformations on the splicing results to obtain the feature map to be compressed; Compressing the feature map to be compressed to obtain a 3-channel feature map in RGB format; Perform a linear operation on the 3-channel feature map to obtain the first linear coefficient and the first bias term.

8. A computer device, characterized in that: The computer device includes a memory, a processor and a transceiver, which are connected via a bus; the memory is used to store a set of computer program instructions and data, and transmit the stored data to the processor, and the processor executes the program instructions stored in the memory to execute the lightweight high-resolution dark-light image enhancement method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the lightweight high-resolution dark-light image enhancement method as described in any one of claims 1 to 6 is implemented.

Images