Image enhancement method and device based on multilevel feature fusion
Through the combination of deep neural networks and attention mechanisms, the multi-level features of low-illumination images are extracted and processed, and the problem of poor image enhancement effect in the prior art is solved, and a higher quality image enhancement effect is achieved.
Patent Information
- Application Number
- CN202411222670.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-29
AI Technical Summary
The existing low-illumination image enhancement algorithms cannot fully retain the original features, resulting in uneven image quality after enhancement, especially in poor processing in limited space scenes.
Deep neural network is used to extract multi-level features, adjust the feature extraction strategy through complexity and diversity measurement values, perform multi-level feature fusion processing, and use attention mechanism to identify key areas for denoising and lighting normalization, and achieve target enhancement image.
It effectively improves the details and contrast of the image, improves the stability and reliability of the image in different environments, fully retains the original features, and improves the visual effect and image quality.
Smart Images

Figure CN119048366A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology or other related fields, and in particular to an image enhancement method and device based on multi-level feature fusion. Background Art
[0002] When working in confined spaces, it is often necessary to shoot confined space scenes to meet construction needs. In order to meet relevant construction requirements, high-quality scene images are usually required. At present, construction parties use electronic equipment to collect relevant images of confined space scenes. However, due to different scenes and environments, the collected images are usually low-light images, which have the problems of weak light intensity, unclear image details, and low contrast. As a result, low-light images have a lot of noise and are blurred, and cannot clearly reflect the characteristics of the original image.
[0003] In the related art, when processing low-light images, an enhancement algorithm is usually used to enhance the features of low-light images. Existing low-light image enhancement algorithms often have improper brightness adjustment, detail loss or over-sharpening when processing low-light images. Especially when processing low-light images in limited spaces, existing low-light image enhancement algorithms cannot fully retain the original features of low-light images, resulting in uneven image quality after algorithm enhancement.
[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present invention provide an image enhancement method and device based on multi-level feature fusion, so as to at least solve the technical problem in the related art that the original features of the image cannot be fully preserved, resulting in poor quality of the enhanced image.
[0006] According to one aspect of an embodiment of the present invention, there is provided an image enhancement method based on multi-level feature fusion, comprising: extracting multi-level features of an original image using a deep neural network, obtaining a first enhanced image based on the multi-level features, wherein the multi-level features include depth information, color information and texture information of the original image; calculating the complexity of the first enhanced image and a diversity measure of the multi-level features, adjusting the feature extraction strategy of the deep neural network based on the complexity and diversity measures, and extracting new multi-level features using the adjusted deep neural network; fusing the new multi-level features to obtain a second enhanced image, wherein the second enhanced image is a noise image; inputting the second enhanced image into a modified neural network model to obtain a third enhanced image, wherein the modified neural network model includes a LeakyReLU activation function; using an attention mechanism to identify key areas of the third enhanced image to obtain a fourth enhanced image, wherein the key areas include at least one of the following: an edge area, a texture area and a saliency area; denoising and illumination normalization are performed on the fourth enhanced image to obtain a target enhanced image.
[0007] Optionally, the step of obtaining a first enhanced image based on multi-level features includes: inputting the original image into a first convolution layer in a deep neural network, performing a convolution operation on the original image and a convolution kernel weight matrix of the first convolution layer, and obtaining a first feature based on the convolution operation result and a bias vector of the first convolution layer. The calculation formula of the first feature is:
[0008]
[0009] Among them, F1 is the first feature, I in is the original image, W1 is the convolution kernel weight matrix of the first convolution layer, * is the convolution operator, b1 is the bias vector of the first convolution layer, is the activation function; the first feature is input into the second convolution layer in the deep neural network, a convolution operation is performed on the first feature and the convolution kernel weight matrix of the second convolution layer, and the second feature is obtained based on the convolution operation result and the bias vector of the second convolution layer. The calculation formula of the second feature is:
[0010]
[0011] Among them, F2 is the second feature, F1 is the first feature, W2 is the convolution kernel weight matrix of the second convolution layer, * is the convolution operator, b2 is the bias vector of the second convolution layer, is the activation function; the second feature is input into the pooling layer in the deep neural network to obtain the pooling feature; the pooling feature is input into the third convolution layer in the deep neural network, and the pooling feature and the convolution kernel weight matrix of the third convolution layer are convolved, and the third feature is obtained based on the convolution operation result and the bias vector of the third convolution layer. The calculation formula of the third feature is:
[0012]
[0013] Among them, F3 is the third feature, P is the pooling feature, W3 is the convolution kernel weight matrix of the third convolution layer, * is the convolution operator, and b3 is the bias vector of the third convolution layer. is the activation function.
[0014] Optionally, the step of obtaining the first enhanced image based on the multi-level features further includes: inputting the third feature into a deconvolution layer in the deep neural network, performing a convolution operation on the third feature and a deconvolution kernel weight matrix of the deconvolution layer, and obtaining a deconvolution output feature based on the operation result and a bias vector of the deconvolution layer, wherein the size of the deconvolution output feature is the same as or similar to the size of the original image, and the calculation formula of the deconvolution output feature is: Y = g(W u *F3+b u );
[0015] Among them, Y is the deconvolution output feature, F3 is the third feature, W u is the deconvolution kernel weight matrix of the deconvolution layer, * is the convolution operator, b u is the bias vector of the deconvolution layer, g(·) is the linear activation function; the deconvolution output features are converted back to the image space of the original image to obtain the first enhanced image. The calculation formula of the first enhanced image is:
[0016]
[0017] Among them, Y is the deconvolution output feature, I ed is the first enhanced image, is the mapping function.
[0018] Optionally, the feature extraction strategy of the deep neural network is adjusted based on the complexity and diversity measurement value, and the step of extracting new multi-level features using the adjusted deep neural network includes: calculating an adaptive factor based on the complexity and diversity measurement value, and the calculation formula of the adaptive factor is:
[0019]
[0020] Among them, C fis the complexity, D is the diversity measure, and σ(·) is the adaptive function. Based on the adaptive factor and multi-level features, the weight of each convolutional layer in the deep neural network is adjusted. The weight calculation formula is:
[0021]
[0022] Among them, W i is the weight of the i-th convolutional layer, is the adaptive factor, N is the number of convolutional layers, F i is the output feature of the i-th convolutional layer, F j is the output feature of the j-th convolutional layer, exp(·) is the natural exponential function, and β is a custom constant. The deep neural network is updated based on the adjusted weights, and the adjusted deep neural network is used to extract new multi-level features.
[0023] Optionally, the step of fusing the new multi-level features to obtain a second enhanced image includes: splicing the new multi-level features to obtain fused features; reconstructing the fused features, convolving the fused features and the convolution kernel weights of the reconstruction layer, and obtaining reconstructed features based on the convolution operation results and the bias vector of the reconstruction layer; and transforming the reconstructed features to obtain a second enhanced image.
[0024] Optionally, when an attention mechanism is used to identify the key area of the third enhanced image, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the step of the channel attention mechanism includes: inputting the third enhanced image into a fully connected layer in the channel attention mechanism, and obtaining a channel weight of the third enhanced image based on the output features of the fully connected layer. The calculation formula of the channel weight of the third enhanced image is:
[0025]
[0026] in, is the channel weight of the third enhanced image, z is the third enhanced image, W fc is the weight matrix of the fully connected layer, g(z, W fc ) represents the output features of the fully connected layer, and σ(·) is the activation function.
[0027] Optionally, when an attention mechanism is used to identify a key area of the third enhanced image, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the step of the spatial attention mechanism includes: based on the third enhanced image and the channel weight, linearly encoding the third enhanced image using the spatial attention mechanism to obtain a feature mapping function of the third enhanced image, and the feature mapping function is:
[0028]
[0029] in, is the feature mapping function, z is the third enhanced image, is the channel weight of the third enhanced image, W f is the weight of linear encoding, b f is the bias vector of linear encoding; based on the feature mapping function, the spatial attention map is obtained, and the calculation formula of the spatial attention map is:
[0030]
[0031] in, is the spatial attention map, z is the third enhanced image, is the channel weight of the third enhanced image, and softmax[·] is the activation function. The spatial attention map and the third enhanced image are element-wise multiplied to obtain the fourth enhanced image. The calculation formula of the element-wise multiplication operation is:
[0032]
[0033] Wherein, Mix(z) is the fourth enhanced image, z is the third enhanced image, is the channel weight of the third enhanced image, is the spatial attention map, is the element-by-element multiplication symbol.
[0034] Optionally, the step of performing denoising and illumination normalization processing on the fourth enhanced image in sequence to obtain a target enhanced image includes: acquiring all pixel values of the fourth enhanced image, and performing denoising processing on the fourth enhanced image based on the pixel values to obtain a denoised image of the fourth enhanced image, wherein the calculation formula for each pixel value in the denoised image is: end (m) = ∑ m,o∈Ω W(m,o)·I end (o);
[0035] Among them, I end (m) is the pixel value at position m in the denoised image, I end (o) is the pixel value at position o in the fourth enhanced image, W(m, o) is the weight of the denoising process, W(m, o) represents the similarity between the pixel value at position m and the pixel value at position o in the fourth enhanced image, Ω is the image domain of the fourth enhanced image, and the pixel value at position m and the pixel value at position o both belong to the image domain Ω; based on the pixel values of all denoised images, the denoised image is subjected to illumination normalization to obtain the target enhanced image, and the calculation formula for each pixel value in the target enhanced image is:
[0036]
[0037] Among them, Ir (m) is the pixel value at position m in the target enhanced image, I end (m) is the pixel value at position m in the denoised image, and τ is a custom constant.
[0038] Optionally, after obtaining the target enhanced image, the method includes: calculating a similarity index, a peak signal-to-noise ratio, a color fidelity, and an edge preservation degree based on the original image and the target enhanced image to obtain an evaluation result of the target enhanced image.
[0039] According to another aspect of an embodiment of the present invention, an image enhancement device based on multi-level feature fusion is also provided, including: an acquisition unit, used to extract multi-level features of an original image using a deep neural network, and acquire a first enhanced image based on the multi-level features, wherein the multi-level features include depth information, color information and texture information of the original image; an adjustment unit, used to calculate the complexity of the first enhanced image and the diversity measure of the multi-level features, adjust the feature extraction strategy of the deep neural network based on the complexity and diversity measure, and use the adjusted deep neural network to extract new multi-level features; a fusion unit, used to fuse the new multi-level features to obtain a second enhanced image, wherein the second enhanced image is a noise image; a correction unit, used to input the second enhanced image into a corrected neural network model to obtain a third enhanced image, wherein the corrected neural network model includes a LeakyReLU activation function; an identification unit, used to identify the key area of the third enhanced image using an attention mechanism to obtain a fourth enhanced image, wherein the key area includes at least one of the following: an edge area, a texture area and a salient area; a processing unit, used to perform denoising and illumination normalization on the fourth enhanced image to obtain a target enhanced image.
[0040] Optionally, the acquisition unit includes: a first convolution module, used to input the original image into the first convolution layer in the deep neural network, perform a convolution operation on the original image and the convolution kernel weight matrix of the first convolution layer, and obtain a first feature based on the convolution operation result and the bias vector of the first convolution layer. The calculation formula of the first feature is:
[0041]
[0042] Among them, F1 is the first feature, I in is the original image, W1 is the convolution kernel weight matrix of the first convolution layer, * is the convolution operator, b1 is the bias vector of the first convolution layer, is an activation function; the second convolution module is used to input the first feature into the second convolution layer in the deep neural network, perform convolution operation on the first feature and the convolution kernel weight matrix of the second convolution layer, and obtain the second feature based on the convolution operation result and the bias vector of the second convolution layer. The calculation formula of the second feature is:
[0043]
[0044] Among them, F2 is the second feature, F1 is the first feature, W2 is the convolution kernel weight matrix of the second convolution layer, * is the convolution operator, b2 is the bias vector of the second convolution layer, is an activation function; a pooling module, used to input the second feature into the pooling layer in the deep neural network to obtain a pooling feature; a third convolution module, used to input the pooling feature into the third convolution layer in the deep neural network, perform a convolution operation on the pooling feature and the convolution kernel weight matrix of the third convolution layer, and obtain the third feature based on the convolution operation result and the bias vector of the third convolution layer. The calculation formula of the third feature is:
[0045]
[0046] Among them, F3 is the third feature, P is the pooling feature, W3 is the convolution kernel weight matrix of the third convolution layer, * is the convolution operator, and b3 is the bias vector of the third convolution layer. is the activation function.
[0047] Optionally, the acquisition unit further includes: a deconvolution module, which is used to input the third feature into a deconvolution layer in the deep neural network, perform a convolution operation on the third feature and a deconvolution kernel weight matrix of the deconvolution layer, and obtain a deconvolution output feature based on the operation result and a bias vector of the deconvolution layer, wherein the size of the deconvolution output feature is the same as or similar to the size of the original image, and the calculation formula of the deconvolution output feature is:
[0048] Y=g(W u *F3+b u );
[0049] Among them, Y is the deconvolution output feature, F3 is the third feature, W u is the deconvolution kernel weight matrix of the deconvolution layer, * is the convolution operator, b u is the bias vector of the deconvolution layer, g(·) is a linear activation function; the conversion module is used to convert the deconvolution output features back to the image space of the original image to obtain the first enhanced image. The calculation formula of the first enhanced image is:
[0050]
[0051] Among them, Y is the deconvolution output feature, I ed is the first enhanced image, is the mapping function.
[0052] Optionally, the adjustment unit includes: a first calculation module, which calculates an adaptive factor based on complexity and diversity metrics, and the calculation formula of the adaptive factor is:
[0053]
[0054] Among them, C f is the complexity, D is the diversity measure, and σ(·) is the adaptive function. The second calculation module adjusts the weight of each convolutional layer in the deep neural network based on the adaptive factor and multi-level features. The weight calculation formula is:
[0055]
[0056] Among them, W i is the weight of the i-th convolutional layer, is the adaptive factor, N is the number of convolutional layers, F i is the output feature of the i-th convolutional layer, F j is the output feature of the j-th convolutional layer, exp(·) is the natural exponential function, and β is a custom constant; the update module updates the deep neural network based on the adjusted weights, and uses the adjusted deep neural network to extract new multi-level features.
[0057] Optionally, the fusion unit includes: a splicing module, which is used to splice new multi-level features to obtain fused features; a reconstruction module, which is used to reconstruct the fused features, perform convolution operations on the fused features and the convolution kernel weights of the reconstruction layer, and obtain reconstructed features based on the convolution operation results and the bias vector of the reconstruction layer; a linear transformation module, which is used to transform the reconstructed features to obtain a second enhanced image.
[0058] Optionally, the recognition unit includes: a first acquisition module, used to input the third enhanced image into a fully connected layer in the channel attention mechanism, and acquire the channel weight of the third enhanced image based on the output features of the fully connected layer. The calculation formula of the channel weight of the third enhanced image is:
[0059]
[0060] in, is the channel weight of the third enhanced image, z is the third enhanced image, W fc is the weight matrix of the fully connected layer, g(z, W fc ) represents the output features of the fully connected layer, and σ(·) is the activation function.
[0061] Optionally, the recognition unit further includes: an encoding module, which uses a spatial attention mechanism to linearly encode the third enhanced image based on the third enhanced image and the channel weight to obtain a feature mapping function of the third enhanced image, and the feature mapping function is:
[0062]
[0063] in, is the feature mapping function, z is the third enhanced image, is the channel weight of the third enhanced image, W f is the weight of linear encoding, b f is the bias vector of linear encoding; the second acquisition module obtains the spatial attention map based on the feature mapping function, and the calculation formula of the spatial attention map is:
[0064]
[0065] in, is the spatial attention map, z is the third enhanced image, is the channel weight of the third enhanced image, and softmax[·] is the activation function; the multiplication module is used to perform element-by-element multiplication operation on the spatial attention map and the third enhanced image to obtain the fourth enhanced image. The calculation formula of the element-by-element multiplication operation is:
[0066]
[0067] Wherein, Mix(z) is the fourth enhanced image, z is the third enhanced image, is the channel weight of the third enhanced image, is the spatial attention map, is the element-by-element multiplication symbol.
[0068] Optionally, the processing unit includes: a denoising module, used to obtain all pixel values of the fourth enhanced image, and perform denoising processing on the fourth enhanced image based on the pixel values to obtain a denoised image of the fourth enhanced image, and the calculation formula for each pixel value in the denoised image is:
[0069] I end (m) = ∑ m,o∈Ω W(m,o)·I end (o);
[0070] Among them, I end (m) is the pixel value at position m in the denoised image, I end(o) is the pixel value at position o in the fourth enhanced image, W(m, o) is the weight of the denoising process, W(m, o) represents the similarity between the pixel value at position m and the pixel value at position o of the fourth enhanced image, Ω is the image domain of the fourth enhanced image, and the pixel value at position m and the pixel value at position o both belong to the image domain Ω; the normalization module performs illumination normalization processing on the denoised image based on the pixel values of all denoised images to obtain a target enhanced image, and the calculation formula for each pixel value in the target enhanced image is:
[0071]
[0072] Among them, I r (m) is the pixel value at position m in the target enhanced image, I end (m) is the pixel value at position m in the denoised image, and τ is a custom constant.
[0073] Optionally, the processing unit further includes: an evaluation module, which calculates a similarity index, a peak signal-to-noise ratio, a color fidelity, and an edge preservation degree based on the original image and the target enhanced image to obtain an evaluation result of the target enhanced image.
[0074] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned image enhancement methods based on multi-level feature fusion.
[0075] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned image enhancement methods based on multi-level feature fusion.
[0076] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, which implements the steps of any one of the above-mentioned image enhancement methods based on multi-level feature fusion when executed by a processor.
[0077] In the present disclosure, a deep neural network is used to extract multi-level features of an original image, a first enhanced image is obtained based on the multi-level features, the complexity of the first enhanced image and the diversity metric of the multi-level features are calculated, the feature extraction strategy of the deep neural network is adjusted based on the complexity and diversity metric, and new multi-level features are extracted using the adjusted deep neural network, thereby further improving the accuracy of multi-level feature extraction, and then a second enhanced image is obtained by fusing the new multi-level features, and the details and contrast of the image are enhanced by effectively fusing the multi-level features, and then a third enhanced image is obtained by inputting the second enhanced image into a modified neural network model, thereby further improving the brightness of the image and clarity, and then the key area of the third enhanced image is identified by adopting the attention mechanism, the key area includes at least one of the following: edge area, texture area and salient area, and the information of the key area is made more prominent by obtaining the fourth enhanced image, and finally the target enhanced image is obtained by denoising and illumination normalization of the fourth enhanced image, which further enhances the details and contrast of the image by resisting the influence of noise and illumination changes, improves the stability and reliability of the image in different environments, effectively improves the visual effect and image quality of the original low-light image, and thus solves the technical problem that the original features of the image cannot be fully retained in the related technology, resulting in poor quality of the enhanced image. BRIEF DESCRIPTION OF THE DRAWINGS
[0078] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0079] Figure 1 is a flowchart of an optional image enhancement method based on multi-level feature fusion according to an embodiment of the present invention;
[0080] Figure 2 is a schematic diagram of an optional image enhancement device based on multi-level feature fusion according to an embodiment of the present invention;
[0081] Figure 3 It is a hardware structure block diagram of an electronic device (or mobile device) of an image enhancement method based on multi-level feature fusion according to an embodiment of the present invention. DETAILED DESCRIPTION
[0082] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0083] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0084] It should be noted that the image enhancement method based on multi-level feature fusion and the device thereof in the present disclosure can be used in the field of image processing technology. When the image is enhanced based on multi-level feature fusion, it can also be used in any field except the field of image processing technology. When the image is enhanced based on multi-level feature fusion, the present disclosure does not limit the application field of the image enhancement method based on multi-level feature fusion and the device thereof.
[0085] The following embodiments of the present invention can be applied to various systems / applications / devices for image enhancement based on multi-level feature fusion. The present invention extracts multi-level features of the original image by using a deep neural network, and adjusts the feature extraction strategy of the deep neural network based on the complexity of the image and the diversity metric of the multi-level features, extracts multi-level features with higher accuracy, and then effectively enhances the details and contrast of the image by fusing the multi-level features, and then further improves the brightness and clarity of the image by inputting the image into the modified neural network model, and then identifies the key areas of the image by using the attention mechanism, so that the information of the key areas is more prominent, and finally, by denoising and normalizing the image to resist the influence of noise and illumination changes, the details and contrast of the image are further enhanced, the visual effect and image quality of the original low-light image are effectively improved, the original features of the image are fully retained, and the quality of the enhanced image is improved.
[0086] The present invention is described in detail below in conjunction with various embodiments.
[0087] Embodiment 1
[0088] According to an embodiment of the present invention, an embodiment of an image enhancement method based on multi-level feature fusion is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0089] Figure 1 is a flowchart of an optional image enhancement method based on multi-level feature fusion according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:
[0090] Step S101, using a deep neural network to extract multi-level features of the original image, and obtaining a first enhanced image based on the multi-level features, wherein the multi-level features include depth information, color information, and texture information of the original image.
[0091] In an embodiment of the present invention, by inputting the original image into a deep neural network, the multi-level features of the original image can be extracted. It should be noted that the original image mentioned in this embodiment may be a low-light image. Low-light images usually have problems such as weak light intensity, unclear image details, and low contrast. By extracting the multi-level features of the low-light image, a first enhanced image can be obtained based on the multi-level features. The multi-level features include the depth information, color information, and texture information of the original image, that is, the multi-level features can reflect the depth, color, and texture of the original image. The first enhanced image is a single-channel grayscale image or a multi-channel color image with the same or similar size as the original image. The multi-level features obtained in this step and the first enhanced image will be used to adjust the deep neural network so that the deep neural network can extract more accurate multi-level features.
[0092] Optionally, the step of obtaining a first enhanced image based on multi-level features includes: inputting the original image into a first convolution layer in a deep neural network, performing a convolution operation on the original image and a convolution kernel weight matrix of the first convolution layer, and obtaining a first feature based on the convolution operation result and a bias vector of the first convolution layer. The calculation formula of the first feature is: Among them, F1 is the first feature, I in is the original image, W1 is the convolution kernel weight matrix of the first convolution layer, * is the convolution operator, b1 is the bias vector of the first convolution layer, is the activation function; the first feature is input into the second convolution layer in the deep neural network, a convolution operation is performed on the first feature and the convolution kernel weight matrix of the second convolution layer, and the second feature is obtained based on the convolution operation result and the bias vector of the second convolution layer. The calculation formula of the second feature is: Among them, F2 is the second feature, F1 is the first feature, W2 is the convolution kernel weight matrix of the second convolution layer, * is the convolution operator, b2 is the bias vector of the second convolution layer, is the activation function; the second feature is input into the pooling layer in the deep neural network to obtain the pooling feature; the pooling feature is input into the third convolution layer in the deep neural network, and the pooling feature and the convolution kernel weight matrix of the third convolution layer are convolved, and the third feature is obtained based on the convolution operation result and the bias vector of the third convolution layer. The calculation formula of the third feature is:
[0093]
[0094] Among them, F3 is the third feature, P is the pooling feature, W3 is the convolution kernel weight matrix of the third convolution layer, * is the convolution operator, and b3 is the bias vector of the third convolution layer. is the activation function.
[0095] In an embodiment of the present invention, by inputting the original image into a deep neural network, multi-level features can be obtained. The deep neural network includes three convolutional layers, one pooling layer and one deconvolution layer. By inputting the original image into the first convolutional layer, the first feature F1 can be obtained, and then the first feature is input into the second convolutional layer, the second feature F2 can be obtained, and then the second feature is input into the pooling layer, and the pooling feature is input into the third convolutional layer, and the third feature F3 can be obtained. The first feature, the second feature and the third feature together constitute the multi-level features of the original image. In the calculation formula of the first feature, the second feature and the third feature, the convolution kernel weight matrix and the bias vector of each convolution layer can be set by itself, and the activation function of each convolution layer can be selected by itself. Usually, the convolution kernel weight matrix, the bias vector and the activation function are determined according to the requirements of the embodiment, and the convolution kernel weight matrix, the bias vector and the activation function of different convolution layers may be the same or different.
[0096] It should be noted that when the second feature is input into the pooling layer, the pooling feature can be obtained, and then the pooling feature is input into the third convolution layer to obtain the third feature, where the calculation formula of the pooling feature is: P = pol (F2);
[0097] Among them, F2 is the second feature, P is the pooling feature, pol(·) is the pooling function, and the pooling function can choose the maximum pooling function or the average pooling function.
[0098] Optionally, the step of obtaining the first enhanced image based on the multi-level features further includes: inputting the third feature into a deconvolution layer in the deep neural network, performing a convolution operation on the third feature and a deconvolution kernel weight matrix of the deconvolution layer, and obtaining a deconvolution output feature based on the operation result and a bias vector of the deconvolution layer, wherein the size of the deconvolution output feature is the same as or similar to the size of the original image, and the calculation formula of the deconvolution output feature is: Y = g(W u *F3+b u );
[0099] Among them, Y is the deconvolution output feature, F3 is the third feature, W u is the deconvolution kernel weight matrix of the deconvolution layer, * is the convolution operator, b u is the bias vector of the deconvolution layer, g(·) is the linear activation function; the deconvolution output features are converted back to the image space of the original image to obtain the first enhanced image. The calculation formula of the first enhanced image is:
[0100]
[0101] Among them, Y is the deconvolution output feature, I ed is the first enhanced image, is the mapping function.
[0102] In an embodiment of the present invention, after obtaining the third feature F3, the third feature is input into the deconvolution layer, and a deconvolution output feature having the same or similar size as the original image can be obtained, that is, a calculation formula for the deconvolution output feature, the purpose of which is to restore the size of the third feature to the size of the original image. In the calculation formula for the deconvolution output feature, the deconvolution kernel weight matrix and the bias vector can be set by themselves, and the linear activation function can be selected by themselves. Usually, the deconvolution kernel weight matrix, the bias vector, and the linear activation function are determined according to the requirements of the embodiment.
[0103] After obtaining the deconvolution output features of the same or similar size as the original image, they are input into the target mapping function to convert the deconvolution output features back to the image space of the original image to obtain the first enhanced image. The mapping function can be selected by yourself. The selected mapping function can convert the deconvolution output features back to the image space of the original image. The first enhanced image will be used to adjust the deep neural network so that the deep neural network can extract more accurate multi-level features.
[0104] Step S102, calculating the complexity of the first enhanced image and the diversity metric of the multi-level features, adjusting the feature extraction strategy of the deep neural network based on the complexity and diversity metric, and using the adjusted deep neural network to extract new multi-level features.
[0105] In an embodiment of the present invention, after obtaining the multi-level features of the original image and the first enhanced image, in order to adjust the deep neural network, the complexity of the first enhanced image and the diversity metric of the multi-level features are first calculated.
[0106] The calculation formula of the complexity of the first enhanced image is:
[0107]
[0108] Among them, I ed is the first enhanced image, C f is the complexity of the first enhanced image, I ed , x and I ed , y I ed The gradients in the x and y directions, W and H are I ed The width and height of I ed,x (x, y) and I ed,y (x, y) are I ed The gradient values along the x-axis and y-axis at the point (x, y).
[0109] The calculation formula for the diversity metric of multi-level features is:
[0110]
[0111] Among them, D is the diversity metric, N is the number of feature layers, and F i and F j are the feature vectors of the i-th layer and the j-th layer respectively, d(F i , F j ) are two eigenvectors F i and F j The distance metric between them, in the embodiment of the present invention, the number of feature layers N is 3, and the feature vector includes a first feature F1, a second feature F2, and a third feature F3.
[0112] After calculating the complexity of the first enhanced image and the diversity measure of the multi-level features, the feature extraction strategy of the deep neural network is adjusted based on the complexity and diversity measure. New multi-level features can be obtained by using the adjusted deep neural network. The new multi-level features will be used for fusion processing to obtain a new enhanced image.
[0113] Optionally, the feature extraction strategy of the deep neural network is adjusted based on the complexity and diversity measurement value, and the step of extracting new multi-level features using the adjusted deep neural network includes: calculating an adaptive factor based on the complexity and diversity measurement value, and the calculation formula of the adaptive factor is:
[0114]
[0115] Among them, C f is the complexity, D is the diversity measure, and σ(·) is the adaptive function. Based on the adaptive factor and multi-level features, the weight of each convolutional layer in the deep neural network is adjusted. The weight calculation formula is:
[0116]
[0117] Among them, W i is the weight of the i-th convolutional layer, is the adaptive factor, N is the number of convolutional layers, F i is the output feature of the i-th convolutional layer, F j is the output feature of the j-th convolutional layer, exp(·) is the natural exponential function, and β is a custom constant. The deep neural network is updated based on the adjusted weights, and the adjusted deep neural network is used to extract new multi-level features.
[0118] In an embodiment of the present invention, the feature extraction strategy of the deep neural network is adjusted based on the complexity and diversity metrics by calculating an adaptive factor and adjusting the weight of each convolutional layer in the deep neural network based on the adaptive factor.
[0119] In the calculation formula of the adaptive factor, σ(C f , D) represents an adaptive function, which is based on the image complexity C f The ratio of the product of the diversity measure D of the multi-level features and their sum is determined when the image complexity C f When the feature diversity metric D is high, the adaptive factor Close to 1, it means that more feature details need to be extracted. f When both the feature diversity metric D and the adaptive factor Close to 0, indicating that feature extraction should be more conservative.
[0120] The adaptive factor is calculated Afterwards, through To adjust the weight W of each convolutional layer in the deep neural network i In the weight calculation formula, β is a constant used to control the slope of the exponential function exp(·), which in turn affects the concentration of weight distribution. is a normalization term, ensuring that the sum of all weights is equal to 1. In the embodiment of the present invention, the number of convolutional layers N is 3, and the feature vector includes the first feature F1, the second feature F2, and the third feature F3. After calculating the new convolutional layer weights, the deep neural network is updated, and the updated deep neural network is used to extract new multi-level features. The new multi-level features will be used for fusion processing to obtain a new enhanced image.
[0121] It should be noted that after obtaining the adjusted weight W i After that, the output feature F of each convolutional layer can be i Perform weighted summation to obtain the result of weighted summation of all hierarchical features according to their weights. The calculation formula for weighted summation is:
[0122]
[0123] Among them, N is the number of convolutional layers, F i is the output feature of the i-th convolutional layer, W i is the weight of the i-th convolutional layer, F fd It is the result of weighted summation of all hierarchical features according to their weights.
[0124] Step S103, performing fusion processing on the new multi-level features to obtain a second enhanced image, wherein the second enhanced image is a noise image.
[0125] In an embodiment of the present invention, after obtaining new multi-level features F1, F2 and F3, the multi-level features are fused to enhance the details and contrast of the image. The second enhanced image obtained after feature fusion is a noise image. The second feature image will be further processed in subsequent steps to enhance more features of the image.
[0126] Optionally, the step of fusing the new multi-level features to obtain a second enhanced image includes: splicing the new multi-level features to obtain fused features; reconstructing the fused features, convolving the fused features and the convolution kernel weights of the reconstruction layer, and obtaining reconstructed features based on the convolution operation results and the bias vector of the reconstruction layer; and transforming the reconstructed features to obtain a second enhanced image.
[0127] In the embodiment of the present invention, the fusion of multi-level features is achieved by splicing the multi-level features, and the calculation formula for the splicing process is:
[0128] F fon =Cat(F1, F2, F3);
[0129] Among them, F fonis the fusion feature, F1 is the first feature, F2 is the second feature, F3 is the third feature, and Cat(·) is the concatenation function. The concatenation function can connect multiple feature vectors together according to the specified dimension. After obtaining the fusion feature, the fusion feature needs to be reconstructed. The calculation formula for the reconstruction process is:
[0130] T r =g(W uc *F fon +b uc );
[0131] Among them, T r is the output feature of the reconstruction layer, F fon is the fusion feature, g(·) is the activation function, which is usually a linear function or a nonlinear function, such as a Sigmoid function or a Tanh function, and W uc is the convolution kernel weight matrix of the reconstruction layer, b uc It is the bias vector of the reconstruction layer. The convolution kernel weight matrix and the bias vector of the reconstruction layer are set by themselves, usually determined according to the requirements of the embodiment.
[0132] The reconstructed features are then transformed to obtain a second enhanced image. The calculation formula for the transformation is:
[0133]
[0134] Among them, I end is the second enhanced image, T r is the output feature of the reconstruction layer, is a simple linear transformation function or a nonlinear transformation function, such as a Sigmoid function or a Tanh function. The function is to ensure that the function output value is within a reasonable range, that is, to ensure that I end The value of is within the range required by the embodiment of the present invention. After obtaining the second enhanced image, the second feature image will be further processed in subsequent steps to enhance more features of the image.
[0135] Step S104: input the second enhanced image into a modified neural network model to obtain a third enhanced image, wherein the modified neural network model includes a LeakyReLU activation function.
[0136] In the embodiment of the present invention, after obtaining the second enhanced image, it is input into the modified neural network model to obtain the third enhanced image. The modified neural network model uses the LeakyReLU activation function, and the calculation formula for processing the second enhanced image using the LeakyReLU activation function is:
[0137] I prd=max(0,I end )+min(0,rI end );
[0138] Among them, I prd is the third enhanced image after being processed by the LeakyReLU activation function, I end is the second enhanced image, r is a constant, and the value of r is usually 0.01 or 0.02. The LeakyReLU activation function allows input values less than zero to have non-zero outputs. After the second enhanced image is processed by the LeakyReLU activation function, the brightness and clarity of the image can be further improved.
[0139] Step S105, using an attention mechanism to identify key areas of the third enhanced image to obtain a fourth enhanced image, where the key areas include at least one of the following: an edge area, a texture area, and a salient area.
[0140] In an embodiment of the present invention, an attention mechanism is used to identify key areas of an image, namely, edge areas, texture areas and salient areas of the image. The attention mechanism of the embodiment of the present invention includes a channel attention mechanism and a spatial attention mechanism. After the third enhanced image is subjected to the channel attention mechanism and the spatial attention mechanism, the features of the key areas of the image can be identified and enhanced to obtain a fourth enhanced image. The fourth enhanced image is then subjected to denoising and illumination normalization processing in subsequent steps to obtain the target enhanced image of the embodiment of the present invention.
[0141] Optionally, when an attention mechanism is used to identify the key area of the third enhanced image, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the step of the channel attention mechanism includes: inputting the third enhanced image into a fully connected layer in the channel attention mechanism, and obtaining a channel weight of the third enhanced image based on the output features of the fully connected layer. The calculation formula of the channel weight of the third enhanced image is:
[0142]
[0143] in, is the channel weight of the third enhanced image, z is the third enhanced image, W fc is the weight matrix of the fully connected layer, g(z, W fc ) represents the output features of the fully connected layer, and σ(·) is the activation function.
[0144] In this embodiment of the present invention, the channel attention mechanism includes a fully connected layer and an activation function. The third enhanced image is input into the fully connected layer to obtain the output feature g(z, W fc), and then input the output feature into the activation function σ(·), the channel weight of the third enhanced image can be obtained. In the calculation formula of the channel weight of the third enhanced image, the weight matrix W of the fully connected layer fc The activation function σ(·) is set by itself, and the weight matrix and the activation function are usually determined according to the requirements of the embodiment. The channel weights of the third enhanced image obtained in this step will be input into the spatial attention mechanism for further processing.
[0145] It should be noted that, in the embodiment of the present invention, the third enhanced image may also be subjected to global average pooling processing to obtain the global average pooling feature of each channel of the third enhanced image. The calculation formula of the global average pooling processing is:
[0146]
[0147] in, is the global average pooling feature of the cth channel of the third enhanced image, x c (i, j) is the feature value of the position on the cth channel of the third enhanced image, W and H are the width and height of the third enhanced image respectively, is the normalization factor. The function of the normalization factor is to make the output image It is a scalar value representing the average characteristic intensity of the c channel.
[0148] Optionally, when an attention mechanism is used to identify a key area of the third enhanced image, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the step of the spatial attention mechanism includes: based on the third enhanced image and the channel weight, linearly encoding the third enhanced image using the spatial attention mechanism to obtain a feature mapping function of the third enhanced image, and the feature mapping function is: in, is the feature mapping function, z is the third enhanced image, is the channel weight of the third enhanced image, W f is the weight of linear encoding, b f is the bias vector of linear encoding; based on the feature mapping function, the spatial attention map is obtained, and the calculation formula of the spatial attention map is: in, is the spatial attention map, z is the third enhanced image, is the channel weight of the third enhanced image, and softmax[·] is the activation function. The spatial attention map and the third enhanced image are element-wise multiplied to obtain the fourth enhanced image. The calculation formula of the element-wise multiplication operation is: Wherein, Mix(z) is the fourth enhanced image, z is the third enhanced image, is the channel weight of the third enhanced image, is the spatial attention map, is the element-by-element multiplication symbol.
[0149] In the embodiment of the present invention, the attention mechanism includes not only a channel attention mechanism but also a spatial attention mechanism. The channel attention mechanism and the spatial attention mechanism jointly identify and enhance the key areas of the third enhanced image. In the spatial attention mechanism, the output features of the channel attention, i.e., the channel weights of the third enhanced image, are and the product of the third enhanced image z As the input of the spatial attention mechanism, the product Linear coding is performed to obtain the feature mapping function of the third enhanced image. In the feature mapping function, the weight W of the linear coding is f and the bias vector b f Self-set, usually determined according to the requirements of the embodiment.
[0150] Then input the feature mapping function into the softmax function to obtain the spatial attention map of the third enhanced image The softmax function is a normalization function, and its expression is:
[0151]
[0152] Among them, z i is the i-th element in the input vector z, e is the base of the natural logarithm, The purpose of the softmax function is to convert each element in the input vector z into a probability value ranging from 0 to 1, and the sum of all probability values is 1.
[0153] After getting the spatial attention map After that, the spatial attention map and the third enhanced image are element-by-element multiplied to obtain the fourth enhanced image. Element-by-element multiplication multiplies the elements at corresponding positions in the two matrices to obtain the elements at corresponding positions in the new matrix. The final Mix(z) is the fourth enhanced image, that is, the enhanced image processed by the channel attention mechanism and the spatial attention mechanism. Compared with the third enhanced image, the image features of the key area of the fourth enhanced image are clearer.
[0154] Step S106, performing denoising and illumination normalization processing on the fourth enhanced image to obtain a target enhanced image.
[0155] In the embodiment of the present invention, the fourth enhanced image is still a noisy image. In this step, non-local mean denoising is performed on it to obtain a clean image with noise removed. Then, illumination normalization is performed on it to reduce the influence of illumination on image clarity. The denoising process and illumination normalization process are combined to be called a robust enhancement strategy. The robust enhancement strategy can further improve the details and contrast of the image. After the fourth enhanced image is processed by the robust enhancement strategy, the target enhanced image of the embodiment of the present invention can be obtained.
[0156] Optionally, the step of performing denoising and illumination normalization processing on the fourth enhanced image in sequence to obtain a target enhanced image includes: acquiring all pixel values of the fourth enhanced image, and performing denoising processing on the fourth enhanced image based on the pixel values to obtain a denoised image of the fourth enhanced image, wherein the calculation formula for each pixel value in the denoised image is: end (m) = ∑ m,o∈Ω W(m,o)·I end (o);
[0157] Among them, I end (m) is the pixel value at position m in the denoised image, I end (o) is the pixel value at position o in the fourth enhanced image, W(m, o) is the weight of the denoising process, W(m, o) represents the similarity between the pixel value at position m and the pixel value at position o in the fourth enhanced image, Ω is the image domain of the fourth enhanced image, and the pixel value at position m and the pixel value at position o both belong to the image domain Ω; based on the pixel values of all denoised images, the denoised image is subjected to illumination normalization to obtain the target enhanced image, and the calculation formula for each pixel value in the target enhanced image is:
[0158]
[0159] Among them, I r (m) is the pixel value at position m in the target enhanced image, I end (m) is the pixel value at position m in the denoised image, and τ is a custom constant.
[0160] In the embodiment of the present invention, the calculation unit of the non-local denoising process is the pixel value of the image. All pixel values of the fourth enhanced image after the denoising process together constitute the denoised image. In the calculation formula of the pixel value, W(m, o) represents the similarity between the pixel value at position m and the pixel value at position o of the fourth enhanced image. The Gaussian kernel is defined for W(m, o):
[0161]
[0162] Where h is the Gaussian kernel width, I end (m) is the noise image Iend The pixel value at position m in the end (o) is the noise image I end The pixel value at position o in , ‖I end (m)-I end (o)‖ is the Euclidean norm, which is used to measure the difference in grayscale values between pixel m and pixel o. is an exponential function. The purpose of this exponential function is to ensure that when the grayscale value difference between pixel m and pixel o is small, the weight W(m, o) is close to 1, and when the grayscale value difference between pixel m and pixel o is large, the weight W(m, o) decreases rapidly to close to 0, so that the pixel o that is similar to pixel m has a greater impact on pixel m, and the pixel o that is not similar to pixel m has a smaller impact on pixel m.
[0163] After the fourth enhanced image is processed by non-local denoising, in order to adapt to different lighting conditions, the embodiment of the present invention further performs illumination normalization processing on the denoised image, and finally obtains the target enhanced image of the embodiment of the present invention. In the calculation formula of illumination normalization, the value of the constant τ is set voluntarily, and is usually determined according to the requirements of the embodiment.
[0164] It should be noted that the embodiment of the present invention can adjust the pixel value of the image by calculating the distribution of the local illumination intensity of the image. The calculation formula of the local illumination intensity distribution is:
[0165] I a (m) = q·I r (m) γ ;
[0166] Among them, I r (m) is the pixel value at position m in the enhanced image, I a (m) is the pixel value at position m in the enhanced image after the local illumination intensity is adjusted, q is the coefficient for adjusting the overall brightness of the enhanced image, γ is a constant used to adjust the contrast of the enhanced image, γ<1 will increase the details of the dark part of the image, γ>1 will increase the details of the bright part of the image, and the value of γ is determined according to the requirements.
[0167] Optionally, after obtaining the target enhanced image, the method includes: calculating a similarity index, a peak signal-to-noise ratio, a color fidelity, and an edge preservation degree based on the original image and the target enhanced image to obtain an evaluation result of the target enhanced image.
[0168] In the embodiment of the present invention, the target enhanced image is the result of the embodiment of the present invention, and the original low-light image is processed from step 101 to step S106 to obtain the target enhanced image. The embodiment of the present invention also evaluates the target enhanced image, and evaluates the enhancement effect of the target enhanced image by calculating the similarity index, peak signal-to-noise ratio, color fidelity, and edge preservation of the original image and the target enhanced image.
[0169] The calculation formula of similarity index is:
[0170]
[0171] Among them, SSIM is the similarity index between the original image and the target enhanced image, μ end and μ in are the average values of the target enhanced image and the original image, respectively. and are the variances of the target enhanced image and the original image, δ endin is the covariance between the target enhanced image and the original image, c1 and c2 are small constants used to avoid the denominator being zero.
[0172] When the similarity index SSIM is less than the preset similarity threshold SSIM y , it is determined that the similarity index of the target enhanced image does not meet the requirement, and the robustness enhancement strategy in step S106 needs to be adjusted.
[0173] The peak signal-to-noise ratio is calculated as:
[0174]
[0175] Among them, PSNR is the peak signal-to-noise ratio of the original image and the target enhanced image. is the maximum possible value of the image pixel, and MSE is the mean square error between the original image and the target enhanced image. The specific calculation formula of MSE is:
[0176]
[0177] Among them, H ei and W ei are the height and width of the target enhanced image, respectively, end (m, o) and I in (m, o) are the pixel values of the target enhanced image and the original image at position (m, o), respectively.
[0178] The mean square error (PSNR) is used to measure the degree of image distortion. When the mean square error (PSNR) is less than the preset mean square error threshold (PSNR), the image will be yWhen it is determined that the distortion index of the target enhanced image does not meet the requirement, the modified neural network model in step S104 needs to be adjusted.
[0179] The formula for calculating color fidelity is:
[0180]
[0181] Among them, CF is the color fidelity of the original image and the target enhanced image, C end (m, o) and C in (m, o) are the color vectors of the target enhanced image and the original image at position (m, o), respectively. ei and W ei are the height and width of the target augmented image, respectively.
[0182] When the color fidelity CF is less than the preset color fidelity threshold CF y , it is determined that the color index of the target enhanced image does not meet the requirements, and the deep neural network in step S101 needs to be adjusted.
[0183] The calculation formula of edge preservation is:
[0184]
[0185] Among them, EB is the edge preservation degree of the original image and the target enhanced image, and are the gradient vectors of the target enhanced image and the original image at position (m, o), is the standard deviation of edge intensity, H ei and W ei are the height and width of the target augmented image, respectively.
[0186] When the edge preservation degree EB is less than the preset edge preservation degree threshold EB y , it is determined that the edge preservation index of the target enhanced image does not meet the requirements, and the attention mechanism in step S105 needs to be adjusted.
[0187] The steps provided by the above-mentioned image enhancement method based on multi-level feature fusion are to extract multi-level features of the original image by using a deep neural network, obtain a first enhanced image based on the multi-level features, calculate the complexity of the first enhanced image and the diversity measure of the multi-level features, adjust the feature extraction strategy of the deep neural network based on the complexity and diversity measure, and use the adjusted deep neural network to extract new multi-level features, thereby further improving the accuracy of multi-level feature extraction, and then obtain a second enhanced image by fusing the new multi-level features, and enhance the details and contrast of the image by effectively fusing the multi-level features, and then obtain a third enhanced image by inputting the second enhanced image into the modified neural network model, thereby further improving the brightness and clarity of the image, and then identify the key points of the third enhanced image by using the attention mechanism. region, the key region includes at least one of the following: an edge region, a texture region and a salient region, and the information of the key region is made more prominent by acquiring a fourth enhanced image. Finally, the fourth enhanced image is subjected to denoising and illumination normalization processing to obtain a target enhanced image, and the image details and contrast are further enhanced by resisting the influence of noise and illumination changes, and the stability and reliability of the image in different environments are improved. The quality of the target enhanced image is evaluated by calculating the similarity index, peak signal-to-noise ratio, color fidelity and edge retention of the original image and the target enhanced image, and a feedback adjustment mechanism is established to continuously optimize the image enhancement effect, thereby effectively improving the visual effect and image quality of the original low-light image, and at the same time, the original features of the low-light image can be fully retained, and the quality of the enhanced image is improved.
[0188] The following is a detailed description in conjunction with another embodiment.
[0189] Embodiment 2
[0190] An image enhancement device based on multi-level feature fusion provided in this embodiment includes multiple implementation units, and each implementation unit corresponds to each implementation step in the above-mentioned embodiment 1.
[0191] Figure 2 is a schematic diagram of an optional image enhancement device based on multi-level feature fusion according to an embodiment of the present invention, such as Figure 2 As shown, the image enhancement device based on multi-level feature fusion may include: an acquisition unit 20, an adjustment unit 21, a fusion unit 22, a correction unit 23, a recognition unit 24, and a processing unit 25.
[0192] The acquisition unit 20 is used to extract multi-level features of the original image using a deep neural network, and acquire a first enhanced image based on the multi-level features, wherein the multi-level features include depth information, color information and texture information of the original image.
[0193] The adjustment unit 21 is used to calculate the complexity of the first enhanced image and the diversity measurement value of the multi-level features, adjust the feature extraction strategy of the deep neural network based on the complexity and diversity measurement value, and use the adjusted deep neural network to extract new multi-level features.
[0194] The fusion unit 22 is used to perform fusion processing on the new multi-level features to obtain a second enhanced image, wherein the second enhanced image is a noise image.
[0195] The correction unit 23 is used to input the second enhanced image into the corrected neural network model to obtain a third enhanced image, wherein the corrected neural network model includes a LeakyReLU activation function.
[0196] The identification unit 24 is used to identify the key area of the third enhanced image by using the attention mechanism to obtain the fourth enhanced image, where the key area includes at least one of the following: an edge area, a texture area, and a salient area.
[0197] The processing unit 25 is used to perform denoising and illumination normalization processing on the fourth enhanced image to obtain a target enhanced image.
[0198] Optionally, the acquisition unit 20 includes: a first convolution module, used to input the original image into the first convolution layer in the deep neural network, perform a convolution operation on the original image and the convolution kernel weight matrix of the first convolution layer, and obtain a first feature based on the convolution operation result and the bias vector of the first convolution layer. The calculation formula of the first feature is:
[0199]
[0200] Among them, F1 is the first feature, I in is the original image, W1 is the convolution kernel weight matrix of the first convolution layer, * is the convolution operator, b1 is the bias vector of the first convolution layer, is an activation function; the second convolution module is used to input the first feature into the second convolution layer in the deep neural network, perform convolution operation on the first feature and the convolution kernel weight matrix of the second convolution layer, and obtain the second feature based on the convolution operation result and the bias vector of the second convolution layer. The calculation formula of the second feature is:
[0201]
[0202] Among them, F2 is the second feature, F1 is the first feature, W2 is the convolution kernel weight matrix of the second convolution layer, * is the convolution operator, b2 is the bias vector of the second convolution layer, is an activation function; a pooling module, used to input the second feature into the pooling layer in the deep neural network to obtain a pooling feature; a third convolution module, used to input the pooling feature into the third convolution layer in the deep neural network, perform a convolution operation on the pooling feature and the convolution kernel weight matrix of the third convolution layer, and obtain the third feature based on the convolution operation result and the bias vector of the third convolution layer. The calculation formula of the third feature is:
[0203]
[0204] Among them, F3 is the third feature, P is the pooling feature, W3 is the convolution kernel weight matrix of the third convolution layer, * is the convolution operator, and b3 is the bias vector of the third convolution layer. is the activation function.
[0205] Optionally, the acquisition unit 20 further includes: a deconvolution module, which is used to input the third feature into a deconvolution layer in the deep neural network, perform a convolution operation on the third feature and a deconvolution kernel weight matrix of the deconvolution layer, and obtain a deconvolution output feature based on the operation result and a bias vector of the deconvolution layer, wherein the size of the deconvolution output feature is the same as or similar to the size of the original image, and the calculation formula of the deconvolution output feature is:
[0206] Y=g(W u *F3+b u );
[0207] Among them, Y is the deconvolution output feature, F3 is the third feature, W u is the deconvolution kernel weight matrix of the deconvolution layer, * is the convolution operator, b u is the bias vector of the deconvolution layer, g(·) is a linear activation function; the conversion module is used to convert the deconvolution output features back to the image space of the original image to obtain the first enhanced image. The calculation formula of the first enhanced image is:
[0208]
[0209] Among them, Y is the deconvolution output feature, I ed is the first enhanced image, is the mapping function.
[0210] Optionally, the adjustment unit 21 includes: a first calculation module, which calculates an adaptive factor based on the complexity and diversity measurement values, and the calculation formula of the adaptive factor is:
[0211]
[0212] Among them, C fis the complexity, D is the diversity measure, and σ(·) is the adaptive function. The second calculation module adjusts the weight of each convolutional layer in the deep neural network based on the adaptive factor and multi-level features. The weight calculation formula is:
[0213]
[0214] Among them, W i is the weight of the i-th convolutional layer, is the adaptive factor, N is the number of convolutional layers, F i is the output feature of the i-th convolutional layer, F j is the output feature of the j-th convolutional layer, exp(·) is the natural exponential function, and β is a custom constant; the update module updates the deep neural network based on the adjusted weights, and uses the adjusted deep neural network to extract new multi-level features.
[0215] Optionally, the fusion unit 22 includes: a splicing module, which is used to splice the new multi-level features to obtain fused features; a reconstruction module, which is used to reconstruct the fused features, perform convolution operations on the fused features and the convolution kernel weights of the reconstruction layer, and obtain reconstructed features based on the convolution operation results and the bias vector of the reconstruction layer; a linear transformation module, which is used to transform the reconstructed features to obtain a second enhanced image.
[0216] Optionally, the recognition unit 24 includes: a first acquisition module, used to input the third enhanced image into a fully connected layer in the channel attention mechanism, and acquire the channel weight of the third enhanced image based on the output features of the fully connected layer. The calculation formula of the channel weight of the third enhanced image is:
[0217]
[0218] in, is the channel weight of the third enhanced image, z is the third enhanced image, W fc is the weight matrix of the fully connected layer, g(z, W fc ) represents the output features of the fully connected layer, and σ(·) is the activation function.
[0219] Optionally, the recognition unit 24 further includes: an encoding module, which uses a spatial attention mechanism to linearly encode the third enhanced image based on the third enhanced image and the channel weight to obtain a feature mapping function of the third enhanced image, and the feature mapping function is:
[0220]
[0221] in, is the feature mapping function, z is the third enhanced image, is the channel weight of the third enhanced image, Wf is the weight of linear encoding, b f is the bias vector of linear encoding; the second acquisition module obtains the spatial attention map based on the feature mapping function, and the calculation formula of the spatial attention map is:
[0222]
[0223] in, is the spatial attention map, z is the third enhanced image, is the channel weight of the third enhanced image, and softmax[·] is the activation function; the multiplication module is used to perform element-by-element multiplication operation on the spatial attention map and the third enhanced image to obtain the fourth enhanced image. The calculation formula of the element-by-element multiplication operation is:
[0224]
[0225] Wherein, Mix(z) is the fourth enhanced image, z is the third enhanced image, is the channel weight of the third enhanced image, is the spatial attention map, is the element-by-element multiplication symbol.
[0226] Optionally, the processing unit 25 includes: a denoising module, which is used to obtain all pixel values of the fourth enhanced image, and perform denoising processing on the fourth enhanced image based on the pixel values to obtain a denoised image of the fourth enhanced image, and the calculation formula for each pixel value in the denoised image is:
[0227] I end (m) = ∑ m,o∈Ω W(m,o)·I end (o);
[0228] Among them, I end (m) is the pixel value at position m in the denoised image, I end (o) is the pixel value at position o in the fourth enhanced image, W(m, o) is the weight of the denoising process, W(m, o) represents the similarity between the pixel value at position m and the pixel value at position o of the fourth enhanced image, Ω is the image domain of the fourth enhanced image, and the pixel value at position m and the pixel value at position o both belong to the image domain Ω; the normalization module performs illumination normalization processing on the denoised image based on the pixel values of all denoised images to obtain a target enhanced image, and the calculation formula for each pixel value in the target enhanced image is:
[0229]
[0230] Among them, I r (m) is the pixel value at position m in the target enhanced image, I end(m) is the pixel value at position m in the denoised image, and τ is a custom constant.
[0231] Optionally, the processing unit 25 further includes: an evaluation module, which calculates a similarity index, a peak signal-to-noise ratio, a color fidelity, and an edge preservation degree based on the original image and the target enhanced image to obtain an evaluation result of the target enhanced image.
[0232] The above-mentioned image enhancement device based on multi-level feature fusion can use a deep neural network to extract multi-level features of the original image through the acquisition unit 20, obtain a first enhanced image based on the multi-level features, calculate the complexity of the first enhanced image and the diversity measurement value of the multi-level features through the adjustment unit 21, adjust the feature extraction strategy of the deep neural network based on the complexity and diversity measurement values, and use the adjusted deep neural network to extract new multi-level features, thereby further improving the accuracy of multi-level feature extraction, and then obtain a second enhanced image by fusing the new multi-level features, and effectively fuse the multi-level features through the fusion unit 22 to enhance the details and contrast of the image, and then input the second enhanced image into the corrected neural network model through the correction unit 23 to obtain a third Enhance the image to further improve the brightness and clarity of the image, and then use the attention mechanism through the recognition unit 24 to identify the key area of the third enhanced image, the key area includes at least one of the following: edge area, texture area and salient area, and obtain the fourth enhanced image to make the information of the key area more prominent. Finally, the fourth enhanced image is denoised and illuminated by the processing unit 25 to obtain the target enhanced image. By resisting the influence of noise and illumination changes, the details and contrast of the image are further enhanced, the stability and reliability of the image in different environments are improved, and the visual effect and image quality of the original low-light image are effectively improved, thereby solving the technical problem that the original features of the image cannot be fully retained in the related technology, resulting in poor quality of the enhanced image.
[0233] The above-mentioned image enhancement device based on multi-level feature fusion may also include a processor and a memory. The above-mentioned acquisition unit 20, adjustment unit 21, fusion unit 22, correction unit 23, recognition unit 24, processing unit 25, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.
[0234] The above processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the low-light image can be enhanced by adjusting the kernel parameters to fully retain the original features of the image and ensure the quality of the enhanced image.
[0235] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.
[0236] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the image enhancement methods based on multi-level feature fusion in the above-mentioned embodiment 1.
[0237] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the image enhancement method based on multi-level feature fusion of any one of the above-mentioned embodiments one.
[0238] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image enhancement method based on multi-level feature fusion described in each embodiment of the present application.
[0239] The present application also provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image enhancement method based on multi-level feature fusion described in each embodiment of the present application are implemented.
[0240] Figure 3 1 is a hardware structure block diagram of an electronic device (or mobile device) according to an image enhancement method based on multi-level feature fusion according to an embodiment of the present invention. Figure 3 As shown, the electronic device may include one or more ( Figure 3 302a, 302b, ..., 302n are used to illustrate) processor 302 (processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), memory 304 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations shown.
[0241] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0242] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0243] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0244] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0245] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0246] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0247] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. An image enhancement method based on multi-level feature fusion, characterized in that: include: Using a deep neural network to extract multi-level features of the original image, and obtaining a first enhanced image based on the multi-level features, wherein the multi-level features include depth information, color information, and texture information of the original image; Calculating the complexity of the first enhanced image and the diversity metric of the multi-level features, adjusting the feature extraction strategy of the deep neural network based on the complexity and the diversity metric, and extracting new multi-level features using the adjusted deep neural network; Performing fusion processing on the new multi-level features to obtain a second enhanced image, wherein the second enhanced image is a noise image; Inputting the second enhanced image into a modified neural network model to obtain a third enhanced image, wherein the modified neural network model includes a LeakyReLU activation function; Using an attention mechanism to identify key areas of the third enhanced image to obtain a fourth enhanced image, wherein the key areas include at least one of the following: an edge area, a texture area, and a salient area; The fourth enhanced image is subjected to denoising and illumination normalization processing to obtain a target enhanced image.
2. The image enhancement method according to claim 1, characterized in that: The step of acquiring a first enhanced image based on the multi-level features comprises: The original image is input into the first convolution layer in the deep neural network, a convolution operation is performed on the original image and the convolution kernel weight matrix of the first convolution layer, and a first feature is obtained based on the convolution operation result and the bias vector of the first convolution layer. The calculation formula of the first feature is: Among them, F1 is the first feature, I in is the original image, W1 is the convolution kernel weight matrix of the first convolution layer, * is the convolution operator, b1 is the bias vector of the first convolution layer, is the activation function; The first feature is input into the second convolution layer in the deep neural network, a convolution operation is performed on the first feature and the convolution kernel weight matrix of the second convolution layer, and a second feature is obtained based on the convolution operation result and the bias vector of the second convolution layer. The calculation formula of the second feature is: Wherein, F2 is the second feature, F1 is the first feature, W2 is the convolution kernel weight matrix of the second convolution layer, * is the convolution operator, b2 is the bias vector of the second convolution layer, is the activation function; Inputting the second feature into a pooling layer in the deep neural network to obtain a pooling feature; The pooled feature is input into the third convolution layer in the deep neural network, a convolution operation is performed on the pooled feature and the convolution kernel weight matrix of the third convolution layer, and a third feature is obtained based on the convolution operation result and the bias vector of the third convolution layer. The calculation formula of the third feature is: Wherein, F3 is the third feature, P is the pooling feature, W3 is the convolution kernel weight matrix of the third convolution layer, * is the convolution operator, b3 is the bias vector of the third convolution layer, is the activation function.
3. The image enhancement method according to claim 2, characterized in that: The step of acquiring a first enhanced image based on the multi-level features further includes: The third feature is input into the deconvolution layer in the deep neural network, and a convolution operation is performed on the third feature and the deconvolution kernel weight matrix of the deconvolution layer. A deconvolution output feature is obtained based on the operation result and the bias vector of the deconvolution layer, wherein the size of the deconvolution output feature is the same as or similar to the size of the original image, and the calculation formula of the deconvolution output feature is: Y=g(W u *F3+b u ); Wherein, Y is the deconvolution output feature, F3 is the third feature, W u is the deconvolution kernel weight matrix of the deconvolution layer, * is the convolution operator, b u is the bias vector of the deconvolution layer, g(·) is the linear activation function; The deconvolution output feature is converted back into the image space of the original image to obtain the first enhanced image. The calculation formula of the first enhanced image is: Wherein, Y is the deconvolution output feature, I ed is the first enhanced image, is the mapping function.
4. The image enhancement method according to claim 1, characterized in that: The step of adjusting the feature extraction strategy of the deep neural network based on the complexity and the diversity metric value, and extracting new multi-level features using the adjusted deep neural network comprises: Based on the complexity and the diversity metric value, an adaptive factor is calculated, and the calculation formula of the adaptive factor is: Among them, C f is the complexity, D is the diversity measure, σ(·) is the adaptive function; Based on the adaptive factor and the multi-level features, the weight of each convolutional layer in the deep neural network is adjusted, and the weight calculation formula is: Among them, W i is the weight of the i-th convolutional layer, is the adaptive factor, N is the number of convolutional layers, F i is the output feature of the i-th convolutional layer, F j is the output feature of the j-th convolutional layer, exp(·) is the natural exponential function, and β is a custom constant; The deep neural network is updated based on the adjusted weights, and new multi-level features are extracted using the adjusted deep neural network.
5. The image enhancement method according to claim 1, characterized in that: The step of fusing the new multi-level features to obtain a second enhanced image comprises: Performing splicing processing on the new multi-level features to obtain fused features; Reconstructing the fused features, performing convolution operation on the fused features and the convolution kernel weights of the reconstruction layer, and obtaining the reconstructed features based on the convolution operation result and the bias vector of the reconstruction layer; The reconstructed features are transformed to obtain the second enhanced image.
6. The image enhancement method according to claim 1, characterized in that: When an attention mechanism is used to identify the key area of the third enhanced image, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the steps of the channel attention mechanism include: The third enhanced image is input into the fully connected layer in the channel attention mechanism, and the channel weight of the third enhanced image is obtained based on the output features of the fully connected layer. The calculation formula of the channel weight of the third enhanced image is: in, is the channel weight of the third enhanced image, z is the third enhanced image, W fc is the weight matrix of the fully connected layer, g(z, W fc ) represents the output feature of the fully connected layer, and σ(·) is the activation function.
7. The image enhancement method according to claim 6, characterized in that: When an attention mechanism is used to identify the key area of the third enhanced image, the attention mechanism includes a channel attention mechanism and a spatial attention mechanism, and the steps of the spatial attention mechanism include: Based on the third enhanced image and the channel weight, the third enhanced image is linearly encoded using a spatial attention mechanism to obtain a feature mapping function of the third enhanced image, where the feature mapping function is: in, is the feature mapping function, z is the third enhanced image, is the channel weight of the third enhanced image, W f is the weight of the linear encoding, b f is the bias vector of the linear encoding; Based on the feature mapping function, a spatial attention map is obtained, and the calculation formula of the spatial attention map is: in, is the spatial attention map, z is the third enhanced image, is the channel weight of the third enhanced image, and softmax[·] is the activation function; An element-by-element multiplication operation is performed on the spatial attention map and the third enhanced image to obtain the fourth enhanced image. The calculation formula of the element-by-element multiplication operation is: Wherein, Mix(z) is the fourth enhanced image, z is the third enhanced image, is the channel weight of the third enhanced image, is the spatial attention map, is the element-wise multiplication symbol.
8. The image enhancement method according to claim 1, characterized in that: The step of sequentially performing denoising and illumination normalization processing on the fourth enhanced image to obtain a target enhanced image comprises: All pixel values of the fourth enhanced image are obtained, and based on the pixel values, the fourth enhanced image is denoised to obtain a denoised image of the fourth enhanced image, wherein a calculation formula for each pixel value in the denoised image is: I end (m)=∑ m,o∈Ω W(m,o)·I end (o); Among them, I end (m) is the pixel value at position m in the denoised image, I end (o) is the pixel value at position o in the fourth enhanced image, W(m, o) is the weight of the denoising process, W(m, o) represents the similarity between the pixel value at position m and the pixel value at position o of the fourth enhanced image, Ω is the image domain of the fourth enhanced image, and the pixel value at position m and the pixel value at position o both belong to the image domain Ω; Based on the pixel values of all the denoised images, the denoised images are subjected to illumination normalization processing to obtain the target enhanced image. The calculation formula for each pixel value in the target enhanced image is: Among them, I r (m) is the pixel value at position m in the target enhanced image, I end (m) is the pixel value at position m in the denoised image, and τ is a custom constant.
9. The image enhancement method according to claim 1, characterized in that: After obtaining the target enhanced image, it includes: Based on the original image and the target enhanced image, a similarity index, a peak signal-to-noise ratio, a color fidelity, and an edge preservation degree are calculated to obtain an evaluation result of the target enhanced image.
10. An image enhancement device based on multi-level feature fusion, characterized in that: include: An acquisition unit, configured to extract multi-level features of an original image using a deep neural network, and acquire a first enhanced image based on the multi-level features, wherein the multi-level features include depth information, color information, and texture information of the original image; an adjustment unit, configured to calculate the complexity of the first enhanced image and a diversity metric of the multi-level features, adjust the feature extraction strategy of the deep neural network based on the complexity and the diversity metric, and extract new multi-level features using the adjusted deep neural network; A fusion unit, used for fusing the new multi-level features to obtain a second enhanced image, wherein the second enhanced image is a noise image; A correction unit, configured to input the second enhanced image into a correction neural network model to obtain a third enhanced image, wherein the correction neural network model includes a LeakyReLU activation function; an identification unit, configured to identify a key area of the third enhanced image by using an attention mechanism to obtain a fourth enhanced image, wherein the key area includes at least one of the following: an edge area, a texture area, and a salient area; The processing unit is used to perform denoising and illumination normalization processing on the fourth enhanced image to obtain a target enhanced image.
