Image enhancement processing method and system based on prior information estimation

Abstract

The invention relates to the field of image processing, in particular to an image enhancement processing method and system based on prior information estimation, and the method comprises the steps: selecting an original image, and recording the size of the original image; carrying out noise estimation on the original image, and making a noise level image with the same size; calculating the contrast of the original image, and making contrast horizontal images with the same size; taking the noise level image and the contrast level image as prior information, and superposing the prior information with the original image to obtain a new image; performing feature extraction on the new image to obtain a feature image; aiming at the feature image, extracting a plurality of feature maps with different scales; respectively carrying out image enhancement processing on each feature map; and integrating the enhanced feature maps to obtain a final enhanced image. According to the image enhancement processing method, the definition and detail expression of the image can be improved, the noise is effectively removed, the detail information of the image is reserved, and the image quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image enhancement processing method and system based on prior information estimation. Background Art

[0002] The statements in this section are merely intended to provide background information related to the technical solution of the present application to aid understanding, and they do not necessarily constitute prior art for the technical solution of the present application.

[0003] Image processing is an important branch of detection technology and has been successfully applied in military, security, medical, industrial and other fields. However, due to the limitations of manufacturing processes and manufacturing costs, many images usually have low contrast, low resolution and low signal-to-noise ratio, which limits their further application in high-end fields. Images can be enhanced to improve image quality, but due to the complex causes of image noise and inconsistent intensity, the generalization performance of existing methods is poor and they are only suitable for enhancement tasks in specific scenes; secondly, many images have low contrast, blurred details and large dynamic range, and the original image details are easily lost after enhancement. Summary of the invention

[0004] In view of the above problems, the present invention proposes an image enhancement processing method and system based on prior information estimation.

[0005] One aspect of the present application relates to an image enhancement processing method based on prior information estimation, comprising: selecting an original image to be processed and recording the size of the original image; performing noise estimation on the original image, obtaining a noise level value, and producing a noise level image with the same size as the original image; calculating the contrast of the original image, and producing a contrast level image with the same size as the original image; superimposing the noise level image and the contrast level image as prior information with the original image to obtain a new image; performing feature extraction on the new image to obtain a feature image; extracting a plurality of feature maps of different scales for the feature image; performing image enhancement processing on each of the plurality of feature maps of different scales respectively; integrating the enhanced plurality of feature maps of different scales to obtain a final enhanced image.

[0006] In one embodiment, the noise estimation of the original image includes: using a noise estimation network to perform noise estimation on the original image, the noise estimation network includes multiple residual modules and a maximum pooling layer, and finally outputting the result through a global average pooling layer.

[0007] In one embodiment, extracting features from the new image includes: inputting the new image into a multi-core convolutional feature extraction network with multiple branches.

[0008] In one embodiment, the multi-core convolutional feature extraction network has three parallel branches, each branch uses convolution kernels of different scales to perform convolution operations to obtain feature maps of different receptive fields, and then calculates the weight of each branch through an adaptive attention mechanism, multiplies the feature maps of each branch by their corresponding weights and adds them together to obtain the final output feature image.

[0009] In one embodiment, extracting a plurality of feature maps of different scales from the feature image includes: extracting a plurality of feature maps of different scales using average pooling of different scales on the feature image.

[0010] In one embodiment, the image enhancement processing is performed on each feature map of the multiple feature maps of different scales respectively, including: each feature map X is input into two branches, in the first branch, the feature map X is first expanded through two convolutional layers to obtain a feature map U, and then the weight of each channel is calculated through an attention mechanism, and the feature map U is weighted to obtain a weighted feature map U′, in the second branch, the feature map X is expanded to a feature map X′ with the same number of channels as the feature map U′ by using a 1×1 convolutional layer, and finally the feature map U′ and the feature map X′ are added and input into the activation function layer to achieve image enhancement processing.

[0011] In one embodiment, integrating the enhanced multiple feature maps of different scales to obtain the final enhanced image includes: using upsampling to restore the enhanced multiple feature maps of different scales to the same size as the original image, then integrating features of different scales through a channel superposition operation, and finally obtaining the final enhanced image through convolution.

[0012] In one embodiment, the image is an infrared image.

[0013] Another aspect of the present application relates to an image enhancement processing system based on prior information estimation, including a memory and a processor, wherein the memory stores a computer program, and the processor can run the computer program to execute the image enhancement processing method of the present application.

[0014] In one embodiment, the image enhancement processing system further includes an image acquisition device for acquiring original images.

[0015] By adopting the scheme of the present application, the noise level estimation and contrast level estimation are used as the prior information of the image, and the image enhancement task of different scenes is adaptively performed according to the noise severity level and the contrast level, and the image details are reconstructed on the multi-scale feature map to improve the clarity and detail expression of the image, thereby effectively removing noise, retaining the image detail information, and improving the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0017] Figure 1 The specific process of an infrared image enhancement processing method based on prior information estimation according to an embodiment is shown;

[0018] Figure 2 shows a noise estimation network according to one embodiment;

[0019] Figure 3 A multi-core convolutional feature extraction network with three branches according to one embodiment is shown;

[0020] Figure 4 shows an ARU-Net network structure diagram according to one embodiment;

[0021] Figure 5 A diagram showing an AR block network structure according to one embodiment is shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] In the following, the image enhancement processing method of the present application is described by taking a preferred infrared image as an example, but it can be understood that the image enhancement processing method of the present invention is also applicable to other types of images.

[0024] Figure 1 The specific process of an infrared image enhancement processing method based on prior information estimation according to an embodiment is shown, which selects the original infrared image to be processed and records the size of the original infrared image. Pure noise images of different types and different intensities of the same size as the original infrared image are produced. The types of pure noise images include but are not limited to non-uniform noise, Gaussian noise, salt and pepper noise, and mixtures of these noises. The intensities of non-uniform noise, Gaussian noise, and salt and pepper noise are recorded respectively. An image containing single noise or mixed noise is superimposed with the original infrared image, that is, the grayscale values ​​of the pixels of the two images are correspondingly added to obtain a noisy infrared image. The noisy infrared image and the clean infrared image constitute a training data pair for training and testing a deep learning model for infrared image enhancement processing. The method specifically includes:

[0025] (1) Select an original infrared image to be processed and record the size of the original infrared image.

[0026] In one embodiment, an infrared image may be selected from the infrared image data set as an original infrared image to be processed, and the size of the original infrared image may be recorded.

[0027] (2) Estimating the noise of the original infrared image, obtaining the noise level value, and producing a noise level image with the same size as the original infrared image.

[0028] In one embodiment, the noise estimation of the original infrared image to be processed can be achieved by a noise estimation network. In one embodiment, the network structure of the noise estimation network is as follows: Figure 2 As shown in the figure, the network includes multiple residual modules and maximum pooling layers, and finally outputs the result through a global average pooling layer. The output of the noise estimation network is the peak signal-to-noise ratio (PSNR). The smaller the PSNR value, the more serious the image is affected by noise, and the lower the noise level of the noisy image. The output peak signal-to-noise ratio can be normalized. Then, a noise level image with the same size as the original infrared image is produced. In the noise estimation network, the network training can be performed for the purpose of estimating the peak signal-to-noise ratio (PSNR) between the noisy infrared image and the clean infrared image.

[0029] (3) Calculate the contrast of the original infrared image and create a contrast level image with the same size as the original infrared image.

[0030] In one embodiment, the contrast of the original infrared image may be calculated and normalized, and then a contrast level image having the same size as the original infrared image may be produced.

[0031] (4) The original infrared image, the noise level image, and the contrast level image are superimposed to form a new infrared image.

[0032] The noise level image and the contrast level image are used as prior information and superimposed with the original infrared image to form a new infrared image.

[0033] (5) Perform feature extraction on the new infrared image to obtain a feature image.

[0034] In one embodiment, the new infrared image is subjected to feature extraction. Specifically, the new infrared image is input into a multi-core convolutional feature extraction network having multiple branches. The multi-core convolutional feature extraction network may have three parallel branches, such as Figure 3 As shown in the figure, each branch uses convolution kernels of different scales to perform convolution operations to obtain feature maps of different receptive fields. Subsequently, the weight of each branch is calculated through the adaptive attention mechanism, and the feature maps of each branch are multiplied by their corresponding weights and added together to obtain the final output feature image.

[0035] In one embodiment, the new infrared image I∈R after superposition is inputH×W×2 (where H represents the height of the image, W represents the width of the image, and 2 represents that the image has two channels). After three parallel convolution branches with kernel sizes of 3, 5, and 7, three feature maps A'∈R with different receptive field sizes are obtained. H×W×C ,A”∈R H×W×C , and A”′∈R H×W×C (Where H represents the height of the image, W represents the width of the image, and C represents the number of image channels. Similar expressions are used to represent similar meanings in the following text.) Then, the feature map A∈R is obtained by adding and summing the corresponding channels. H×W×32 , to integrate the feature information of the three branches. Perform global average pooling (GAP) on each channel of the feature map after fusion information to obtain the feature map M∈R 1×1×C ,for

[0036] In order to reduce redundant information, M is compressed to obtain the feature map N∈R 1×1×C' :

[0037] A=A'+A"+A"' (1)

[0038] N=FC(GAP(A)) (2)

[0039] Then we start to calculate the weights corresponding to the three feature maps, and expand the compressed feature map N using three fully connected layers FC1, FC2 and FC3 with different parameters to obtain feature maps a∈R of the same size 1×1×C , b∈R 1×1×C and c∈R 1 ×1×C The values ​​on the same channel of the three feature maps are input into the softmax function to obtain three feature maps containing weight values. Finally, they are weighted summed with the initial A', A", and A'' to obtain the final multi-core fusion feature image V:

[0040]

[0041] V i =a' i ·A'+b' i ·A”+c' i A”' (4)

[0042] Among them, μ i (u=a,b,c) represents the eigenvalue of the corresponding feature map after the i-th channel, u' i Represents the eigenvalue of the ith channel of the corresponding feature map after softmax, V i Represents the value of the feature image V on the i-th channel after weighted summation.

[0043] (6) For the feature image, extract multiple feature maps of different scales.

[0044] For the feature image V, average pooling of different scales (including 1×1, 2×2, 4×4, 5×5 and 16×16, etc.) can be used to extract multi-scale feature maps.

[0045] (7) Performing image enhancement processing on each feature map of the multiple feature maps of different scales respectively.

[0046] In one embodiment, performing image enhancement processing on each feature map includes: inputting each feature map X into two branches, in the first branch, the feature map X is firstly expanded through two convolutional layers to obtain a feature map U, and then the weight of each channel is calculated through an attention mechanism, and the feature map U is weighted to obtain a weighted feature map U′, and in the second branch, a 1×1 convolutional layer is used to expand the feature map X to a feature map X′ with the same number of channels as the feature map U′, and finally the feature map U′ and the feature map X′ are added and input into the activation function layer to realize image enhancement processing.

[0047] In one embodiment, each scale feature map can be input into the ARU-Net network proposed in the present invention for enhancement processing. The ARU-Net network structure according to one embodiment is as follows: Figure 4 As shown, Figure 4 The AR block network structure in Figure 5 As shown in Figure 2. After the feature map is input to the AR block, it passes through two branches, one for calculating the channel attention mechanism and the other as the identity mapping in the residual network. The feature maps of the two branches are summed at the output. The identity mapping branch uses a 1×1 convolutional layer to expand the feature map X to a feature map X′ with the same number of channels as the feature map U′.

[0048] (8) Integrate the enhanced feature maps of multiple different scales to obtain the final enhanced infrared image.

[0049] In one embodiment, upsampling is used to restore the enhanced feature maps of multiple different scales to the same size as the original infrared image, and then the features of different scales are integrated through a channel superposition operation, and finally a final enhanced infrared image is obtained through convolution.

[0050] In one embodiment, the present application provides an image enhancement processing system based on prior information estimation, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is capable of running the computer program to execute the image enhancement processing method described in the present application. In one embodiment, the image enhancement processing system also includes an image acquisition device for acquiring an original image.

[0051] The image enhancement processing method based on prior estimation of the present application uses noise level estimation and contrast level estimation as prior information of the image, and according to the noise severity level and contrast level, it adapts to the image enhancement task of different scenes. In addition, the image details are reconstructed on the multi-scale feature map to improve the clarity and detail expression of the image. This method can effectively improve the robustness and generalization performance of image enhancement processing. By combining the estimated image prior information with the deep learning algorithm, the present invention can effectively remove noise, retain image detail information, and improve image quality.

[0052] References herein to "various embodiments", "some embodiments", "one embodiment", or "an embodiment" refer to specific features, structures, or properties described in conjunction with the embodiments being included in at least one embodiment. Therefore, the phrases "in various embodiments", "in some embodiments", "in one embodiment", or "in an embodiment" appearing throughout the present invention do not necessarily refer to the same embodiment. In addition, specific features, structures, or properties may be combined in any suitable manner in one or more embodiments. Therefore, specific features, structures, or properties shown or described in conjunction with one embodiment may be combined in whole or in part with features, structures, or properties of one or more other embodiments without restriction, as long as the combination is not illogical or inoperable. Expressions similar to "according to A", "based on A", "through A", or "using A" appearing herein are intended to be non-exclusive, that is, "according to A" may include "according to A only" or "according to A and B", unless it is specifically stated that its meaning is "according to A only". In this application, for the sake of clarity, some exemplary operating steps are described in a certain order, but those skilled in the art will understand that each of these operating steps is not essential, and some of the steps may be omitted or replaced by other steps. These operation steps do not have to be executed sequentially in the manner shown; on the contrary, some of these operation steps can be executed in different orders or in parallel according to actual needs, as long as the new execution manner is not illogical or inoperable.

[0053] Having thus described several aspects of at least one embodiment of the present invention, it will be appreciated that various changes, modifications and improvements will readily occur to those skilled in the art. Such changes, modifications and improvements are intended to be within the spirit and scope of the present invention. Although the present invention has been described by way of some embodiments, the present invention is not limited to the embodiments described herein, and also includes various changes and variations made without departing from the scope of the present invention.

Claims

1. An image enhancement processing method based on prior information estimation, comprising: Selecting an original image to be processed, and recording the size of the original image; Perform noise estimation on the original image, obtain the noise level value, and create a noise level image with the same size as the original image; Calculate the contrast of the original image and make an image with the same contrast level as the original image; Using the noise level image and the contrast level image as prior information to superimpose on the original image to obtain a new image; Performing feature extraction on the new image to obtain a feature image; For the feature image, extracting a plurality of feature maps of different scales; Performing image enhancement processing on each of the feature maps of the multiple different scales respectively; The enhanced feature maps of multiple different scales are integrated to obtain the final enhanced image.

2. The image enhancement processing method according to claim 1, wherein: The noise estimation of the original image comprises: The noise estimation network is used to perform noise estimation on the original image. The noise estimation network includes multiple residual modules and a maximum pooling layer, and finally outputs the result through a global average pooling layer.

3. The image enhancement processing method according to claim 1, wherein: Extracting features from the new image includes: The new image is input into a multi-core convolutional feature extraction network with multiple branches.

4. The image enhancement processing method according to claim 3, wherein: The multi-core convolutional feature extraction network has three parallel branches. Each branch uses convolution kernels of different scales to perform convolution operations to obtain feature maps of different receptive fields. The weight of each branch is then calculated through an adaptive attention mechanism. The feature maps of each branch are multiplied by their corresponding weights and then added to obtain the final output feature image.

5. The image enhancement processing method according to claim 1, wherein: The extracting a plurality of feature maps of different scales from the feature image comprises: Average pooling of different scales is used on the feature image to extract multiple feature maps of different scales.

6. The image enhancement processing method according to claim 1, wherein: The performing image enhancement processing on each of the feature maps of the multiple feature maps of different scales comprises: Each feature map X is input into two branches. In the first branch, the feature map X first passes through two convolutional layers to expand the number of channels to obtain the feature map U. Then, the weight of each channel is calculated through the attention mechanism, and the feature map U is weighted to obtain the weighted feature map U′. ​​In the second branch, a 1×1 convolutional layer is used to expand the feature map X to a feature map σ′ with the same number of channels as the feature map U′. ​​Finally, the feature map U′ and the feature map σ′ are added and input into the activation function layer to realize image enhancement processing.

7. The image enhancement processing method according to claim 1, wherein: The step of integrating the enhanced feature maps of multiple different scales to obtain the final enhanced image includes: The enhanced feature maps of multiple different scales are restored to the same size as the original image by upsampling, and then the features of different scales are integrated through channel superposition operation, and finally the final enhanced image is obtained through convolution.

8. The image enhancement processing method according to claim 1, wherein: The image is an infrared image.

9. An image enhancement processing system based on prior information estimation, comprising a memory and a processor, wherein the memory stores a computer program, and the processor can run the computer program to execute the image enhancement processing method according to any one of claims 1 to 8.

10. The image enhancement processing system according to claim 9, further comprising an image acquisition device for acquiring original images.

Images