Infrared image enhancement method based on pseudo noise and convolutional neural network

Abstract

The invention provides an infrared image enhancement method based on pseudo noise and a convolutional neural network, and the method comprises the steps: collecting high-definition visible light image data, carrying out the cleaning, graying and fuzzification of the collected image data, adding pseudo noise, and building an initial data set; performing data enhancement on the obtained initial data set, and establishing a sample data set comprising a training set, a verification set and a test set; constructing an infrared image enhancement network based on a convolutional neural network, and training and optimizing the network by using the sample data set to obtain an optimal infrared image enhancement model; dividing an actually collected infrared image into three channels, inputting the three channels into the infrared image enhancement model for enhancement, and merging the results of the three channels to obtain a final infrared enhanced image. The method can effectively remove the noise of the infrared image, enhances the target features in the infrared image, and enables the high-quality infrared image to be obtained under a complex environment condition.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an infrared image enhancement method based on pseudo noise and convolutional neural network. The method can be applied to various scenarios requiring high-definition infrared images, such as military reconnaissance, security monitoring, medical imaging, traffic monitoring, environmental monitoring, etc. Background Art

[0002] In traditional infrared image processing methods, due to the limitations of infrared imaging equipment, the collected infrared images are often interfered by various noises, resulting in a decrease in image quality. In addition, due to environmental factors and equipment performance limitations, the resolution and contrast of the image often cannot meet the needs of practical applications.

[0003] Therefore, how to effectively remove noise and improve image resolution and contrast is an important topic in the field of infrared image processing. Summary of the invention

[0004] In order to solve the problem that the infrared image processing methods in the prior art cannot meet the needs of practical applications, the purpose of the present invention is to provide an infrared image enhancement method based on pseudo-noise and convolutional neural network, which can effectively remove the noise of the infrared image and enhance the target features in the infrared image, so that high-quality infrared images can be obtained even under complex environmental conditions.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] An infrared image enhancement method based on pseudo noise and convolutional neural network comprises the following steps:

[0007] Collect high-definition visible light image data, clean, grayscale, blur and add pseudo-noise to the collected image data to establish an initial data set; when adding pseudo-noise to the image data, extract noise patches from the grayscale image of the uniform background intensity area, then randomly enhance the noise patches, and add them to the blurred image after mean subtraction and resizing to generate image data with pseudo-noise, and use the pseudo-noise image and the corresponding clear image as the image pair of the initial data set;

[0008] Perform data enhancement on the obtained initial data set and establish a sample data set including training set, validation set and test set;

[0009] Construct an infrared image enhancement network based on convolutional neural network, and use sample data sets to train and optimize the network to obtain the optimal infrared image enhancement model;

[0010] The actually collected infrared image is divided into three channels and input into the infrared image enhancement model for enhancement. The results of the three channels are merged to obtain the final infrared enhanced image.

[0011] According to an infrared image enhancement method based on pseudo-noise and convolutional neural network provided by the present invention, when collecting visible light image data, a camera is used to collect visible light image data containing target and environmental features, and the collected image data is cleaned to screen out images with clear targets; then the screened images are grayed to become single-channel images; then blurring is performed to reduce the resolution of the image; then, pseudo-noise is added to the blurred image to generate an image with noise; finally, the image with noise and the corresponding image before blurring are used as an image pair of a data set to establish an initial data set.

[0012] According to an infrared image enhancement method based on pseudo noise and convolutional neural network provided by the present invention, data enhancement is performed on the acquired initial data set, including:

[0013] The initial data set is enhanced by flipping, scaling, and random angle rotation, and the initial data set is divided into training set, validation set, and test set in a ratio of 8:1:1 to establish a sample data set.

[0014] According to an infrared image enhancement method based on pseudo noise and convolutional neural network provided by the present invention, when the image data is fuzzy processed, the bicubic interpolation method is used to resample the screened image, and the expression formula of the bicubic interpolation method is as follows:

[0015]

[0016] Where I(x,y) is the pixel value of the original image at point (x,y), i and j are the displacements in the horizontal and vertical directions respectively, I′(x,y) is the pixel value of the interpolated image at point (x,y), and w ij is the weight of the convolution kernel at position (i, j), and the convolution kernel w i It is expressed as the following formula:

[0017]

[0018] According to an infrared image enhancement method based on pseudo noise and convolutional neural network provided by the present invention, when adding pseudo noise to image data, the method specifically includes the following steps:

[0019] Determine the uniform background intensity area in the image, that is, the area where there is no target or the environment around the target is relatively uniform;

[0020] In the determined uniform background intensity area, some small areas are extracted as noise patches, which only contain background noise areas and do not contain any target information;

[0021] Perform random enhancement operations on the extracted noise patches, including random flipping, rotation, and brightness adjustment;

[0022] After mean subtraction and resizing, it is added to the blurred image to produce an image with noise patches, i.e., a pseudo-noise image;

[0023] The pseudo noise image and the corresponding clear image are used as the image pair of the data set to establish an initial data set; wherein, the pseudo noise image is used as the input of the model, and the clear grayscale image is used as the label corresponding to the input, so as to establish the initial data set.

[0024] According to an infrared image enhancement method based on pseudo noise and convolutional neural network provided by the present invention, the process of extracting noise patches is expressed as the following formula:

[0025] P i =I′(x i :x i +w i ,y j :y j +h j )

[0026] Among them, P i represents the noise patch, I′(x,y) represents the pixel value of the blurred image at position (x,y), and I′(x i :x i +w i ,y j :y j +h j ) represents a sub-image extracted from a grayscale image, and the coordinates of its upper left corner are (x i ,y j ), the coordinates of the lower right corner are (x i +w i ,y j +h j ).

[0027] According to the infrared image enhancement method based on pseudo noise and convolutional neural network provided by the present invention, when generating an image with a noise patch, first calculate P i The average value μ i , v i The calculation is expressed as the following formula:

[0028]

[0029] Where N is the number of pixels in the noise patch, and w i and h j are the width and height of the noise patch, respectively;

[0030] Then, the mean value is subtracted from each pixel value to obtain the mean-subtracted noise patch P i ,, expressed as the following formula:

[0031] P i ,(x,y)=P i (x,y)-μ i .

[0033] According to an infrared image enhancement method based on pseudo noise and convolutional neural network provided by the present invention, when constructing an infrared image enhancement network, a denoising network and a contrast enhancement network are included. For the denoising network, in addition to the first two convolution layers, the network is composed of alternating convolution (Conv) layers and rectified linear units (ReLU), and there are 7 Conv layers and 5 ReLU layers in the network;

[0034] The Conv layer performs linear convolution on each input feature map tensor or input image patch tensor to generate an output feature map. The convolution operation in each Conv layer retains the height and width of the input feature map to the layer through the correct padding method. Finally, the residual learning technique is used to enhance the denoising performance by adding input to estimate the noise residual.

[0035] For the contrast enhancement network, the network is composed of 7 Conv layers, 9 ReLU layers, 1 GAP, 2 Dense layers and 2 sigmoid functions, among which Conv-8 acts on a single pixel and supplements the detailed features of the model. The next four convolutional layers Conv-9, Conv-10, Conv-11 and Conv-12 extract coarse-grained features from different spatial scales. Conv-11 and Conv-12 use dilated convolution to expand the receptive field without increasing the computational cost. Finally, these multi-scale complementary features are fused with the following Conv-13.

[0036] According to an infrared image enhancement method based on pseudo noise and convolutional neural network provided by the present invention, the following steps are further performed:

[0037] The target attention unit is added after the Conv-13 convolutional layer to further refine the output features from the spatial attention branch and the channel attention branch respectively;

[0038] Among them, the spatial attention branch acts as a pattern selector to suppress clutter background and irrelevant features in the spatial domain to highlight target details and suppress oversaturation effects;

[0039] Among them, the channel attention branch is used as a feature selector and a heavy weight feature to emphasize the effective channel features. It includes a global average pool and 2 fully connected layers. The final output of the target attention unit can be expressed as the following formula:

[0040] Y=M*XW

[0041] Among them, X is the output of the Conv-13 convolutional layer, M represents the spatial mask, W represents the channel weight, represents the pixel-level multiplication, and represents the tensor product.

[0042] According to an infrared image enhancement method based on pseudo-noise and convolutional neural network provided by the present invention, when training and optimizing the infrared image enhancement network, the enhanced image is compared with the target clear image, and the model results are evaluated using standard peak signal-to-noise ratio and structural similarity metrics to quantify the difference between the enhanced image and the target image, thereby obtaining the optimal infrared image enhancement model.

[0043] It can be seen that compared with the prior art, the method provided by the present invention has the following beneficial effects:

[0044] 1. The present invention can generate a large amount of image data with pseudo-noise by extracting noise patches from the grayscale image of the uniform background intensity area, performing random enhancement, mean subtraction and resizing, and then adding them to the image. These pseudo-noise images not only enrich the diversity of the data set, but also make the data set closer to the noise conditions that may exist in actual infrared images, thereby improving the generalization ability of the model.

[0045] 2. The present invention further enhances the initial data set, including operations such as rotation, scaling, and cropping, to generate more training samples, which helps the model learn more features and improve the robustness and accuracy of the model.

[0046] 3. The present invention uses sample data sets to train and optimize the CNN network to obtain the optimal infrared image enhancement model, which can learn how to map the input infrared image to a clearer and higher contrast output image.

[0047] 4. The present invention divides the actually collected infrared image into three channels and inputs them into the infrared image enhancement model, which can make full use of the grayscale information and possible pseudo-color information of the infrared image to improve the enhancement effect of the model. The present invention combines the output results of the three channels to obtain the final infrared enhanced image, which can ensure that the enhanced image has higher clarity and contrast while retaining the original information.

[0048] 5. The infrared image enhanced by this method is clearer in visual effect and has higher contrast, which is conducive to subsequent feature extraction, recognition or tracking. Since this method adds pseudo noise to the data set, the trained model can better cope with various noise interferences that may exist in infrared images in practical applications, improving the adaptability and stability of the model.

[0049] Therefore, the infrared image enhancement algorithm based on pseudo-noise and convolutional neural network in the present invention enhances the infrared image containing infrared targets, resamples the image through bicubic interpolation, adds noise patches to the resampled image, designs denoising network and contrast enhancement network, so that the algorithm can effectively remove the noise of the infrared image while enhancing the contrast of the infrared image, so that high-quality infrared images can be obtained even under complex environmental conditions.

[0050] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of an embodiment of an infrared image enhancement method based on pseudo noise and convolutional neural network of the present invention.

[0052] Figure 2 It is a schematic diagram of a visible light image with a clear target according to an embodiment of an infrared image enhancement method based on pseudo-noise and convolutional neural network of the present invention.

[0053] Figure 3 The present invention is about an infrared image enhancement method based on pseudo noise and convolutional neural network embodiment Figure 2 Grayscale schematic diagram.

[0054] Figure 4 The present invention is about an infrared image enhancement method based on pseudo noise and convolutional neural network embodiment Figure 3 Schematic diagram after blurring.

[0055] Figure 5 The present invention is about an infrared image enhancement method based on pseudo noise and convolutional neural network embodiment Figure 4 Schematic diagram of the image after adding pseudo noise.

[0056] Figure 6 It is a schematic diagram of a DCEN network constructed in an embodiment of an infrared image enhancement method based on pseudo-noise and convolutional neural network of the present invention.

[0057] Figure 7 It is a schematic diagram of infrared images before and after enhancement according to an embodiment of an infrared image enhancement method based on pseudo noise and convolutional neural network of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] See also Figures 1 to 5 The present invention provides an infrared image enhancement method based on pseudo noise and convolutional neural network, the method comprising the following steps:

[0061] Step S1, collecting high-definition visible light image data, cleaning, graying, blurring and adding pseudo-noise to the collected image data to establish an initial data set; wherein, when adding pseudo-noise to the image data, extracting noise patches from the grayscale image of the uniform background intensity area, then randomly enhancing the noise patches, and adding them to the blurred image after mean subtraction and resizing to generate image data with pseudo-noise, and taking the pseudo-noise image and the corresponding clear image as an image pair of the initial data set;

[0062] Step S2, performing data enhancement on the acquired initial data set, and establishing a sample data set including a training set, a validation set, and a test set;

[0063] Step S3, constructing an infrared image enhancement network based on a convolutional neural network, and using a sample data set to train and optimize the network to obtain an optimal infrared image enhancement model;

[0064] Step S4, dividing the actually collected infrared image into three channels and inputting them into the infrared image enhancement model for enhancement, and merging the results of the three channels to obtain a final infrared enhanced image.

[0065] In the above step S1, when collecting visible light image data, the following steps are specifically included:

[0066] Step S11, using a camera to collect visible light image data containing target and environmental features, and cleaning the collected image data to filter out images with clear targets;

[0067] Step S12, graying the filtered image to make it a single-channel image; then blurring it to reduce the resolution of the image;

[0068] Step S13, adding pseudo noise to the blurred image to generate an image with noise;

[0069] Step S14, taking the noisy image and the corresponding image before blurring as an image pair of a data set to establish an initial data set.

[0070] In the above step S2, when establishing the sample data set, it includes:

[0071] Step S21, data enhancement is performed on the acquired initial data set, including horizontal flipping, upside down flipping, random scaling and random angle rotation, where random scaling refers to scaling the image by a random multiple in the interval [0,0.5], and random angle selection refers to rotating the image by a random angle in the interval [-20°,20°].

[0072] Step S22, divide the initial data set into a training set, a validation set and a test set in a ratio of 8:1:1, and establish a sample data set to prepare for subsequent model training and testing. Of course, the ratio of division in this embodiment depends on the specific data set size and task requirements.

[0073] In this embodiment, if Figure 4 As shown in the figure, when blurring the image data, the bicubic interpolation method is used to resample the filtered image. The bicubic interpolation method uses 16 nearest neighbor pixels and calculates the value of the new pixel through cubic polynomial fitting to generate a smoother image and retain as many image details and textures as possible. The expression formula of the bicubic interpolation method is as follows:

[0074]

[0075] Where I(x,y) is the pixel value of the original image at point (x,y), i and j are the displacements in the horizontal and vertical directions respectively, I′(x,y) is the pixel value of the interpolated image at point (x,y), and w ij is the weight of the convolution kernel at position (i, j), and the convolution kernel w i It is expressed as the following formula:

[0076]

[0077] In this embodiment, when adding pseudo noise to image data, noise patches are extracted from the grayscale image in the uniform background intensity area, the noise patches are randomly enhanced, and added to the blurred image after mean subtraction and resizing to generate an image with pseudo noise. Finally, the pseudo noise image and the corresponding clear grayscale image are used as the image pair of the data set. Figure 5 As shown, when adding pseudo noise to image data, the following steps are specifically included:

[0078] Step S121, determining a uniform background intensity region in the image, that is, a region without a target or where the surrounding environment of the target is relatively uniform.

[0079] Step S122: extract some small areas as noise patches in the determined uniform background intensity area. These areas only contain background noise and do not contain any target information. The process of extracting noise patches can be expressed as the following formula:

[0080] P i =I′(x i :x i +w i ,y j :y j +h j )

[0081] Among them, P i represents the noise patch, I′(x,y) represents the pixel value of the resampled image at position (x,y), and I′(x i :x i +w i ,y j :y j +h j ) represents the sub-image extracted from the image, and the coordinates of its upper left corner are (x i ,y j ), the coordinates of the lower right corner are (x i +w i ,y j +h j ).

[0082] Step S123, performing random enhancement operations on the extracted noise patches, including random flipping, rotation, and brightness adjustment. These enhancement operations can simulate noise changes in real scenes and increase the robustness of the model.

[0083] Step S124, after mean subtraction and resizing, the image is added to the blurred image to generate an image with noise. First, calculate P i The average value μ i , μ i The calculation is expressed as the following formula:

[0084]

[0085] Where N is the number of pixels in the noise patch, and w i and h j are the width and height of the noise patch, respectively.

[0086] Then, the mean value is subtracted from each pixel value to obtain the mean-subtracted noise patch P i , expressed as formula:

[0087] P i ,(x,y)=P i (x,y)-μ i

[0088] Step S125, taking the noisy image and the corresponding clear grayscale image as an image pair of a data set, wherein the noisy image is taken as an input of the model and the clear image is taken as a label corresponding to the input, thereby establishing an initial data set.

[0089] In this embodiment, if Figure 6 As shown, when constructing the infrared image enhancement network, step S3 includes the following steps:

[0090] Step S31, construct a denoising network. In addition to the first two convolution layers, the denoising network consists of alternating Conv layers and ReLU layers. There are 7 Conv layers and 5 ReLU layers in the network. The Conv layer performs linear convolution on each input feature map tensor or input image patch tensor to generate an output feature map. The convolution operation in each Conv layer retains the height and width of the input feature map to the layer through the correct padding method. Finally, the residual learning technology is used to enhance the denoising performance by adding input to estimate the noise residual. The denoising network can effectively remove noise from the image, restore image details, and improve the overall visual effect of the image.

[0091] Step S32, construct a contrast enhancement network, which extracts multi-scale spatial features of infrared images by combining convolutional units, and uses target attention units composed of spatial attention mechanism and channel attention mechanism to refine these features. The network consists of 7 Conv layers, 9 ReLU layers, 1 GAP, 2 Dense layers and 2 sigmoid functions, among which Conv-8 acts on a single pixel and supplements the detailed features of the model. The next four convolutional layers (Conv-9, Conv-10, Conv-11 and Conv-12) extract coarse-grained features from different spatial scales. Conv-11 and Conv-12 use dilated convolution to expand the receptive field without increasing the computational cost. These multi-scale complementary features are then fused with the following Conv-13, and the above-mentioned Conv-9, Conv-10, Conv-11, Conv-12 and Conv-13 convolutional layers are all followed by ReLU activation functions. A target attention unit is added after the Conv-13 convolutional layer to further refine the output features from the spatial attention branch and the channel attention branch, respectively.

[0092] As a mode selector, the spatial attention branch can suppress cluttered background and irrelevant features in the spatial domain, highlight target details, and suppress oversaturation effects. The channel attention branch mainly acts as a feature selector and reweights features, emphasizing effective channel features, which is implemented by global average pooling (GAP) and then 2 fully connected layers, called Dense1 and Dense2. The final output of the target attention unit can be expressed as the following formula:

[0093] Y=M*XW

[0094] Where X is the output of the Conv-13 convolutional layer, M represents the spatial mask, W represents the channel weight, represents pixel-level multiplication, and represents the tensor product.

[0095] When training the infrared image enhancement network, in order to obtain better visual quality and higher objective evaluation indicators in the final enhanced image, a combination of L1 loss and SSIM loss function is used to train the model.

[0096] Among them, L1 loss is also called Mean Absolute Error (MAE), which is used in image enhancement to measure the absolute pixel-level difference between the enhanced image and the real image. In these tasks, L1 loss can help the model generate images that are closer to the real image in terms of pixel values, thereby promoting the preservation of details and textures. This is because L1 loss linearly penalizes the error of each pixel without imposing excessive penalties on large errors, which can avoid the outliers of certain pixels from affecting the overall loss too much. L1 loss pays more attention to the overall average absolute difference rather than the extreme differences of certain specific pixels. In tasks such as image denoising, super-resolution, and image restoration, the use of L1 loss can make the generated images more realistic in details and textures, and closer to the real image. L1 loss is expressed by the following formula:

[0097]

[0098] Where N is the total number of pixels, y i is the true value of the i-th sample, x i Represents the predicted value of the i-th sample.

[0099] SSIM is a more complex metric that takes into account the brightness, contrast, and structural information of the image. The higher the SSIM value, the more similar the enhanced image is to the target image in visual quality. The value of SSIM ranges from -1 to 1, where 1 indicates perfect similarity. SSIM is expressed as the formula:

[0100]

[0101] Among them, x and y are the enhanced image and the corresponding clear image, μ x and μ y is their average brightness, is their contrast (variance), σ xy is their covariance, c1 and c2 are small constants used to avoid the denominator being zero.

[0102] When the predicted image is closer to the label, the SSIM value is larger and closer to 1. Therefore, the SSIM loss function is finally expressed as the following formula:

[0103] L ssim =1-SSIM(x,y)

[0104] Among them, x and y represent the enhanced image and the corresponding clear image respectively. Therefore, the total loss of the model can be expressed as the following formula:

[0105] L total =αL l1 +βLssim

[0106] Among them, α and β are hyperparameters, referring to L l1 and L ssim The respective weights are positive numbers used to balance the impact of different losses. Here, the values ​​are all 0.5.

[0107] When optimizing the infrared image enhancement network, the enhanced image is compared with the target clear image, and the model results are evaluated using standard peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics to quantify the difference between the enhanced image and the target image and obtain the optimal infrared image enhancement model.

[0108] Among them, PSNR is a metric based on mean square error (MSE), which evaluates the pixel-level difference between the enhanced image and a high-quality target image. The higher the PSNR, the closer the enhanced image is to the target image, expressed as the following formula:

[0109]

[0110] Among them, MAXI is the maximum pixel value of the image, MSE is the mean square error between the original image and the processed image, and MSE is expressed as the following formula:

[0111]

[0112] Where I is the original image, is the processed image, M and N are the number of rows and columns of the image respectively.

[0113] In this embodiment, if Figure 6 As shown, when using the trained DCEN model, step S4 includes: after inputting an infrared image containing a target, converting the three channels of the infrared image into grayscale images, and inputting them into the model for enhancement. The model will output the enhanced results of the three channels of the infrared image, and then the three results are combined into one image.

[0114] In summary, the present invention can generate a large amount of image data with pseudo-noise by extracting noise patches from the grayscale image of the uniform background intensity area, and adding them to the image after random enhancement, mean subtraction and resizing. These pseudo-noise images not only enrich the diversity of the data set, but also make the data set closer to the noise conditions that may exist in the actual infrared image, thereby improving the generalization ability of the model.

[0115] Furthermore, the present invention performs further data enhancement on the initial data set, including operations such as rotation, scaling, and cropping, which can generate more training samples, help the model learn more features, and improve the robustness and accuracy of the model.

[0116] Furthermore, the present invention uses sample data sets to train and optimize the CNN network to obtain an optimal infrared image enhancement model, which can learn how to map the input infrared image to a clearer and higher-contrast output image.

[0117] Furthermore, the present invention divides the actually collected infrared image into three channels and inputs them into the infrared image enhancement model, which can make full use of the grayscale information and possible pseudo-color information of the infrared image to improve the enhancement effect of the model. The present invention combines the output results of the three channels to obtain the final infrared enhanced image, which can ensure that the enhanced image has higher clarity and contrast while retaining the original information.

[0118] Furthermore, the infrared image enhanced by this method is clearer in visual effect and has higher contrast, which is conducive to subsequent feature extraction, recognition or tracking. Since this method adds pseudo noise to the data set, the trained model can better cope with various noise interferences that may exist in infrared images in practical applications, improving the adaptability and stability of the model.

[0119] Therefore, the infrared image enhancement algorithm based on pseudo-noise and convolutional neural network in the present invention enhances the infrared image containing infrared targets, resamples the image through bicubic interpolation, adds noise patches to the resampled image, designs denoising network and contrast enhancement network, so that the algorithm can effectively remove the noise of the infrared image while enhancing the contrast of the infrared image, so that high-quality infrared images can be obtained even under complex environmental conditions.

[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. An infrared image enhancement method based on pseudo noise and convolutional neural network, characterized in that: The following steps are involved: Collect high-definition visible light image data, clean, grayscale, blur and add pseudo-noise to the collected image data to establish an initial data set; when adding pseudo-noise to the image data, extract noise patches from the grayscale image of the uniform background intensity area, then randomly enhance the noise patches, and add them to the blurred image after mean subtraction and resizing to generate image data with pseudo-noise, and use the pseudo-noise image and the corresponding clear image as the image pair of the initial data set; Perform data enhancement on the obtained initial data set and establish a sample data set including training set, validation set and test set; Construct an infrared image enhancement network based on convolutional neural network, and use sample data sets to train and optimize the network to obtain the optimal infrared image enhancement model; The actually collected infrared image is divided into three channels and input into the infrared image enhancement model for enhancement. The results of the three channels are merged to obtain the final infrared enhanced image.

2. The method according to claim 1, characterized in that: When collecting visible light image data, a camera is used to collect visible light image data containing target and environmental features, and the collected image data is cleaned to filter out images with clear targets; then the filtered images are grayed to become single-channel images; and then fuzzy processing is performed to reduce the resolution of the image; Afterwards, pseudo noise is added to the blurred image to generate a noisy image; finally, the noisy image and the corresponding image before blurring are used as image pairs of the data set to establish the initial data set.

3. The method according to claim 1, characterized in that: The data enhancement of the acquired initial data set includes: The initial data set is enhanced by flipping, scaling, and random angle rotation, and the initial data set is divided into training set, validation set, and test set in a ratio of 8:1:1 to establish a sample data set.

4. The method according to claim 1, characterized in that: When blurring the image data, the bicubic interpolation method is used to resample the filtered image. The expression formula of the bicubic interpolation method is as follows: Among them, I(x,y) is the pixel value of the original image at point (x,y), i and j are the displacements in the horizontal and vertical directions respectively, and I ′ (x, y) is the pixel value of the interpolated image at point (x, y), w ij is the weight of the convolution kernel at position (i, j), and the convolution kernel w i It is expressed as the following formula:

5. The method according to claim 1, characterized in that: When adding pseudo noise to image data, the following steps are specifically included: Determine the uniform background intensity area in the image, that is, the area where there is no target or the environment around the target is relatively uniform; In the determined uniform background intensity area, some small areas are extracted as noise patches, which only contain background noise areas and do not contain any target information; Perform random enhancement operations on the extracted noise patches, including random flipping, rotation, and brightness adjustment; After mean subtraction and resizing, it is added to the blurred image to produce an image with noise patches, i.e., a pseudo-noise image; The pseudo noise image and the corresponding clear image are used as the image pair of the data set to establish an initial data set; wherein, the pseudo noise image is used as the input of the model, and the clear grayscale image is used as the label corresponding to the input, so as to establish the initial data set.

6. The method according to claim 5, characterized in that: The process of extracting noise patches is expressed as the following formula: P i =I ′ (x i :x i +w i ,y j :y j +h j ) Among them, P i represents the noise patch, I ′ (x, y) represents the pixel value of the blurred image at position (x, y), I ′ (x i :x i +w i ,y j :y j +h j ) represents a sub-image extracted from a grayscale image, and the coordinates of its upper left corner are (x i ,y j ), the coordinates of the lower right corner are (x i +w i ,y j +h j ).

7. The method according to claim 5, characterized in that: When generating an image with a noise patch, we first calculate P i The average value μ i , μ i The calculation is expressed as the following formula: Where N is the number of pixels in the noise patch, and w i and h j are the width and height of the noise patch, respectively; Then, the mean value is subtracted from each pixel value to obtain the mean-subtracted noise patch P i , , expressed as the following formula: P i , (x,y)=P i (x,y)-μ i 。 8. The method according to claim 1, characterized in that: When constructing the infrared image enhancement network, it includes a denoising network and a contrast enhancement network. For the denoising network, in addition to the first two convolution layers, the network consists of alternating convolution (Conv) layers and rectified linear units (ReLU). There are 7 Conv layers and 5 ReLU layers in the network. The Conv layer performs linear convolution on each input feature map tensor or input image patch tensor to generate an output feature map. The convolution operation in each Conv layer retains the height and width of the input feature map to the layer through the correct padding method. Finally, the residual learning technique is used to enhance the denoising performance by adding input to estimate the noise residual. For the contrast enhancement network, the network is composed of 7 Conv layers, 9 ReLU layers, 1 GAP, 2 Dense layers and 2 sigmoid functions, among which Conv-8 acts on a single pixel and supplements the detailed features of the model. The next four convolutional layers Conv-9, Conv-10, Conv-11 and Conv-12 extract coarse-grained features from different spatial scales. Conv-11 and Conv-12 use dilated convolution to expand the receptive field without increasing the computational cost. Finally, these multi-scale complementary features are fused with the following Conv-13.

9. The method according to claim 8, characterized in that Also execute: A target attention unit is added after the Conv-13 convolutional layer to further refine the output features from the spatial attention branch and the channel attention branch respectively; Among them, the spatial attention branch acts as a pattern selector to suppress clutter background and irrelevant features in the spatial domain to highlight target details and suppress oversaturation effects; Among them, the channel attention branch is used as a feature selector and a heavy weight feature to emphasize the effective channel features. It includes a global average pool and 2 fully connected layers. The final output of the target attention unit can be expressed as the following formula: Y=M*XW Among them, X is the output of the Conv-13 convolutional layer, M represents the spatial mask, W represents the channel weight, represents the pixel-level multiplication, and represents the tensor product.

10. The method according to any one of claims 1 to 9, characterized in that: When training and optimizing the infrared image enhancement network, the enhanced image is compared with the target clear image, and the model results are evaluated using standard peak signal-to-noise ratio and structural similarity metrics to quantify the difference between the enhanced image and the target image and obtain the optimal infrared image enhancement model.

Images