A method and system for denoising an infrared image based on a convolutional neural network structure

Abstract

The application discloses a method and system for removing noise from an infrared image based on a convolutional neural network structure, which adopts a convolutional neural network architecture, uses a second-order attention mechanism and a regional-level non-local module to improve image feature extraction and noise residual fitting, removes noise from an original image, and obtains a clear image; the convolutional neural network architecture in the method is divided into 15 layers, and each step is executed in each layer of the convolutional neural network architecture; the system internally operates according to each step in the method for removing noise from an infrared image by using a convolutional neural network. The application can maintain good noise removal performance under different noise environments, can retain a large amount of infrared image details, and can achieve a good noise removal effect even for infrared images mixed with real noise.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of infrared image information processing, and particularly relates to a method and system for removing noise from an infrared image based on a convolutional neural network structure. BACKGROUND

[0002] Semiconductor lasers have high photoelectric conversion efficiency, have wide wavelength coverage, long service life, can be directly modulated, have low cost and other characteristics, are widely used in optical communication systems, consumer electronics, scientific research and military, medical and cosmetic, instrument sensing and other fields, and are one of the important research directions of lasers at present. With the continuous progress of manufacturing processes, packaging technologies and material science, the research field of semiconductor lasers has correspondingly entered higher energy and higher power. However, in the process of producing laser chips, defects inevitably occur after complex process. At present, 640*512 infrared focal planes and 1060nm VCSEL lasers are mostly used as light sources to design image acquisition systems. By adjusting the horizontal and up-down positions of the light path and focal plane, the light and imaging effect are optimized. However, due to the influence of noise, non-uniformity error and other factors, the collected infrared image is blurred and unclear, and the defect position cannot be identified, which seriously affects the subsequent defect discrimination and measurement functions. Therefore, it will lead to the decline of image quality, and the infrared thermal image also has noise, and compared with visible light images, the noise in infrared images is also more serious. However, in the existing infrared image denoising technology, there is the problem of poor denoising effect

[0003] Since the convolutional neural network is widely used in image-related tasks, it has excellent performance in image classification, image semantic segmentation, image retrieval, object detection and other computer vision problems, providing a new opportunity for infrared image denoising. SUMMARY

[0004] The present application aims at the problems of poor quality and serious noise of infrared images collected by laser chips, and provides a method and system for removing noise from infrared images using a convolutional neural network to solve the problem of poor effect in the existing infrared image denoising technology.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0006] The present application provides a method for removing noise from infrared images using a convolutional neural network. The model established by the method uses a convolutional neural network system structure, improves image feature extraction and noise residual fitting using a second-order attention mechanism and a regional non-local module, removes noise from the original image, and obtains a clear image. The network structure used by the method includes the following regions:

[0007] The first region is a down-sampling module region.

[0008] The second region: the convolutional layer region;

[0009] The third region: the Residual Dense Block (RDB) region;

[0010] The fourth region: the up-sampling module region;

[0011] The fifth region: the rectified linear unit (ReLU) region.

[0012] Further, the five regions in the convolutional neural network architecture specifically include the following 15 layers:

[0013] The first layer: a down-sampling module, which is used to reduce the size of the input image and increase the receptive field;

[0014] The second layer: a noise level estimation map (M) addition layer, which is used to enhance the robustness and denoising ability of the network;

[0015] The third layer: a convolutional layer, which is established to extract basic shallow features. The shallow features are transmitted to the deeper network through a skip connection;

[0016] The fourth layer: a region-wise non-local module (RN-NL), which aims to utilize the rich structural features, similarity, and contextual information of the infrared image;

[0017] The fifth to ninth layers: Residual Dense Block modules (RDBs), which can obtain deep feature matrices when shallow feature matrices pass through the RDBs;

[0018] The tenth layer: a region-wise non-local module (RN-NL);

[0019] The eleventh layer: a convolutional layer;

[0020] The twelfth layer: an up-sampling module, which restores the feature matrix to the original image size;

[0021] The thirteenth to fourteenth layers: composed of a convolutional layer and a rectified linear unit (ReLU);

[0022] The fifteenth layer: a convolutional layer.

[0023] Further, the convolutional layer is Conv 3x3.

[0024] Further, the fifth to ninth layers of the convolutional neural network structure are designed with a second-order channel attention mechanism (SOCA) at the end of the RDB.

[0025] Further, the method for denoising infrared images using a convolutional neural network is executed in the above-mentioned 15 layers, and specifically includes the following steps:

[0026] Step 1: collect the infrared image to obtain the target image I;

[0027] Step 2: sample and divide the target image I of size n_chxh x w into a sub-image of size 4n_chxh / 2xw / 2;

[0028] Step 3: pre-process each sub-image to obtain A 0 , extract a 2x2 image block, and recombine the pixels in different channels of the output image;

[0029] Step 4: add an additional channel composed of a noise estimation image M at the input end, the resolution of the additional channel is the same as A 0 , the M controls the trade-off between noise reduction and detail preservation, and the processed image A 1 has a size of (4n_ch+1)xh / 2xw / 2;

[0030] Step 5: input the pre-processed image A 1 into the convolution layer to extract shallow features, fully utilize the rich structural features, similarity and context information of the infrared image, use the spatial correlation of the features to non-locally enhance the image block features and transmit them to the subsequent layers, and the image block enhanced by the non-local module in this step is denoted as image A R ;

[0031] Step 6: input A R into the RDB to extract deep features and obtain a deep feature map A D ;

[0032] Step 7: combine the deep feature map A D with the shallow features of the image A 1 , respectively perform F RL-NL (·) operation and convolution layer calculation, and obtain A C ;

[0033] Step 8: restore the feature matrix A C of size 4n_chxh / 2xw / 2 to size n_chxh x w, denoted as A F ;

[0034] Step 9: reconstruct the noise error of the image A F in step 8, through two convolution layers and ReLU operation, finally reconstruct the noise residual N by one convolution layer, remove the noise residual N from the target image I, and complete the denoising.

[0035] Further, the mathematical model of the pixel recombination is:

[0036] wherein c is an image channel, x is an image pixel horizontal coordinate, y is an image pixel vertical coordinate, n_ch is a channel number, h is a height, w is a width, 0<=c<=4n_ch, 0<=x<=h, 0<=w.

[0037] Further, the mathematical model in step 5 is recorded as:

[0038] A R =F RL-NL (A 1 ), wherein F RL-NL (·) is a hidden function, representing an image block enhanced in the region level by the non-local module.

[0039] Further, the mathematical model adopted for restoring A C to A F with a size of n_ch*h*w in step 8 is recorded as: A F =Up(A C ), wherein Up(·) is an up-sampling process.

[0040] The application also provides a system for removing noise from an infrared image, comprising:

[0041] An infrared lens (IL), an infrared detector (ID) and a personal computer (PC) are used to collect images by IL and ID, and the collected images are displayed on the PC, and then the noise is removed by the processor.

[0042] Further, the system uses the method for removing noise from an infrared image by using a convolutional neural network to remove noise from the infrared image, and obtains a clean infrared image.

[0043] The application has the advantages that: the convolutional neural network architecture is used, the second-order attention mechanism and the region-level non-local module are used to improve image feature extraction and noise residual fitting, and the noise of the original image is removed. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The figure is a structure diagram of the convolutional neural network of the application. DETAILED DESCRIPTION

[0045] In order to make the purposes, technical solutions and beneficial effects of the present application clearer, the present application will be further described in detail below in combination with embodiments. Obviously, the described embodiments are only part of, rather than all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present application.

[0046] Embodiment one

[0047] In combination with Figure 1 As shown in the drawings, the present application provides a method for denoising infrared images by using a convolutional neural network. The convolutional neural network structure is established to process the collected laser chip images, fit the noise residuals of the images, remove the noise of the original images, and obtain a clear image. The specific convolutional neural network structure is divided into the following regions, including:

[0048] First region: down-sampling module region;

[0049] Second region: convolutional layer region;

[0050] Third region: Resiudual Dense Block RDB region;

[0051] Fourth region: up-sampling module region;

[0052] Fifth region: rectified linear unit ReLU region.

[0053] Based on the network structure of the above five regions, the network structure specifically includes 15 layers, as shown in the drawings: Figure 1

[0054] First layer: down-sampling module, used for reducing the size of the input image and increasing the receptive field;

[0055] Second layer: noise level estimation map (M) addition layer, used for enhancing the robustness and denoising ability of the network;

[0056] Third layer: convolutional layer, which is established to extract basic shallow features. The shallow features are transmitted to the deep network through the skip connection;

[0057] Fourth layer: regional non-local module (RN-NL), which aims to utilize the rich structural features, similarity and context information of the infrared image;

[0058] Fifth layer to ninth layer: Resiudual Dense Block module (RDB), when the shallow feature matrix passes through the RDB, the deep feature matrix can be obtained. In particular, a second-order channel attention mechanism (SOCA) is designed at the end of the RDB; ​

[0059] Tenth layer: regional level non-local module (RN-NL);

[0060] Eleventh layer: convolution layer;

[0061] Twelfth layer: up-sampling module, restoring the feature matrix to the original image size;

[0062] Thirteenth layer to fourteenth layer: composed of convolution layer and rectified linear unit (ReLU);

[0063] Fifteenth layer: convolution layer.

[0064] The convolution layer in the convolutional neural network structure in the above description is Conv 3x3.

[0065] Embodiment two

[0066] A method for denoising an infrared image by using a convolutional neural network is adopted, and the specific operation steps are as follows:

[0067] Step 1: collecting the infrared image;

[0068] Step 2: sampling and dividing the target image I with a size of n_chxhwx into sub-images with a size of 4n_chxh / 2xw / 2, wherein n_ch is the number of channels, h is the height, and w is the width.

[0069] Step 3: pre-processing each sub-image to obtain A 0 , extracting a 2x2 image block, and recombining the pixels in different channels of the output image;

[0070] The mathematical model used for recombining the image pixels in this step is: Wherein, c is the image channel, x is the image pixel horizontal coordinate, and y is the image pixel vertical coordinate, wherein 0≤c≤4n_ch, 0≤x≤h, 0≤w.

[0071] Step 4: adding an additional channel composed of a noise estimation image M at the input end, the resolution of the additional channel is the same as A 0 , and the M controls the trade-off between noise reduction and detail preservation, and the processed image has a size of (4n_ch+1)xh / 2xw / 2. 1 .

[0072] Step 5: inputting the pre-processed image A 1 into the RL-NL after extracting the shallow layer features by the convolution layer, fully utilizing the rich structural features, similarity and context information of the infrared image, using the spatial correlation of the features, non-locally enhancing the image block features and transmitting them to the subsequent layers, and the image block enhanced by the non-local module in this step is denoted as image AR ;

[0073] k x k RL-NL module means that the input features are first divided into k 2 equal-sized grid blocks, each RL-NL module is processed by the subsequent layer, and in this step, the A R = F RL-NL (A 1 ) mathematical model is used for processing, where F RL-NL (·) is a hidden function, representing the enhanced image blocks in the regional level by the non-local module.

[0074] Step 6: input A R into the RDB to perform feature depth extraction, and obtain a depth feature map A D ;

[0075] Specifically, each residual block is composed of three basic blocks in series, and the proportion of each basic block is controlled by β = 0.2. SOCA is added at the end of the basic block residual network to learn the dependency between features. The second-order feature statistics are used to adaptively learn the dependency between features, and the channel features are adjusted. Given a C x H x W feature map F = [f1,..., f c ], the size of the feature map C is H x W, the feature map is reconstructed into a feature matrix X, and the sample covariance is obtained by , where s = W x H, e s and I 1 are s x s identity matrices;

[0076] The covariance is obtained by covariance normalization Σ = UΛU T , Λ is a diagonal matrix of non-decreasing eigenvalues, and is obtained by where α is a positive real number, is a channel descriptor;

[0077] When α = 1, there is no standardization; when α < 1, the eigenvalues are divergent; and when α > 1, the eigenvalues are nonlinearly contracted;

[0078] The normalized covariance matrix represents the correlation of channel mode characteristics. By merging the global covariance, the normalized covariance matrix is used as a channel descriptor, by reduction , the channel direction statistics Z ∈ R C ×1 can be obtained. Then the c-dimensional, H GCP (·) represents the global covariance aggregation function, y c is a channel descriptor, C is the total number of channel descriptor components, and Zc To count the statistics of C, the global covariance pool is used to fully utilize the feature interdependence by using aggregated information;

[0079] The gating function is w=sigmod(w U δ(w D z)),where w D , w U is the weight set of the convolution layer, and δ(·) is the function of ReLU. The channel attention mechanism is used to adaptively adjust the residual components of the RDB in the deep learning network. After the above operation, A rl-nl is obtained by RDB. D .

[0080] Step 7: The deep feature map A D is combined with the shallow features of the image A 1 to perform F RL-NL (·) operation and convolution layer calculation respectively, and A C is obtained.

[0081] Step 8: The feature matrix A C with the size of 4n_ch×h / 2×w / 2 is restored to the size of n_ch×h×w, denoted as A F ; the mathematical model used is A F =Up(A C ), where Up(·) is the up-sampling process.

[0082] Step 9: The image A F described in step 8 is reconstructed with noise error, and through the operation of two convolution layers and ReLU, finally a convolution layer is used to reconstruct the noise residual N, and the noise residual N is removed from the target image I, and the denoising is completed.

[0083] The convolution layer in the convolution neural network structure used in the above method for denoising the infrared image is Conv 3×3.

[0084] Among them, combined with the 15-layer convolution neural network structure in Embodiment One, step 1, step 2 and step 3 are performed in the first layer in the down-sampling module; step 4 is performed in the second layer noise level estimation map (M) addition layer; step 5 is performed in the third layer convolution layer and the fourth layer regional level non-local module (RN-NL); step 6 is performed in the fifth layer to the ninth layer Resiudual Dense Block module (RDB); step 7 is performed in the tenth layer regional level non-local module (RN-NL) and the eleventh layer convolution layer; step 8 is performed in the twelfth layer up-sampling module; and step 9 is performed in the thirteenth layer to the fifteenth layer.

[0085] Therefore, the whole process of the method for denoising infrared images by using the convolutional neural network is executed in the 15-layer convolutional neural network structure.

[0086] Embodiment three

[0087] The application further provides a system for denoising infrared images by using the convolutional neural network, comprising:

[0088] An infrared lens (IL) is arranged to collect images by IL;

[0089] An infrared detector (ID) is arranged to collect images by ID;

[0090] A personal computer (PC) is arranged to display the collected images.

[0091] When the system is used to denoise infrared images, the convolutional neural network structure in the embodiment one is adopted to execute the system internally, and each step in the method for denoising infrared images by using the convolutional neural network in the embodiment two is operated.

[0092] The above merely provides the preferred embodiments of the application, but the protection scope of the application is not limited to this. Any changes or replacements within the technical scope disclosed by the application can be easily conceived by those skilled in the art, and should be covered within the protection scope of the application.

Claims

1. A method for denoising an infrared image based on a convolutional neural network structure, characterized in that, The network structure used in the method comprises the following regions: The first region: a down-sampling module region; The second region: a convolutional layer region; The third region: a ResiudualDenseBlock RDB region; The fourth region: an up-sampling module region; The fifth region: a rectified linear unit ReLU region; The denoising method is specifically performed according to the following steps: Step 1: collecting an infrared image to obtain a target image I; Step 2: sampling and dividing the target image I with a size of n_ch x h x w into a sub-image with a size of 4n_ch x h / 2 x w / 2; Step 3: Preprocessing each sub-image to obtain A 0 extracting a 2x2 image block and reorganizing the pixels thereof in different channels of the output image; Step 4: Add an additional channel consisting of a noise estimation image M at the input, with the same resolution as A 0 , which controls the trade-off between noise reduction and detail preservation, resulting in a processed image of size (4n ch+1) x h / 2i x w / 2 1 ; Step 5: the pre-processed image A 1 After the shallow feature extraction by the convolutional layer, the input RL-NL fully utilizes the rich structural features, similarity and context information of the infrared image, uses the spatial correlation of the features to non-locally enhance the image block features and transmit them to the subsequent layers. The image block enhanced by the non-local module in this step is denoted as image A R ; Step 6: input A into RDB R , perform feature depth extraction to obtain a depth feature map A D ; Step 7: combine the deep feature map A D with the shallow features of image A 1 , respectively, to perform F RL-NL (·) operation and convolution layer calculation to obtain A C ; Step 8: The feature matrix A of size 4n_ch x h / 2 x w / 2 C is restored to size n_ch x h x w, denoted as A F ; Step 9: Reconstructing the noise error of the image A described in step 8 F The reconstruction noise error is performed by two convolution layers and ReLU operations, and finally a noise residual N is reconstructed by a convolution layer. The noise residual N is removed from the target image I to complete the denoising.

2. The method for removing noise from an infrared image based on a convolutional neural network structure according to claim 1, characterized in that, The convolutional neural network structure specifically comprises the following 15 layers: The first layer: a down-sampling module; The second layer: a noise level estimation map M adding layer; The third layer: a convolutional layer; The fourth layer: a non-local module RN-NL at a region level; The fifth layer to the ninth layer: a ResiudualDenseBlock RDB module; The tenth layer: a non-local module RN-NL at a region level; The eleventh layer: a convolutional layer; The twelfth layer: an up-sampling module; The thirteenth layer to the fourteenth layer: composed of a convolutional layer and a rectified linear unit ReLU; The fifteenth layer: a convolutional layer.

3. The method of claim 2, wherein, The convolutional layer is Conv3x3.

4. The method for removing noise from an infrared image based on a convolutional neural network structure according to claim 3, characterized in that, The fifth layer to the ninth layer of the convolutional neural network structure are designed with a second-order channel attention mechanism SOCA at the end of the RDB.

5. The method of claim 1, wherein, The mathematical model for recombining the pixels in step 3 is: wherein c is an image channel, x is an image pixel horizontal coordinate, y is an image pixel vertical coordinate, n_ch is a channel number, h is a height, w is a width, 0≤c≤4n_ch, 0≤x≤h, 0≤w.

6. The method for removing noise from an infrared image based on a convolutional neural network structure according to claim 1, characterized in that, The mathematical model in step 5 is: A R = F RL-NL (A 1 ); where F RL-NL (·) is the implicit function, representing the image block enhanced by the non-local module at the region level.

7. The method of claim 1, wherein, The step 8 in A C restored to size n_ch x h x w F The mathematical model used is noted: A F = Up(A C ) where Up(·) is an up-sampling process.

8. A system for denoising an infrared image based on a convolutional neural network structure, characterized in that, The system adopts the method for denoising an infrared image based on a convolutional neural network structure according to any one of claims 1-7, and is composed of an infrared lens IL, an infrared detector ID, and a personal computer PC. The image is collected by the IL and the ID, the collected image is displayed on the PC, and then the denoising processing is performed on the processor.

Images