Model training method and correction method for infrared image non-uniformity correction

Abstract

The present invention discloses a model training method and correction method for infrared image non-uniformity correction. The training method comprises: obtaining training data and blackbody calibration data based on the same infrared detector, and obtaining real noise data corresponding to different calibration temperatures based on the blackbody calibration data; inputting the training data into a non-uniformity correction network to perform non-uniformity correction, thereby obtaining a global noise image and a corrected output image; inputting the corrected output image into a semantic segmentation network to perform grayscale segmentation, thereby obtaining a plurality of grayscale regions; finding the calibration temperatures corresponding to the different grayscale regions based on the blackbody calibration data, and splicing the real noise data at the calibration temperatures by region to obtain a reference noise image; and calculating the loss function of the non-uniformity correction network to adjust the network parameters and obtain a trained non-uniformity correction network model. The model trained by this method can effectively remove non-uniformity noise in infrared images over a wide temperature range.

Description

Technical Field

[0001] The present invention belongs to the technical field of infrared image processing, and in particular relates to a model training method and a correction method for infrared image non-uniformity correction. Background Art

[0002] Due to process limitations, it's impossible to achieve consistent characteristics for all pixels across an infrared detector's entire pixel array. When all pixels receive infrared energy at the same illumination intensity, their output characteristics vary, manifesting as spatial noise or fixed pattern noise in the image. This often requires correction in infrared image processing. Assuming the target temperature is within a certain range and the characteristics of each pixel are linear, compensation can be applied to each pixel's output to artificially make the final output of all pixels consistent under the same input. This is called nonuniformity correction (NUC).

[0003] Currently, traditional infrared non-uniformity correction methods mainly use reference radiation source methods, such as one-point correction and two-point correction methods. Among them, the one-point correction algorithm calculates an offset from the reference image to correct the image. This algorithm is simple and easy to implement, but only considers additive noise. In practical applications, it can only achieve good correction results near the calibration point. Two-point correction assumes that the response of each detection unit in the infrared focal plane detection array is linear and different. Based on the principle that two points determine a straight line, this algorithm uses two reference images collected from black bodies at different temperatures to calculate the gain and offset of each pixel and perform linear correction on the image. Two-point correction is widely used in actual engineering. However, the response of real infrared imaging systems is not linear. Therefore, two-point correction is not suitable for infrared imaging systems with a wide operating temperature range.

[0004] Because the regression models used in traditional calibration algorithms are relatively simple, it is difficult to deeply explore the noise characteristics and cannot adapt to working conditions over a wide temperature range. To overcome this limitation, researchers have proposed many correction methods based on different working principles and scenarios based on image denoising algorithms and combined with the characteristics of non-uniform noise. These methods can be divided into scene methods and single-frame methods. These algorithms do not rely on reference radiation sources, but they rely on prior assumptions about the noise model. For example, the 1D-GF algorithm assumes that there is a linear relationship between stripe non-uniform noise and the actual image grayscale value, while the MHE algorithm assumes that adjacent columns of pixels in the image have similar histogram distributions. Considering the complex components of actual non-uniform noise, it is difficult for prior assumptions to meet the actual situation, so these algorithms often suffer from severe distortion, affecting the correction effect.

[0005] In recent years, with the continuous development of deep learning in the field of computer vision, researchers have proposed several deep learning-based methods for removing streak non-uniformity noise. Examples include SNRCNN, SNRWDNN, and ICSRN. While these methods outperform several classic streak removal methods in both quantitative and qualitative evaluations, they also have limitations. When encountering high-intensity noise, these algorithms perform poorly. Furthermore, these deep learning-based methods all use simulated noise models as training sets, which are difficult to accurately represent actual camera noise and thus cannot be applied in practice. Summary of the Invention

[0006] In order to solve the above problems existing in the prior art, the present invention provides a model training method and correction method for infrared image non-uniformity correction. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides a model training method for infrared image non-uniformity correction, comprising:

[0008] Step 1: Acquire training data and blackbody calibration data using a blackbody as a reference radiation source based on the same infrared detector, and obtain real noise data corresponding to different calibration temperatures based on the blackbody calibration data; wherein the training data includes multiple infrared noise images; and the blackbody calibration data includes multiple calibration images at different calibration temperatures;

[0009] Step 2: Input the training data into a non-uniformity correction network to perform non-uniformity correction to obtain a global noise image and a corrected output image;

[0010] Step 3: Input the corrected output image into a semantic segmentation network for grayscale segmentation to obtain a number of grayscale regions;

[0011] Step 4: Find the calibration temperature corresponding to different grayscale areas based on the blackbody calibration data, and splice the real noise data at the calibration temperature by area to obtain the reference noise;

[0012] Step 5: Calculate the loss function of the non-uniformity correction network based on the global noise image and the reference noise to adjust the network parameters and obtain a trained non-uniformity correction network model.

[0013] A second aspect of the present invention provides a method for correcting infrared image non-uniformity, comprising:

[0014] The infrared image to be corrected is input into the trained non-uniformity correction network model for correction processing to remove the non-uniform noise in the image and obtain the non-uniformity correction result;

[0015] The non-uniformity correction network is trained using the training method provided in the above embodiment.

[0016] Beneficial effects of the present invention:

[0017] 1. This paper designs a method for self-supervised training of a deep learning non-uniformity correction network using a reference radiation source. On the one hand, it uses blackbody calibration data to extract real noise as a reference, which is more in line with actual conditions. On the other hand, it introduces a semantic segmentation network for grayscale segmentation, which can better consider the noise distribution in different brightness areas in the image, and ultimately achieve effective removal of non-uniform noise across a wide temperature range. The correction network trained using this method can effectively remove non-uniform noise from real infrared detectors and is applicable to infrared imaging systems with a wide operating temperature range.

[0018] 2. The non-uniformity correction network designed by the present invention adopts a multi-level and multi-scale convolution structure, which can deeply explore the noise characteristics from different scales. Compared with the existing network, it has better feature extraction effect, thereby improving the correction results of the network.

[0019] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a model training method for infrared image non-uniformity correction provided by an embodiment of the present invention;

[0021] Figure 2 This is a flow chart of another model training method for infrared image non-uniformity correction provided by an embodiment of the present invention;

[0022] Figure 3 1 is a schematic diagram of a data processing flow of a non-uniformity correction network provided by an embodiment of the present invention;

[0023] Figure 4 2 is a network structure diagram of a multi-scale convolution module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0025] Example 1

[0026] See Figure 1-2 , Figure 1 1 is a flow chart of a model training method for infrared image non-uniformity correction provided by an embodiment of the present invention. Figure 21 is a flow chart of another model training method for infrared image non-uniformity correction provided by an embodiment of the present invention; the training method includes:

[0027] Step 1: Based on the same infrared detector, obtain training data and blackbody calibration data using a blackbody as a reference radiation source, and obtain real noise data corresponding to different calibration temperatures based on the blackbody calibration data; wherein the training data includes multiple infrared noise images; the blackbody calibration data includes multiple calibration images at different calibration temperatures.

[0028] Specifically, the infrared detector to be calibrated is used to collect multiple infrared noisy images of different scenes as training data, also known as the training set. The types of scenes can be greater than 10, and the number of collected images can be greater than 500.

[0029] At the same time, the same detector is used to collect calibration data for the blackbody, and the blackbody temperature is adjusted to the working temperature range of the detector. The blackbody calibration data of different calibration temperatures are collected at a certain temperature interval and recorded as I T1 , I T2 ,……,I Tn For example, blackbody calibration images at different calibration temperatures may be obtained at intervals of 5°C.

[0030] Furthermore, based on the calibration images collected at different calibration temperatures, the grayscale mean value corresponding to each calibration image is subtracted from the calibration image to obtain the true noise data corresponding to different calibration temperatures. The formula is as follows:

[0031] noise Ti =I Ti -mean(I Ti )i=1,2,…,n;

[0032] Among them, noise Ti Indicates the calibration temperature T i The corresponding real noise data, I Ti Indicates the calibration temperature T i The corresponding calibration image, mean(I Ti ) represents the calibration image I Ti The corresponding grayscale mean, n represents the number of calibration data.

[0033] Step 2: Input the training data into the non-uniformity correction network for non-uniformity correction to obtain the global noise image and the corrected output image.

[0034] Specifically, this embodiment designs a structure including a multi-stage multi-scale convolution module for performing multiple feature extractions at different scales on the input data to obtain rich noise characteristics.

[0035] Optionally, as an implementation method, this embodiment designs a three-level multi-scale convolution module, namely: a first multi-scale convolution module, a second multi-scale convolution module, and a third multi-scale convolution module, which can mine noise characteristics from different scale depths to improve the correction effect. Then step 2 specifically includes:

[0036] 21) Using the first multi-scale convolution module f 1_MSC Perform feature extraction on the input infrared noise image to obtain the first image feature F 01_MSC .

[0037] See Figure 3 , Figure 3 The figure is a schematic diagram of a data processing flow of a non-uniformity correction network provided by an embodiment of the present invention.

[0038] It can be understood that when using the first multi-scale convolution module f 1_MSC Before feature extraction of the input infrared noise image, it also includes:

[0039] The first convolutional layer f1 is used to extract features from the input infrared noise image I to obtain the sixth feature image F 01 , recorded as:

[0040] F 01 =f1(I);

[0041] Among them, the first convolutional layer f1 adopts a 3×3 convolution with a stride of 1.

[0042] Then, the sixth feature image F 01 Input to the first multi-scale convolution module f 1_MSC Perform feature extraction to obtain the first feature image F 01_MSC , recorded as:

[0043] F 01_MSC =f 1_MSC (F 01 ).

[0044] 22) For the first feature image F 01_MSC After downsampling, it is input to the second multi-scale convolution module f 2_MSC Extract features and obtain the second feature image F 02_MSC .

[0045] Specifically, we first pass a 2×2 convolution module f with a stride of 2. d For the first feature image F 01_MSC Perform 2 times downsampling to obtain the seventh feature image F 01_down , expressed as:

[0046] F 01_down =f d(F 01_MSC ).

[0047] Then, the first convolutional layer f2 is used to transform the seventh feature image F 01_down Perform feature extraction to obtain the eighth feature image F 02 , recorded as:

[0048] F 02 =f2(F 01_down ).

[0049] Among them, the first convolutional layer f2 is the same as the first convolutional layer f1, and both use 3×3 convolution with a stride of 1.

[0050] Finally, the eighth feature image F 02 Input to the second multi-scale convolution module f 2_MSC Perform feature extraction to obtain the second feature image F 02_MSC , recorded as:

[0051] F 02_MSC =f 2_MSC (F 02 ).

[0052] 23) For the second feature image F 02_MSC After downsampling, it is input to the third multi-scale convolution module f 3_MSC Extract features and get the third feature image F 03_MSC .

[0053] Specifically, first, a 2×2 convolution module f with a stride of 2 is still used. d For the second feature image F 02_MSC Perform 2 times downsampling to obtain the ninth feature image F 02_down , expressed as:

[0054] F 02_down =f d (F 02_MSC ).

[0055] Then, the third convolutional layer f3 is used to transform the ninth feature image F 02_down Perform feature extraction to obtain the tenth feature image F 03 , recorded as:

[0056] F 03 =f3(F 02_down ).

[0057] Among them, the third convolutional layer f3 also uses a 3×3 convolution with a stride of 1, just like the first convolutional layer f1.

[0058] Finally, the tenth feature image F 03 Input to the third multi-scale convolution module f3_MSC Perform feature extraction to obtain the third feature image F 03_MSC , recorded as:

[0059] F 03_MSC =f 3_MSC (F 03 ).

[0060] 24) For the second feature image F 02_MSC and the third feature image F 03_MSC Perform upsampling to keep the feature size consistent with the input image, and obtain the fourth feature image F 02_up and the fifth feature image F 03_up .

[0061] Specifically, using sub-pixel convolution f ps For the second feature image F 02_MSC Perform 2x upsampling to obtain the fourth feature image F 02_up ; At the same time, using sub-pixel convolution f ps2 For the third feature image F 03_MSC Perform 4 times upsampling to obtain the fifth feature image F 03_up , the formula is:

[0062]

[0063] 25) The first feature image F 01_MSC , the fourth feature image F 02_up and the fifth feature image F 03_up The channels are superimposed and a global noise image is obtained by convolution.

[0064] Specifically, the first feature image F 01_MSC , the fourth feature image F 02_up and the fifth feature image F 03_up Superimpose in the channel dimension and then pass through the fourth convolutional layer f4 to obtain the global noise image, which is recorded as:

[0065] I noise =f4(concat(F 01_MSC ,F 02_up ,F 03_up ));

[0066] Among them, I noise Represents the global noise image, and the fourth convolutional layer f4 adopts a 3×3 convolution with a stride of 1.

[0067] 26) Subtract the input infrared noise image from the global noise image to obtain the final corrected output image, which is expressed as:

[0068] Iclean =II noise ;

[0069] Among them, I clean represents the corrected output image, I represents the input infrared noise image, I noise represents the global noise image.

[0070] Furthermore, in this embodiment, the first multi-scale convolution module f 1_MSC , the second multi-scale convolution module f 2_MSC , the third multi-scale convolution module f 3_MSC Using the same network structure, all three methods include several parallel multi-scale strip convolution modules. These modules are used to extract multi-scale features from the input features, fuse the extracted features, and add them to the input features to form the output features of the multi-scale convolution module. Each strip convolution module consists of a pair of serially connected 1×n convolutions and n×1 convolutions.

[0071] Specifically, since the non-uniform noise is mainly column-wise stripe noise, this embodiment designs a strip convolution module, in which two strip convolutions can extract horizontal and vertical feature information respectively.

[0072] Optional, as an implementation, see Figure 4 , Figure 4 This is a network structure diagram of the multi-scale convolution module provided by an embodiment of the present invention. In the network structure provided in the figure, each multi-scale convolution module first undergoes a 3×3 convolution, and then passes through three parallel multi-scale strip convolution modules, namely 1×5 convolution and 5×1 convolution in series, 1×7 convolution and 7×1 convolution in series, and 1×11 convolution and 11×1 convolution in series. The results of the three parallel modules are superimposed together by channel, fused through a 1×1 convolution, and then added to the result of the initial 3×3 convolution to obtain the output features of the multi-scale convolution module.

[0073] The non-uniformity correction network designed in the present invention adopts a multi-level and multi-scale convolutional structure, which can deeply explore the noise characteristics from different scales. Compared with the existing network, it has better feature extraction effect, thereby improving the correction results of the network.

[0074] Step 3: Input the corrected output image into the semantic segmentation network for grayscale segmentation to obtain several grayscale regions.

[0075] Optionally, as an implementation, the semantic segmentation network in this embodiment adopts a Unet network structure. Furthermore, the Unet network proposed in the document "Ronneberger O, Fischer P, Brox TU-Net: Convolutional networks for biomedical image segmentation. Proceedings of the 18th International Conference on Medical Image Computing and Computer-assisted Intervention. Munich: Springer, 2015. 234–241" can be adopted.

[0076] Specifically, the corrected output image is input into the semantic segmentation network, divided into multiple regions according to the image grayscale, and the grayscale variance of each region is used as the loss function of the segmentation network. The loss function of the segmentation network is expressed as:

[0077]

[0078] Among them, Loss seg represents the loss function of the segmentation network, N represents the total number of segmented regions, S i Represents the i-th segmented area A i The number of pixels, S represents the total number of pixels in the image, std(A i ) represents area A i The standard deviation of the grayscale values.

[0079] Optionally, as an implementation method, the output type of the segmentation network can be set to 4 in this embodiment, and the input image I and the background are clean It is divided into five areas, named A1, A2, A3, A4, and A5.

[0080] Then the loss function of the segmentation network U-net is:

[0081]

[0082] Step 4: Based on the blackbody calibration data, find the calibration temperature corresponding to different grayscale areas, and splice the real noise data at the calibration temperature by area to obtain the reference noise image.

[0083] 41) Calculate the grayscale mean values of different grayscale areas respectively, and find the calibration temperature closest to each grayscale mean value in the blackbody calibration data;

[0084] 42) The real noise data corresponding to different calibration temperatures are spliced into a reference noise image according to the region, which is recorded as reference-noise.

[0085] Step 5: Calculate the loss function of the non-uniformity correction network based on the global noise image and the reference noise image to adjust the network parameters and obtain a trained non-uniformity correction network model.

[0086] Among them, the loss function of the non-uniform correction network is recorded as Loss nuc , which can be expressed as:

[0087] Loss nuc =MSE(I noise ,reference-noise).

[0088] According to the above steps, the training of the correction network and the grayscale segmentation network is completed. After the network training converges, the correction network in step 2 can work alone. Input any infrared image containing noise into the correction network, and the network output I clean As the final non-uniformity correction result.

[0089] The present invention designs a method for self-supervised training of a deep learning non-uniformity correction network using a reference radiation source. On the one hand, blackbody calibration data is used to extract real noise as a reference, which is more in line with actual conditions. On the other hand, a semantic segmentation network is introduced for grayscale segmentation, which can better take into account the noise distribution in different brightness areas in the image, and ultimately achieve effective removal of non-uniform noise in a wide temperature range. The correction network trained using this method can effectively remove non-uniform noise from real infrared detectors and is applicable to infrared imaging systems with a wide operating temperature range.

[0090] Example 2

[0091] This embodiment provides a method for correcting infrared image non-uniformity, including:

[0092] The infrared image to be corrected is input into the trained non-uniformity correction network model for correction processing to remove the non-uniform noise in the image and obtain the non-uniformity correction result;

[0093] The non-uniformity correction network is trained using the training method provided in the above embodiment 1. The processing process of the non-uniformity correction network on the input image can also be referred to the above embodiment 1.

[0094] Therefore, this method can also effectively remove the non-uniform noise of real infrared detectors and is applicable to infrared imaging systems with a wide operating temperature range.

[0095] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0097] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A model training method for infrared image non-uniformity correction, characterized in that: include: Step 1: Acquire training data and blackbody calibration data using a blackbody as a reference radiation source based on the same infrared detector, and obtain real noise data corresponding to different calibration temperatures based on the blackbody calibration data; wherein the training data includes multiple infrared noise images; and the blackbody calibration data includes multiple calibration images at different calibration temperatures; Step 2: Input the training data into a non-uniformity correction network to perform non-uniformity correction to obtain a global noise image and a corrected output image; Step 3: Input the corrected output image into a semantic segmentation network for grayscale segmentation to obtain a number of grayscale regions; Step 4: Find the calibration temperature corresponding to different grayscale areas based on the blackbody calibration data, and splice the real noise data at the calibration temperature by area to obtain a reference noise image; Step 5: Calculate the loss function of the non-uniformity correction network based on the global noise image and the reference noise image to adjust the network parameters and obtain a trained non-uniformity correction network model.

2. The model training method for infrared image non-uniformity correction according to claim 1, characterized in that: In step 1, obtaining real noise data corresponding to different calibration temperatures according to the blackbody calibration data includes: The grayscale mean value corresponding to each calibration image is subtracted from the calibration image to obtain the real noise data corresponding to different calibration temperatures.

3. The model training method for infrared image non-uniformity correction according to claim 1, characterized in that: In step 2, the non-uniformity correction network is configured to have a structure with multi-level and multi-scale convolution modules, which is used to perform feature extraction of input data at multiple scales to obtain rich noise characteristics.

4. The model training method for infrared image non-uniformity correction according to claim 3, characterized in that: Step 2 includes: 21) Using the first multi-scale convolution module f 1_MSC Extract features from the input infrared noise image and obtain the first feature image F 01_MSC ; 22) For the first feature image F 01_MSC After downsampling, it is input to the second multi-scale convolution module f 2_MSC Extract features and obtain the second feature image F 02_MSC ; 23) For the second feature image F 02_MSC After downsampling, it is input to the third multi-scale convolution module f 3_MSC Extract features and get the third feature image F 03_MSC ; 24) The second feature image F 02_MSC and the third feature image F 03_MSC Perform upsampling to keep the feature size consistent with the input image, and obtain the fourth feature image F 02_up and the fifth feature image F 03_up ; 25) The first feature image F 01_MSC , the fourth feature image F 02_up and the fifth feature image F 03_up Superposition is performed in the channel dimension, and a global noise image is obtained by convolution; 26) Subtracting the input infrared noise image from the global noise image to obtain a final corrected output image.

5. The model training method for infrared image non-uniformity correction according to claim 4, characterized in that: The first multi-scale convolution module f 1_MSC , the second multi-scale convolution module f 2_MSC , the third multi-scale convolution module f 3_MSC Each includes a plurality of parallel multi-scale strip convolution modules; the plurality of parallel multi-scale strip convolution modules are respectively used to extract multi-scale features from the input features, and fuse the extracted multiple features and add them to the input features to serve as the output features of the multi-scale convolution module; Each strip convolution module consists of a pair of serially connected 1×n convolution and n×1 convolution.

6. The model training method for infrared image non-uniformity correction according to claim 1, characterized in that: In step 3, the segmentation network adopts the Unet network structure.

7. The model training method for infrared image non-uniformity correction according to claim 1, characterized in that: Step 3 includes: The corrected output image is input into a semantic segmentation network, divided into multiple regions according to the image grayscale, and the grayscale variance of each region is used as the loss function of the segmentation network.

8. The model training method for infrared image non-uniformity correction according to claim 7, characterized in that: The loss function of the segmentation network is expressed as: Among them, Loss seg represents the loss function of the segmentation network, N represents the total number of segmented regions, S i Represents the i-th segmented area A i The number of pixels, S represents the total number of pixels in the image, std(A i ) represents area A i The standard deviation of the grayscale value.

9. The model training method for infrared image non-uniformity correction according to claim 1, characterized in that: Step 4 includes: 41) calculating the grayscale mean values of different grayscale areas respectively, and finding the calibration temperature closest to each grayscale mean value in the blackbody calibration data; 42) The real noise data corresponding to different calibration temperatures are spliced into a reference noise according to the region.

10. A method for correcting non-uniformity of infrared images, characterized in that: include: The infrared image to be corrected is input into the trained non-uniformity correction network model for correction processing to remove the non-uniform noise in the image and obtain the non-uniformity correction result; Wherein, the non-uniformity correction network model is obtained by training using the training method described in any one of claims 1-8.

Images