Multi-frame image pixel-by-pixel non-uniformity correction method based on 3D residual network

Abstract

The invention discloses a multi-frame image pixel-by-pixel non-uniformity correction method based on a 3D residual network. The method comprises the following steps: training a multi-frame pixel-by-pixel non-uniformity correction network by using acquired multi-frame infrared images; the multi-frame pixel-by-pixel non-uniformity correction network comprises a pixel-by-pixel gain correction sub-network and a pixel-by-pixel offset correction sub-network; inputting a plurality of frames of infrared images obtained in real time into the trained pixel-by-pixel gain correction sub-network to obtain a pixel-by-pixel gain correction factor; correcting the multi-frame infrared image by using the pixel-by-pixel gain correction factor to obtain a gain-corrected multi-frame image; and inputting the multi-frame image subjected to gain correction into a trained pixel-by-pixel offset correction sub-network to obtain a pixel-by-pixel offset correction factor, and correcting the multi-frame image subjected to gain correction by using the pixel-by-pixel offset correction factor to obtain a multi-frame image subjected to non-uniformity correction. According to the method, the local excessive smoothness of the pixels is effectively inhibited, so that the problem that weak and small targets and texture details are possibly filtered out is avoided to a greater extent.

Description

Technical Field

[0001] The invention relates to the technical field of image processing in infrared remote sensing detection, in particular to a method for correcting pixel-by-pixel non-uniformity of multi-frame images based on a 3D residual network. Background Art

[0002] Infrared remote sensing detection technology uses infrared equipment to convert the infrared radiation energy on the surface of the object to be measured into an infrared grayscale image, and then performs target detection after preprocessing such as non-uniformity correction and image frame registration on the infrared image to obtain the target's motion trajectory. This technology is widely used in various fields such as meteorological forecasting, environmental monitoring, urban planning, and agricultural production, and can extract the target motion conditions of various fields from infrared images. However, in the actual application of infrared remote sensing detection technology, the target objects it detects often show the characteristics of weak targets. According to the definition of the International Society for Optical Engineering, targets with a local signal-to-noise ratio of less than 3.16 (5dB) and a pixel size of ≤9×9 are considered weak targets. In infrared remote sensing detection technology for weak targets, non-uniform noise is the main reason that restricts target detection performance. In the prior art, a large number of correction methods have been studied for different forms of non-uniform noise causes. In general, non-uniformity correction methods can be divided into two categories: calibration-based correction methods and scene-based correction methods.

[0003] The calibration-based correction method uses the two-point or multi-point response of each sensor detection element to the blackbody radiation source, and calculates the correction parameters through mathematical fitting. The advantage of this type of method is that the algorithm is simple, but its correction effect will be affected by changes in ambient temperature and integration time. Therefore, the calibration-based correction method needs to be performed regularly, and during calibration, the detection system needs to suspend the task to observe the blackbody radiation source used for calibration, which does not meet the requirements for fast response in the process of weak target detection.

[0004] The scene-based method uses scene information to achieve adaptive non-uniformity correction of images. It not only has the advantage of not affecting the normal operation of the infrared system, but also can avoid problems such as response drift and nonlinearity in calibration-based correction methods. Therefore, the scene-based method has become the main research direction of infrared non-uniformity correction. The main scene-based correction methods currently include Kalman filtering, time domain high-pass filtering, traditional neural network methods, image registration-based methods, and constant statistics methods. In addition, in recent years, many scholars have proposed non-uniformity correction methods based on deep learning, using deep learning methods. Deep learning has made great progress in the field of image denoising such as non-uniformity correction with its excellent non-uniformity estimation ability. However, although the existing methods have made many attempts and efforts to retain image details while denoising, they have never been able to fundamentally solve the problem that image details such as weak targets may be filtered out due to excessive image smoothing. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a multi-frame image pixel-by-pixel non-uniformity correction method based on a 3D residual network to address the problem that excessive image smoothing may filter out image details such as weak targets.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for correcting pixel-by-pixel non-uniformity of multi-frame images based on a 3D residual network, comprising the following steps:

[0007] S1. Using the acquired multiple frames of infrared images to train a multi-frame pixel-by-pixel non-uniformity correction network; the multi-frame pixel-by-pixel non-uniformity correction network includes a pixel-by-pixel gain correction subnetwork and a pixel-by-pixel bias correction subnetwork; the pixel-by-pixel gain correction subnetwork includes a plurality of cascaded first 3D residual units, and the last first 3D residual unit is connected to the first convolutional layer; the pixel-by-pixel bias correction subnetwork includes a plurality of cascaded second 3D residual units, and the last second 3D residual unit is connected to the second convolutional layer;

[0008] S2, inputting the multi-frame infrared images acquired in real time into the trained pixel-by-pixel gain correction subnetwork to obtain a pixel-by-pixel gain correction factor;

[0009] S3, correcting the multiple frames of infrared images using the pixel-by-pixel gain correction factor to obtain gain-corrected multiple frames of images;

[0010] S4. Input the gain-corrected multi-frame images into the trained pixel-by-pixel offset correction subnetwork to obtain a pixel-by-pixel offset correction factor, and use the pixel-by-pixel offset correction factor to correct the gain-corrected multi-frame images to obtain a non-uniformity corrected multi-frame image.

[0011] The present invention utilizes the powerful nonlinear estimation capability of the deep learning network to convert the estimation problem of the pixel value into the estimation problem of the detection element correction factor, and then adopts the global statistical average of the correction factor reusability to effectively suppress the local over-smoothing of the pixel value that may filter out weak targets and texture details.

[0012] Infrared targets often present the characteristics of small targets. Over-smoothing of non-uniformity correction may filter out small targets as noise points, resulting in subsequent missed detection of targets. The present invention is designed based on two sub-networks of multiple frames using 3D residual units, achieving a strong generalization ability for pixel-by-pixel non-uniformity of different intensities. The present invention effectively suppresses local over-smoothing of pixel pixels, thereby largely avoiding the problem of filtering out small targets and texture details.

[0013] In step S2 of the present invention, the first 3D residual unit and the second 3D residual unit each include a plurality of cascaded 3D convolutional layers, and each of the 3D convolutional layers is sequentially connected to a batch normalization layer and an activation function.

[0014] In the present invention, the number of the first 3D residual units and the number of the second 3D residual units are both 8.

[0015] In the present invention, the activation function is a Relu activation function.

[0016] In step S2 of the present invention, the process of obtaining the pixel-by-pixel gain correction factor includes: removing the maximum and minimum values ​​between frames of the output of one channel of the first convolutional layer, and taking the mean value between frames after removing the maximum and minimum values ​​as the pixel-by-pixel gain correction factor. The present invention uses the mean value between frames based on outlier removal as the final estimate of the correction factor for the pre-estimated tensor, which further effectively suppresses local over-smoothing of pixel pixels.

[0017] In step S2 of the present invention, the pixel-by-pixel gain correction factor G ij The calculation formula is: G ij =1 / g ij ; Among them, g ij Represents the gain of the detector in the i-th row and j-th column of the array detector.

[0018] In step S4 of the present invention, the process of obtaining the pixel-by-pixel bias correction factor includes: removing the maximum and minimum values ​​between frames of the output of one channel of the second convolutional layer, and taking the mean value between frames after removing the maximum and minimum values ​​as the pixel-by-pixel bias correction factor, which further effectively suppresses local over-smoothing of pixel pixels.

[0019] In step S4 of the present invention, the pixel-by-pixel offset correction factor O is ij The calculation formula is: ij =-o ij / g ij ; Among them, g ij and ij They represent the gain and bias of each detection element respectively, and i and j represent the i-th row and j-th column of the array detection element respectively.

[0020] In step S4 of the present invention, the element of the i-th row and j-th column of the f-th frame image of the non-uniformity correction is The expression is: G ij is the pixel-by-pixel gain correction factor, y(f) ij =g ij ·x(f) ij +o ij ,x(f) ij and y(f) ijrepresent the actual response and observed value of each pixel in the fth frame, respectively, and g ij and ij Respectively represent the gain and bias of each detector element, i and j represent the i-th row and j-th column of the array detector element, respectively. ij is the pixel-by-pixel bias correction factor.

[0021] Compared with the prior art, the present invention has the following beneficial effects: the present invention uses 3D residual units to design two sub-networks based on multiple frames, achieving a strong generalization capability for pixel-by-pixel non-uniformity of different intensities. The two sub-networks output a pre-estimated tensor of the correction factor, and then based on the pixel-by-pixel non-uniformity response model, the pre-estimated tensor uses the inter-frame mean based on outlier removal as the final estimate of the correction factor, effectively suppressing local over-smoothing of pixel pixels, thereby largely avoiding the problem of filtering out weak targets and texture details. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the pixel-by-pixel non-uniformity response of multiple frames according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of a 3D residual unit according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of a clean infrared small target sequence frame according to an embodiment of the present invention;

[0026] Figure 5(a) to Figure 5(g) The visualization results of each method for low-intensity pixel-by-pixel non-uniform infrared weak target sequence frame correction; Figure 5 (a) clean image, Figure 5 (b) non-uniform image, Figure 5 (c) THPF, Figure 5 (d) NLMF, Figure 5 (e) DnCNN, Figure 5 (f) GCDN, Figure 5 (g) method of an embodiment of the present invention;

[0027] Figure 6(a) to Figure 6(g) It is the visualization result of each method for the correction of the frame of infrared weak target sequence with pixel-by-pixel non-uniformity in intensity; Figure 6 (a) clean image, Figure 6 (b) non-uniform image, Figure 6 (c) THPF, Figure 6 (d) NLMF, Figure 6 (e) DnCNN, Figure 6 (f) GCDN, Figure 6 (g) method of the embodiment of the present invention;

[0028] Figure 7(a) to Figure 7(g)It is the visualization result of each method for high-intensity pixel-by-pixel non-uniform infrared weak target sequence frame correction; Figure 7 (a) clean image, Figure 7 (b) non-uniform image, Figure 7 (c) THPF, Figure 7 (d) NLMF, Figure 7 (e) DnCNN, Figure 7 (f) GCDN, Figure 7 (g) method of an embodiment of the present invention;

[0029] Figure 8(a) to Figure 8(f) It is the visualization result of each method for correcting the data10 sequence frame; Fig. 8(a) original image frame, Fig. 8(b) THPF, Fig. 8(c) NLMF, Fig. 8(d) DnCNN, Fig. 8(e) GCDN, Fig. 8(f) the method of the embodiment of the present invention;

[0030] Figure 9(a) to Figure 9(f) It is the visualization result of each method for correcting the data13 sequence frame; Fig. 9 (a) original image frame, Fig. 9 (b) THPF, Fig. 9 (c) NLMF, Fig. 9 (d) DnCNN, Fig. 9 (e) GCDN, Fig. 9 (f) the method of the embodiment of the present invention;

[0031] Figure 10(a) to Figure 10(f) It is the visualization result of each method for correcting the data11 sequence frame; Fig. 10(a) original image frame, Fig. 10(b) THPF, Fig. 10(c) NLMF, Fig. 10(d) DnCNN, Fig. 10(e) GCDN, Fig. 10(f) the method of the embodiment of the present invention;

[0032] Figure 11(a) to Figure 11(f) It is the visualization result of each method for correcting the data6 sequence frame; Figure 11 (a) original image frame, Figure 11 (b) THPF, Figure 11 (c) NLMF, Figure 11 (d) DnCNN, Figure 11 (e) GCDN, Figure 11 (f) the method of the embodiment of the present invention;

[0033] Figure 12(a) to Figure 12(f) It is the visualization result of each method for correcting the data19 sequence frame; Figure 12 (a) original image frame, Figure 12 (b) THPF, Figure 12 (c) NLMF, Figure 12 (d) DnCNN, Figure 12 (e) GCDN, Figure 12 (f) method of an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Example 1

[0036] Embodiment 1 of the present invention proposes a method for correcting pixel-by-pixel non-uniformity of multi-frame images based on a 3D residual network. A strong generalization capability for pixel-by-pixel non-uniformity of different intensities is achieved. The 3D residual network outputs a pre-estimated tensor of the correction factor, and then based on the pixel-by-pixel non-uniformity response model, the pre-estimated tensor uses the inter-frame mean based on outlier removal as the final estimate of the correction factor, which effectively suppresses local over-smoothing of pixel pixels, thereby largely avoiding the problem of filtering out weak targets and texture details.

[0037] like Figure 1 As shown, the specific implementation process of this embodiment includes:

[0038] Acquire multiple frames of infrared images;

[0039] Establishing a multi-frame pixel-by-pixel non-uniformity correction network, the multi-frame pixel-by-pixel non-uniformity correction network includes a pixel-by-pixel gain correction sub-network and a pixel-by-pixel bias correction sub-network;

[0040] Training a multi-frame pixel-by-pixel non-uniformity correction network;

[0041] Inputting multiple frames of infrared images into the trained pixel-by-pixel gain correction subnetwork to obtain a pixel-by-pixel gain correction factor;

[0042] Correcting multiple infrared images by using pixel-by-pixel gain correction factors to obtain gain-corrected multiple infrared images;

[0043] Input the gain-corrected multi-frame images into the trained pixel-by-pixel bias correction subnetwork to obtain the pixel-by-pixel bias correction factor;

[0044] The pixel-by-pixel offset correction factor is used to correct the gain of the multi-frame image, so as to obtain the multi-frame image with the non-uniformity correction completed.

[0045] In this embodiment, the infrared image is acquired by an infrared array camera. The infrared array camera is formed by arranging infrared light collectors to form a special two-dimensional detector. This two-dimensional detector can be divided into line scanning detection elements and area array detection elements according to the arrangement form. Multiple frames of infrared images are detected and collected by the area array detection elements. The area array detection elements do not share amplifiers, so the noise they generate is pixel-by-pixel non-uniform noise. Figure 2 shown.

[0046] The gain and offset of pixel-by-pixel non-uniform noise do not have the characteristics of being shared by columns or rows. The array detectors are multiplexed in the time domain, and since their non-uniform noise factors do not change in a short time, it can be considered that the gain and offset of pixel-by-pixel non-uniform noise have the characteristics of being shared by sequence frames.

[0047] The multi-frame pixel-by-pixel non-uniformity can be expressed as follows:

[0048] y(f) ij =g ij ·x(f) ij +o ij (1)

[0049] where x(f) ij and y(f) ij represent the actual response and observed value of each pixel in the fth frame, respectively, and g ij and ij Respectively represent the gain and bias of each detector. i and j are the i-th row and j-th column of the array detector.

[0050] Based on the response model of formula (1), the pixel-by-pixel non-uniformity correction process can be expressed as follows:

[0051]

[0052] is the pixel of the fth frame, where G ij and O ij They are the pixel-by-pixel gain correction factor and the pixel-by-pixel bias correction factor respectively:

[0053] G ij =1 / g ij (3)

[0054] O ij =-o ij / gi j (4)

[0055] Therefore, the problem of pixel-by-pixel non-uniformity correction is to estimate the pixel-by-pixel gain correction factor G ij and pixel-by-pixel bias correction factor O ij problem.

[0056] In one implementation, this embodiment estimates two deep learning sub-networks for estimating pixel-by-pixel gain correction factors and pixel-by-pixel bias correction factors, and cascades the two sub-networks for training through a multiplication structure and an addition structure. Finally, the multi-frame pixel-by-pixel non-uniformity correction network obtained by training can perform pixel-by-pixel non-uniformity correction on real infrared sequence frames (multi-frame infrared images).

[0057] The pixel gain correction subnetwork has a total of 9 layers, of which layers 1-8 are 3D residual units, and the last layer is a 3×3×3 convolution without activation function, which outputs a channel (C=1) as an estimate of the pixel-by-pixel gain correction factor. The maximum and minimum values ​​between frames of the channel are removed as outliers, and the mean value between frames after removal is used as the pixel-by-pixel gain correction factor.

[0058] The input image and output pixel-by-pixel gain correction factor of the pixel-by-pixel gain correction subnetwork are passed to a multiplication structure (corresponding to G in formula (2) ij y(f) ij ), forming a pixel-by-pixel bias correction subnetwork for gain correction of multi-frame image input.

[0059] The pixel-by-pixel bias correction subnetwork is similar to the pixel gain correction subnetwork, with a total of 9 layers, of which layers 1-8 are 3D residual units, and the last layer is a 3×3×3 convolution without an activation function, which outputs a channel (C=1) as an estimate of the pixel-by-pixel bias correction factor, and removes the maximum and minimum values ​​of the channel between frames as outliers, and uses the mean value between frames after removal as the pixel-by-pixel bias correction factor.

[0060] The input image of the pixel-by-pixel bias correction subnetwork is combined with the pixel-by-pixel deviation correction factor and passed to an additive structure to obtain multiple frames of images with completed non-uniformity correction.

[0061] In one implementation, the 3D residual unit is as follows Figure 3 As described above, it is composed of two 3D convolutional layers in cascade, each of which uses a 3×3×3 convolution kernel and outputs 64 channels. At the same time, batch normalization and ReLU activation function are used after each convolution.

[0062] In order to verify the effectiveness and superiority of the network model proposed in the above embodiment, the method of the embodiment of the present invention is compared with the classic time domain high-pass filtering method (THPF), non-local mean filtering method (NLMF) and DnCNN and GCDN based on deep network models in recent years. The following simulates pixel-by-pixel non-uniform noise of different intensities and compares the correction performance of each method. At the same time, experiments and comparisons are also carried out on real infrared non-uniform sequence frame data.

[0063] In the simulation experiment, TensorFlow software was used to build the network model, and the adaptive momentum optimizer (Adam Optimizer) was used to train the network model on a single GPU NVIDIA Tesla V100S-PCIe-32GB. The network was randomly initialized, the number of training data was 35,000, the batch size was set to 2, the initial learning rate was set to 0.001, and the learning rate decreased by 10 times every 2 rounds, for a total of 6 rounds of training. Each training data is 7 consecutive frame images, and the size of each image is 128*128.

[0064] In the experiment of infrared weak target sequence frames with different intensity pixel-by-pixel non-uniformity, the following mainly studies and compares the generalization performance of various methods for different intensity pixel-by-pixel non-uniformity, as well as the impact of these methods on weak targets in pixel-by-pixel non-uniform infrared images. In this experiment, five objective evaluation indicators are used for comparative analysis, namely root mean square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), image roughness (IR), brightness non-uniformity (Ln) and local signal-to-clutter ratio (LSCR). In the experiment of real pixel-by-pixel non-uniform infrared sequence frames, image roughness (IR), brightness non-uniformity (Ln) and local signal-to-clutter ratio (LSCR) are mainly used to compare and analyze the problems of over-smoothing and weak targets, and the structural similarity (SSIM) of the image frames before and after correction is calculated as a reference indicator.

[0065] This experiment selected 150 real clean infrared dim target images from the SIRST dataset, expanded each image into 7 frames, and added random jitter to the sequence frames, such as Figure 4 As shown in the figure, the small target is located in the middle of the marked red frame.

[0066] Different intensities of pixel-by-pixel non-uniform noise are added to the sequence frame data. Low-intensity pixel-by-pixel non-uniform multiplicative noise g ij It obeys the uniform distribution of (1-0.05,1+0.05), additive noise o ij It follows a Gaussian distribution with a mean of 0 and a standard deviation of 5. Multiplicative noise g with medium-intensity pixel-by-pixel non-uniformity ij It obeys the uniform distribution of (1-0.10,1+0.10), additive noise o ij It follows a Gaussian distribution with a mean of 0 and a standard deviation of 15. High-intensity pixel-by-pixel non-uniformity multiplicative noise g ij It obeys the uniform distribution of (1-0.15,1+0.15), additive noise o ij It follows a Gaussian distribution with a mean of 0 and a standard deviation of 25.

[0067] Figure 5(a) to Figure 5(g) It is the visualization result of each method for low-intensity pixel-by-pixel non-uniform infrared weak target sequence frame correction. Table 1 is the evaluation index results. Figure 6(a) to Figure 6(g) It is the visualization result of each method for the frame correction of infrared weak target sequence with medium intensity pixel-by-pixel non-uniformity. Table 2 is the evaluation index results. Figure 7(a) to Figure 7(g) It is the visualization result of each method for the correction of high-intensity pixel-by-pixel non-uniform infrared weak target sequence frames. Table 3 is the evaluation index results.

[0068] It can be seen from the experimental results that the correction effect of the THPF method on the pixel-by-pixel non-uniformity of different intensities is very limited, and the peak signal-to-noise ratio does not exceed 30dB. Especially when correcting the low-intensity pixel-by-pixel non-uniformity, the performance of the correction result of the THPF method in terms of mean square error and peak signal-to-noise ratio indicators has decreased. The NLMF method has a good correction effect on low-intensity pixel-by-pixel non-uniformity, but the correction effect on medium-intensity and high-intensity pixel-by-pixel non-uniformity is limited. From the perspective of peak signal-to-noise ratio and structural similarity indicators, DnCNN and GCDN have good correction effects on pixel-by-pixel non-uniformity of different intensities, but from the perspective of visualization results, these two methods have obvious image blurring for high-intensity pixel-by-pixel non-uniformity, among which DnCNN also has obvious image artifacts. In addition, from the perspective of local signal-to-noise ratio, these two methods have the phenomenon of filtering out weak targets during the correction process. Compared with the above method, the method of this embodiment has the best correction performance for pixel-by-pixel non-uniformity of different intensities. Among them, the peak signal-to-noise ratios all reached more than 40dB, and the peak signal-to-noise ratio of low-intensity pixel-by-pixel non-uniformity correction was even as high as 50.1269dB. In addition, the structural similarity also reached more than 0.98. Although the image roughness is not the lowest among all methods, it is basically equivalent to other methods. Moreover, in combination with the local signal-to-clutter ratio, other methods have over-smoothing, while the method of this embodiment has enhanced the local signal-to-clutter ratio. In general, compared with the above methods, the method of this embodiment has the best generalization performance for pixel-by-pixel non-uniformity of different intensities, and has the ability to stably avoid weak targets from being filtered out during the pixel-by-pixel non-uniformity correction process.

[0069] Table 1 Correction evaluation results of various methods for low-intensity pixel-by-pixel non-uniform infrared sequence frames

[0070]

[0071] Table 2 Correction evaluation results of various methods for infrared sequence frames with medium intensity pixel-by-pixel non-uniformity

[0072]

[0073] Table 3 Correction evaluation results of various methods for high-intensity pixel-by-pixel non-uniform infrared sequence frames

[0074]

[0075] The infrared image weak aircraft target detection and tracking data set under ground / air background has pixel-by-pixel non-uniform noise. This experiment selects data10, data13, data11, data6, and data19 as real sequence frame test data. In each selected data segment, 1 frame is taken every 10 frames, and 7 frames are used as a test data. Among them, each data segment uses 70 frames, and the 5 data segments have a total of 350 frames.

[0076] Since there is no clean reference image for real non-uniform infrared images, this experiment cannot evaluate the reference index. In this experiment, image roughness (IR), brightness non-uniformity (Ln), structural similarity index (SSIM) and local signal-to-noise ratio (SSIM) are used to comprehensively analyze the non-uniformity correction and its over-smoothing problem. Figure 8(a) to Figure 8(f) to Figure 12(a) to Figure 12(f) is the visualization result of each method correcting the 5 sequence frame data segments, where the red box is the target point and the green circle is the blind pixel. Tables 4 to 8 are the evaluation index results respectively. Table 9 is the average evaluation result of each method on the 5 data segments.

[0077] Generally speaking, the smaller the image roughness (IR) and brightness non-uniformity (Ln), the stronger the ability of each method to remove noise. However, too small image roughness and brightness non-uniformity may also mean that the image is over-smoothed during the denoising process. On the other hand, since there is no clean reference image for the real non-uniform image, the structural similarity index (SSIM) of this embodiment is compared and calculated with the original noise image. The higher the structural similarity index, the smaller the image blur, artifacts and over-smoothing problems caused by non-uniformity. However, too high a structural similarity index may also mean that the correction method fails to effectively remove the noise in the image. In comparison, in the quantitative evaluation of non-uniformity correction of real infrared weak target images, the local signal-to-clutter ratio (LSCR) has a stronger reference. The lower the local signal-to-clutter ratio of the weak target, the more serious the problem of over-smoothing. Therefore, this experiment is based on the local signal-to-clutter ratio, combined with image roughness and structural similarity indicators to analyze the non-uniformity correction and over-smoothing problems of each method.

[0078] From the quantitative indicators of the above experimental results, it can be seen that the structural similarity index of the method of this embodiment is the highest on all data segments, and all reach above 0.95, which proves the ability of the method of this embodiment to avoid excessive smoothing of the image and retain image details. Although the image roughness and brightness non-uniformity of the method of this embodiment are not the lowest compared with other methods, a comprehensive analysis shows that the method of this embodiment achieves effective non-uniformity correction. First, it can be seen that the non-uniform noise contained in the visualized original image itself is relatively weak, so the decrease in the image roughness and brightness non-uniformity index after effective correction will not be very large. Secondly, from the visualization results of the corrected image, other methods all cause excessive smoothing of the image and blurring of the image details, while the method of this embodiment not only retains rich image details and weak targets, but also effectively corrects the blind pixels. In the visualization results, the corrected image without a red frame to mark the weak target means that its weak target has been completely filtered out, and this visualization result is also reflected in the quantitative indicator of the local signal-to-noise ratio. From the quantitative evaluation index of local signal-to-clutter ratio, other methods have weakened the signal strength of small targets to varying degrees, especially data10 with the lowest local signal-to-clutter ratio. Other methods have filtered out small targets, and only the method of this embodiment can still avoid small targets from being filtered out. In general, compared with other methods, the method of this embodiment has greatly improved the over-smoothing problem that is common in denoising methods, and can effectively filter out non-uniform noise while retaining image detail information and small targets, providing reliable performance guarantee and improvement for subsequent image registration, small target detection, etc.

[0079] Table 4 Evaluation index results of each method for the correction of the real sequence frame data10

[0080]

[0081] Table 5 Evaluation index results of each method for the correction of real sequence frame data13

[0082]

[0083] Table 6 Evaluation index results of each method for the correction of the real sequence frame data11

[0084]

[0085] Table 7 Evaluation index results of each method for the correction of the real sequence frame data6

[0086]

[0087] Table 8 Evaluation index results of each method for the correction of real sequence frame data19

[0088]

[0089] Table 9 Average results of evaluation indicators of each method for 5 real sequence frame corrections

[0090]

[0091] Example 2

[0092] Embodiment 2 of the present invention provides a terminal device corresponding to the above-mentioned embodiment 1. The terminal device may be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.

[0093] The terminal device of this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in the above-mentioned embodiment 1.

[0094] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0095] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0096] Example 3

[0097] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to the above embodiment 1, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the above embodiment 1 are implemented.

[0098] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0099] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0100] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0103] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for pixel-by-pixel non-uniformity correction of multi-frame images based on a 3D residual network, characterized in that: The following steps are involved: S1. Using the acquired multiple frames of infrared images to train a multi-frame pixel-by-pixel non-uniformity correction network; the multi-frame pixel-by-pixel non-uniformity correction network includes a pixel-by-pixel gain correction sub-network and a pixel-by-pixel bias correction sub-network; the pixel-by-pixel gain correction sub-network includes a plurality of cascaded first 3D residual units, The last first 3D residual unit is connected to the first convolutional layer; the pixel-by-pixel offset correction subnetwork includes a plurality of cascaded second 3D residual units, and the last second 3D residual unit is connected to the second convolutional layer; S2, inputting the multi-frame infrared images acquired in real time into the trained pixel-by-pixel gain correction subnetwork to obtain a pixel-by-pixel gain correction factor; S3, correcting the multiple frames of infrared images using the pixel-by-pixel gain correction factor to obtain gain-corrected multiple frames of images; S4. Input the gain-corrected multi-frame images into the trained pixel-by-pixel offset correction subnetwork to obtain a pixel-by-pixel offset correction factor, and use the pixel-by-pixel offset correction factor to correct the gain-corrected multi-frame images to obtain a non-uniformity corrected multi-frame image.

2. The method for correcting pixel-by-pixel non-uniformity of multi-frame images based on a 3D residual network according to claim 1, characterized in that: In step S2, the first 3D residual unit and the second 3D residual unit each include a plurality of cascaded 3D convolutional layers, and each of the 3D convolutional layers is sequentially connected to a batch normalization layer and an activation function.

3. The method for correcting pixel-by-pixel non-uniformity of multi-frame images based on a 3D residual network according to claim 1, characterized in that: The number of the first 3D residual units and the number of the second 3D residual units are both 8.

4. The method for correcting pixel-by-pixel non-uniformity of multi-frame images based on a 3D residual network according to claim 2, characterized in that: The activation function is the Relu activation function.

5. The method for correcting pixel-by-pixel non-uniformity of multi-frame images based on 3D residual network according to claim 1, characterized in that: In step S2, the process of obtaining the pixel-by-pixel gain correction factor includes: The maximum and minimum values ​​between frames of the output of one channel of the first convolutional layer are removed, and the inter-frame mean after removing the maximum and minimum values ​​is used as the pixel-by-pixel gain correction factor.

6. The method for correcting pixel-by-pixel non-uniformity of multi-frame images based on a 3D residual network according to claim 1, characterized in that: In step S2, the pixel-by-pixel gain correction factor G ij The calculation formula is: ij =1 / g ij ; Among them, g ij Represents the gain of the detector in the i-th row and j-th column of the array detector.

7. The method for correcting pixel-by-pixel non-uniformity of multi-frame images based on a 3D residual network according to claim 1, characterized in that: In step S4, the process of obtaining the pixel-by-pixel bias correction factor includes: removing the maximum and minimum values ​​between frames of the output of one channel of the second convolutional layer, and taking the inter-frame mean after removing the maximum and minimum values ​​as the pixel-by-pixel bias correction factor.

8. The method for correcting pixel-by-pixel non-uniformity of multi-frame images based on a 3D residual network according to claim 1, characterized in that: In step S4, the pixel-by-pixel offset correction factor O is calculated. ij The calculation formula is: ij =-o ij / g ij ; Among them, g ij and ij They represent the gain and bias of each detection element respectively, and i and j represent the i-th row and j-th column of the array detection element respectively.

9. The method for correcting pixel-by-pixel non-uniformity of multiple frames of images based on a 3D residual network according to any one of claims 1 to 8, characterized in that: In step S4, the element of the i-th row and j-th column of the f-th frame image of the non-uniformity correction is The expression is: G ij is the pixel-by-pixel gain correction factor, y(f) ij =g ij ·x(f) ij +o ij ,x(f) ij and y(f) ij represent the actual response and observed value of each pixel in the fth frame, respectively, and g ij and ij Respectively represent the gain and bias of each detector element, i and j represent the i-th row and j-th column of the array detector element, respectively. ij is the pixel-by-pixel bias correction factor.

Images