A Pixel-by-Pixel Non-Uniformity Correction Method for Multi-Frame Images Based on 3D Residual Networks

Abstract

This invention discloses a method for pixel-by-pixel non-uniformity correction of multi-frame images based on a 3D residual network. The method trains a multi-frame pixel-by-pixel non-uniformity correction network using acquired multi-frame infrared images. The multi-frame pixel-by-pixel non-uniformity correction network includes a pixel-by-pixel gain correction sub-network and a pixel-by-pixel bias correction sub-network. The real-time acquired multi-frame infrared images are input into the trained pixel-by-pixel gain correction sub-network to obtain a pixel-by-pixel gain correction factor. The pixel-by-pixel gain correction factor is used to correct the multi-frame infrared images, resulting in a gain-corrected multi-frame image. The gain-corrected multi-frame image is then input into the trained pixel-by-pixel bias correction sub-network to obtain a pixel-by-pixel bias correction factor. This factor is used to correct the gain-corrected multi-frame image, resulting in a non-uniformity-corrected multi-frame image. This invention effectively suppresses local over-smoothing of pixels, thus largely avoiding the problem of potentially filtering out weak targets and texture details.

Description

Technical Field

[0001] This invention relates to the field of image processing technology in infrared remote sensing, and in particular to a method for pixel-by-pixel non-uniformity correction of multi-frame images based on 3D residual networks. Background Technology

[0002] Infrared remote sensing technology converts the infrared radiation energy of the surface of the object being measured into an infrared grayscale image using infrared equipment. After preprocessing the infrared image, such as non-uniformity correction and inter-frame registration, target detection is performed to obtain the target's trajectory. This technology has wide applications in various fields such as weather forecasting, environmental monitoring, urban planning, and agricultural production, enabling the extraction of the motion of targets of interest in various fields from infrared images. However, in practical applications, the targets detected by infrared remote sensing technology often exhibit the characteristics of small targets. According to the definition of the International Society for Optics and Photonics (ISO), targets with a local signal-to-noise ratio <3.16 (5dB) and a pixel size ≤9×9 are considered small targets. In infrared remote sensing technology targeting small targets, non-uniform noise is the main factor restricting target detection performance. Existing technologies have conducted extensive research on correction methods for different forms of non-uniform noise. Generally speaking, non-uniformity correction methods can be mainly divided into two categories: calibration-based correction methods and scene-based correction methods.

[0003] Calibration-based correction methods utilize the two- or multi-point responses of each sensor element to a blackbody radiation source, calculating correction parameters through mathematical fitting. The advantage of this method is its simple algorithm, but its correction effectiveness is affected by variations in ambient temperature and integration time. Therefore, calibration-based correction methods require periodic calibration, and during calibration, the detection system needs to pause its operation to observe the blackbody radiation source used for calibration. This does not meet the requirement for rapid response in the detection of weak targets.

[0004] Scene-based methods utilize scene information to achieve adaptive non-uniformity correction of images. They not only have the advantage of not affecting the normal operation of the infrared system, but also avoid problems such as response drift and nonlinearity in calibration-based correction methods. Therefore, scene-based methods have become a major research direction in infrared non-uniformity correction. Current main scene-based correction methods include Kalman filtering, temporal high-pass filtering, traditional neural networks, image registration-based methods, and constant statistics methods. In addition, in recent years, many scholars have proposed non-uniformity correction methods based on deep learning. Deep learning, with its excellent non-uniformity estimation capabilities, has made great strides in image denoising, including non-uniformity correction. However, despite numerous attempts and efforts by existing methods to preserve image details while denoising, they have not fundamentally solved the problem that excessive image smoothing may filter out image details such as weak targets. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a multi-frame image pixel-by-pixel non-uniformity correction method based on 3D residual network to address the shortcomings of existing technology, thereby solving the problem that excessive smoothing of images may filter out image details such as weak targets.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for pixel-by-pixel non-uniformity correction of multi-frame images based on 3D residual networks, comprising the following steps:

[0007] S1. Train a multi-frame pixel-by-pixel non-uniformity correction network using the acquired multi-frame infrared images; the multi-frame pixel-by-pixel non-uniformity correction network includes a pixel-by-pixel gain correction sub-network and a pixel-by-pixel bias correction sub-network; the pixel-by-pixel gain correction sub-network includes multiple cascaded first 3D residual units, the last first 3D residual unit being connected to a first convolutional layer; the pixel-by-pixel bias correction sub-network includes multiple cascaded second 3D residual units, the last second 3D residual unit being connected to a second convolutional layer;

[0008] S2. Input the multi-frame infrared images acquired in real time into the trained pixel-by-pixel gain correction sub-network to obtain the pixel-by-pixel gain correction factor.

[0009] S3. Correct the multi-frame infrared images using the pixel-by-pixel gain correction factor to obtain gain-corrected multi-frame images;

[0010] S4. Input the gain-corrected multi-frame images into the trained pixel-by-pixel bias correction sub-network to obtain the pixel-by-pixel bias correction factor. Use the pixel-by-pixel bias correction factor to correct the gain-corrected multi-frame images to obtain non-uniformity-corrected multi-frame images.

[0011] This invention utilizes the powerful nonlinear estimation capabilities of deep learning networks to transform the problem of estimating pixel values ​​into the problem of estimating probe correction factors. Furthermore, it employs global statistical averaging with reusable correction factors to effectively suppress local oversmoothing of pixel values ​​that may filter out weak targets and texture details.

[0012] Infrared targets often exhibit characteristics of being small and weak. Over-smoothing during non-uniformity correction may filter out these small targets as noise points, leading to missed detections in subsequent tests. This invention employs 3D residual units in a multi-frame sub-network design, achieving strong generalization capability for pixel-by-pixel non-uniformity of varying intensities. This invention effectively suppresses local over-smoothing of pixels, thereby largely avoiding the problem of filtering out small targets and texture details.

[0013] In step S2 of the present invention, the first 3D residual unit and the second 3D residual unit each include multiple cascaded 3D convolutional layers, and each 3D convolutional layer is followed by a batch normalization layer and an activation function in sequence.

[0014] In this invention, there are 8 first 3D residual units and 8 second 3D residual units.

[0015] In this invention, the activation function is the ReLU activation function.

[0016] In step S2 of this invention, the process of obtaining the pixel-by-pixel gain correction factor includes: removing the maximum and minimum values ​​of the output of one channel of the first convolutional layer between frames, and using the mean value of the frames after removing the maximum and minimum values ​​as the pixel-by-pixel gain correction factor. This invention uses the mean value of the frames based on outlier removal as the final estimate of the correction factor for the pre-estimated tensor, which further effectively suppresses local over-smoothing of pixel pixels.

[0017] In step S2 of this invention, the pixel-by-pixel gain correction factor G ij The calculation formula is: G ij =1 / g ij Among them, g ij This represents the gain of the detector element in the i-th row and j-th column of the area array detector element.

[0018] In step S4 of this invention, the process of obtaining the pixel-by-pixel bias correction factor includes: removing the maximum and minimum values ​​of the output of one channel of the second convolutional layer between frames, and using the mean value of the frames after removing the maximum and minimum values ​​as the pixel-by-pixel bias correction factor. This further effectively suppresses local over-smoothing of pixel pixels.

[0019] In step S4 of this invention, the pixel-by-pixel offset correction factor O ij The calculation formula is: O ij =-o ij / g ij Among them, g ij and o ij Let i and j represent the gain and bias of each detector element, respectively, and let i and j represent the i-th row and j-th column of the area array detector element, respectively.

[0020] In step S4 of this invention, the element in the i-th row and j-th column of the f-th frame image after non-uniformity correction... The expression is: G ij y(f) is the pixel-by-pixel gain correction factor. ij =g ij ·x(f) ij +o ij , x(f) ij and y(f) ijLet g represent the actual response and observed value of each pixel in the f-th frame, respectively. ij and o ij Let i and j represent the gain and bias of each detector element, respectively, and let i and j represent the i-th row and j-th column of the area array detector element, respectively. ij This is the pixel-by-pixel offset correction factor.

[0021] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention employs 3D residual units in the design of two sub-networks across multiple frames, achieving strong generalization ability for pixel-by-pixel non-uniformity of varying intensities. The two sub-networks output a pre-estimated tensor of the correction factor. Then, based on the pixel-by-pixel non-uniformity response model, the inter-frame mean based on outlier removal is used as the final estimate of the correction factor, effectively suppressing local over-smoothing of pixel areas and thus largely avoiding the potential filtering out of weak targets and texture details. Attached Figure Description

[0022] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of multi-frame pixel-by-pixel non-uniformity response in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the 3D residual unit structure according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of a clean infrared weak target sequence frame according to an embodiment of the present invention;

[0026] Figures 5(a) to 5(g) The visualization results of each method for correcting the sequence of weak infrared targets with low intensity pixel-by-pixel non-uniformity are shown in Figure 5(a), clean image, Figure 5(b), non-uniform image, Figure 5(c), THPF, Figure 5(d), NLMF, Figure 5(e), DnCNN, Figure 5(f), GCDN, and Figure 5(g), the method of the present invention.

[0027] Figures 6(a) to 6(g) The images are visualizations of the frame correction results for the medium-intensity pixel-by-pixel non-uniform infrared weak target sequence by various methods; Figure 6(a) clean image, Figure 6(b) non-uniform image, Figure 6(c) THPF, Figure 6(d) NLMF, Figure 6(e) DnCNN, Figure 6(f) GCDN, Figure 6(g) method of the present invention.

[0028] Figures 7(a) to 7(g)The results are visualizations of the frame correction of high-intensity pixel-by-pixel non-uniform infrared weak target sequences by various methods; Figure 7(a) clean image, Figure 7(b) non-uniform image, Figure 7(c) THPF, Figure 7(d) NLMF, Figure 7(e) DnCNN, Figure 7(f) GCDN, Figure 7(g) method of the present invention.

[0029] Figures 8(a) to 8(f) The following are visualization results of the correction of data10 sequence frames by various methods; Figure 8(a) Original image frame, Figure 8(b) THPF, Figure 8(c) NLMF, Figure 8(d) DnCNN, Figure 8(e) GCDN, Figure 8(f) Method of the embodiment of the present invention;

[0030] Figures 9(a) to 9(f) The following are visualization results of the correction of the data13 sequence frames by various methods; Figure 9(a) Original image frame, Figure 9(b) THPF, Figure 9(c) NLMF, Figure 9(d) DnCNN, Figure 9(e) GCDN, Figure 9(f) Method of the embodiment of the present invention;

[0031] Figures 10(a) to 10(f) The following are visualization results of the correction of the data11 sequence frames by various methods; Figure 10(a) Original image frame, Figure 10(b) THPF, Figure 10(c) NLMF, Figure 10(d) DnCNN, Figure 10(e) GCDN, Figure 10(f) Method of the embodiment of the present invention;

[0032] Figures 11(a) to 11(f) The following are visualization results of the correction of the data6 sequence frames by various methods; Figure 11(a) Original image frame, Figure 11(b) THPF, Figure 11(c) NLMF, Figure 11(d) DnCNN, Figure 11(e) GCDN, Figure 11(f) Method of the embodiment of the present invention;

[0033] Figures 12(a) to 12(f) The following are visualization results of the correction of the data19 sequence frames by various methods; Figure 12(a) Original image frame, Figure 12(b) THPF, Figure 12(c) NLMF, Figure 12(d) DnCNN, Figure 12(e) GCDN, Figure 12(f) Method of the embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example 1

[0036] Embodiment 1 of this invention proposes a multi-frame image per-pixel non-uniformity correction method based on a 3D residual network. It achieves strong generalization ability for per-pixel non-uniformity of varying intensities. The 3D residual network outputs a pre-estimated tensor of the correction factor. Then, based on the per-pixel non-uniformity response model, the inter-frame mean based on outlier removal is used as the final estimate of the correction factor, effectively suppressing local over-smoothing of pixels and thus largely avoiding the potential filtering out of weak targets and texture details.

[0037] like Figure 1 As shown, the specific implementation process of this embodiment includes:

[0038] Acquire multiple frames of infrared images;

[0039] A multi-frame pixel-by-pixel non-uniformity correction network is established, which includes a pixel-by-pixel gain correction sub-network and a pixel-by-pixel bias correction sub-network.

[0040] Training a multi-frame pixel-by-pixel non-uniformity correction network;

[0041] Input multiple frames of infrared images into the trained pixel-by-pixel gain correction subnetwork to obtain the pixel-by-pixel gain correction factor.

[0042] Gain-corrected multi-frame infrared images are obtained by correcting multi-frame infrared images using a pixel-by-pixel gain correction factor.

[0043] Input the gain-corrected multi-frame images into the trained pixel-by-pixel bias correction sub-network to obtain the pixel-by-pixel bias correction factor.

[0044] The gain of multiple frames is corrected by the pixel-by-pixel offset correction factor to obtain multiple frames of images with non-uniformity correction completed.

[0045] In this embodiment, infrared images are acquired using an infrared array camera. An infrared array camera is a dedicated two-dimensional detector formed by arranging infrared light collectors. This type of detector can be categorized into line-scan detector elements and area array detector elements based on their arrangement. Multiple frames of infrared images are acquired by the area array detector elements. Since the area array detector elements do not share amplifiers, the noise they generate is pixel-by-pixel non-uniform noise. Figure 2 As shown.

[0046] The gain and bias of pixel-by-pixel non-uniform noise do not exhibit column-by-column or row-by-row sharing characteristics. Area array detectors are multiplexed in the time domain, and since their non-uniform noise factor does not change over a short period, the gain and bias of pixel-by-pixel non-uniform noise can be considered to have frame-by-frame sharing characteristics.

[0047] Multi-frame pixel-wise non-uniformity can be represented as follows:

[0048] y(f) ij =g ij ·x(f) ij +o ij (1)

[0049] Where x(f) ij and y(f) ij Let g represent the actual response and observed value of each pixel in the f-th frame, respectively. ij and o ij These represent the gain and bias of each detector element, respectively. i and j are the i-th row and j-th column of the area array detector element, respectively.

[0050] Based on the response model of equation (1), the pixel-by-pixel non-uniformity correction process can be expressed as follows:

[0051]

[0052] These are the pixels of the f-th frame, where G ij and O ij These are the pixel-by-pixel gain correction factor and the pixel-by-pixel bias correction factor:

[0053] G ij =1 / g ij (3)

[0054] O ij =-o ij / gi j (4)

[0055] Therefore, the problem of per-pixel non-uniformity correction is to estimate the per-pixel gain correction factor G. ij and per-pixel offset correction factor O ij The problem.

[0056] In one implementation, this embodiment uses two deep learning sub-networks for estimating pixel-by-pixel gain correction factor and pixel-by-pixel bias correction factor, and trains the two sub-networks by cascading them together through a multiplication structure and an addition structure. The resulting multi-frame pixel-by-pixel non-uniformity correction network can perform pixel-by-pixel non-uniformity correction on real infrared sequence frames (multi-frame infrared images).

[0057] The pixel gain correction subnetwork has a total of 9 layers, of which layers 1-8 are 3D residual units, and the last layer is a 3×3×3 convolution without activation function. Its output channel (C=1) is used as a pre-estimate of the pixel gain correction factor. The maximum and minimum values ​​of this channel between frames are removed as outliers, and the mean value between frames after removal is used as the pixel gain correction factor.

[0058] The input image and the output pixel-by-pixel gain correction factor of the pixel-by-pixel gain correction subnetwork are passed to a multiplication structure (corresponding to G in formula (2)). ij ·y(f) ij This forms a gain-corrected multi-frame image input pixel-by-pixel bias correction subnetwork.

[0059] The per-pixel bias correction sub-network is similar to the pixel gain correction sub-network, with a total of 9 layers. Layers 1-8 are 3D residual units, and the last layer is a 3×3×3 convolution without activation function. It outputs one channel (C=1) as a pre-estimate of the per-pixel bias correction factor. The maximum and minimum values ​​of this channel between frames are removed as outliers, and the mean value between frames after removal is used as the per-pixel bias correction factor.

[0060] The input image of the pixel-by-pixel bias correction subnetwork is combined with the pixel-by-pixel bias correction factor and passed to an additive structure to obtain a multi-frame image that has completed non-uniformity correction.

[0061] In one implementation, the 3D residual unit is as follows: Figure 3 The method consists of two cascaded 3D convolutional layers, each using a 3×3×3 convolutional kernel and outputting 64 channels. Batch normalization and ReLU activation functions are applied after each convolution.

[0062] To verify the effectiveness and superiority of the network model proposed in the above embodiments, comparative experiments were conducted with the classical temporal high-pass filtering method (THPF), non-local mean filtering method (NLMF), and recent deep network model-based methods such as DnCNN and GCDN. The following simulations of pixel-by-pixel non-uniform noise of different intensities were performed, and the correction performance of each method was compared. Simultaneously, experiments and comparisons were also conducted on real infrared non-uniform sequence frame data.

[0063] In the simulation experiment, the network model was built using TensorFlow software and trained on a single GPU, NVIDIA Tesla V100S-PCIe-32GB, employing the Adam Optimizer. The network was randomly initialized with 35,000 training data points, a batch size of 2, and an initial learning rate of 0.001. The learning rate decreased by a factor of 10 every two epochs, for a total of six epochs. Each training dataset consisted of seven consecutive frames, each image being 128*128 pixels in size.

[0064] In experiments on infrared image sequences with varying intensities of pixel-by-pixel non-uniformity and weak targets, the following analysis primarily compares the generalization performance of different methods for pixel-by-pixel non-uniformity and their impact on weak targets in pixel-by-pixel non-uniform infrared images. Five objective evaluation metrics were used for comparison and analysis: root mean square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), image roughness (IR), brightness non-uniformity (Ln), and local signal-to-noise ratio (LSCR). In experiments on real pixel-by-pixel non-uniform infrared image sequences, image roughness (IR), brightness non-uniformity (Ln), and local signal-to-noise ratio (LSCR) were primarily used to compare and analyze the issues of over-smoothing and weak targets. Simultaneously, the structural similarity (SSIM) of the image frames before and after correction was calculated as a reference metric.

[0065] This experiment selected 150 real, clean infrared images of weak targets from the SIRST dataset, expanded each image into 7 frames, and added random jitter to the frame sequence, such as... Figure 4 As shown. The weak target is located in the exact center of the marked red box.

[0066] Different levels of pixel-by-pixel non-uniformity noise were added to the sequence frame data. Low-intensity pixel-by-pixel non-uniformity multiplicative noise g ij It follows a uniform distribution in (1-0.05, 1+0.05), with additive noise o. ij It follows a Gaussian distribution with a mean of 0 and a standard deviation of 5. The multiplicative noise g exhibits moderate-intensity pixel-by-pixel nonuniformity. ij It follows a uniform distribution in (1-0.10, 1+0.10), with additive noise o. ij It follows a Gaussian distribution with a mean of 0 and a standard deviation of 15. The multiplicative noise g, representing high-intensity pixel-by-pixel nonuniformity, is... ij It follows a uniform distribution in (1-0.15, 1+0.15), with additive noise o. ij It follows a Gaussian distribution with a mean of 0 and a standard deviation of 25.

[0067] Figures 5(a) to 5(g) The results are visualizations of the frame correction of low-intensity pixel-by-pixel non-uniform infrared weak target sequences by various methods. Table 1 shows the evaluation index results. Figures 6(a) to 6(g) Table 2 shows the visualization results of frame correction for medium-intensity pixel-by-pixel non-uniform infrared weak target sequences by various methods, and the evaluation index results are shown in Table 2. Figures 7(a) to 7(g) Table 3 shows the visualization results of the various methods for correcting high-intensity pixel-by-pixel non-uniform infrared weak target sequence frames, and the evaluation index results are shown in Table 3.

[0068] The experimental results show that the THPF method has very limited effectiveness in correcting pixel-by-pixel non-uniformity of varying intensities, with peak signal-to-noise ratios (PSNRs) not exceeding 30 dB. Particularly when correcting low-intensity pixel-by-pixel non-uniformity, the THPF method's performance in terms of mean squared error (MSE) and PNR actually decreases. The NLMF method shows good correction performance for low-intensity pixel-by-pixel non-uniformity, but its effectiveness is limited for medium and high-intensity non-uniformity. In terms of PNR and structural similarity, DnCNN and GCDN both show good correction performance for pixel-by-pixel non-uniformity of varying intensities. However, visualization results show significant image blurring for high-intensity pixel-by-pixel non-uniformity, with DnCNN exhibiting noticeable image artifacts. Furthermore, in terms of local signal-to-noise ratio (SNR), both methods filter out weak targets during the correction process. Compared to the methods mentioned above, the method in this embodiment exhibits the best correction performance for pixel-by-pixel non-uniformity of varying intensities. The peak signal-to-noise ratio (PSNR) of all methods reached over 40 dB, with the PSNR for low-intensity pixel-by-pixel non-uniformity correction even reaching 50.1269 dB. Furthermore, the structural similarity also exceeded 0.98. While the image roughness was not the lowest among all methods, it was comparable to other methods. Moreover, considering the local signal-to-noise ratio, other methods exhibited over-smoothing, while the method in this embodiment enhanced the local signal-to-noise ratio. In summary, compared to the methods mentioned above, the method in this embodiment demonstrates optimal generalization performance for pixel-by-pixel non-uniformity of varying intensities and possesses a stable ability to prevent the filtering out of weak targets during pixel-by-pixel non-uniformity correction.

[0069] Table 1. Evaluation results of corrections for low-intensity pixel-by-pixel non-uniform infrared sequence frames using various methods.

[0070]

[0071] Table 2 shows the correction evaluation results of each method for medium-intensity pixel-by-pixel non-uniform infrared sequence frames.

[0072]

[0073] Table 3 shows the correction evaluation results of each method for high-intensity pixel-by-pixel non-uniform infrared sequence frames.

[0074]

[0075] The dataset for detecting and tracking small aircraft targets in infrared images against a ground / air background contains pixel-by-pixel non-uniform noise. This experiment selects data10, data13, data11, data6, and data19 as the real sequence frame test data. Within each selected data segment, one frame is taken every 10 frames, with 7 frames constituting one test data set. Each data segment contains 70 frames, for a total of 350 frames across the 5 data segments.

[0076] Since a clean reference image is not available for real non-uniform infrared images, this experiment cannot evaluate reference metrics. In this experiment, image roughness (IR), brightness non-uniformity (Ln), structural similarity index (SSIM), and local signal-to-noise ratio (SSIM) are used to comprehensively analyze non-uniformity correction and its over-smoothing problem. Figures 8(a) to 8(f) to Figures 12(a) to 12(f) The table presents a visualization of the correction results for each method on five sequence frame data segments, with target points within red boxes and blind pixels within green circles. Tables 4 through 8 show the evaluation metrics results. Table 9 presents the average evaluation results for each method across the five data segments.

[0077] Generally, the smaller the image roughness (IR) and brightness non-uniformity (Ln), the stronger the noise removal ability of each method. However, excessively small image roughness and brightness non-uniformity may also indicate over-smoothing during the denoising process. On the other hand, since there is no clean reference image for real non-uniform images, the structural similarity index (SSIM) in this embodiment is calculated by comparing it with the original noisy image. The higher the SSIM, the smaller the image blurring, artifacts, and over-smoothing problems caused by non-uniformity. However, an excessively high SSIM may also indicate that the correction method has failed to effectively remove noise from the image. In comparison, the local signal-to-noise ratio (LSCR) is more relevant in the quantitative evaluation of non-uniformity correction for real infrared weak target images. The lower the LSCR of weak targets, the more severe the over-smoothing problem. Therefore, this experiment analyzes the non-uniformity correction and over-smoothing problems of each method based on the LSCR, combined with image roughness and structural similarity index.

[0078] The quantitative indicators from the above experimental results show that the structural similarity index of the method in this embodiment is the highest across all data segments, exceeding 0.95. This demonstrates the method's ability to avoid excessive image smoothing and preserve image details. Although the image roughness and brightness non-uniformity of the method in this embodiment are not the lowest compared to other methods, comprehensive analysis suggests that the method achieves effective non-uniformity correction. First, it can be seen that the non-uniformity noise contained in the original visualized image is relatively weak, therefore the reduction in image roughness and brightness non-uniformity after effective correction is not significant. Second, from the visualization results of the corrected image, other methods cause excessive image smoothing and blurring of image details, while the method in this embodiment not only preserves rich image details and weak targets but also effectively corrects blind pixels. In the visualization results, the corrected image without weak targets marked with red boxes indicates that the weak targets have been completely filtered out, a result also reflected in the local signal-to-noise ratio quantitative index. From the perspective of the local signal-to-noise ratio (SNR) quantitative evaluation index, other methods weaken the signal strength of weak targets to varying degrees, especially in data10, where the local SNR is the lowest. Other methods filter out weak targets entirely, while only the method in this embodiment can still prevent their removal. In summary, compared to other methods, the method in this embodiment effectively improves the over-smoothing problem commonly found in denoising methods. It can effectively filter out non-uniform noise while preserving image details and weak targets, providing reliable performance assurance and improvement for subsequent image registration and weak target detection.

[0079] Table 4 shows the evaluation metrics of each method for correcting the real sequence frame data10.

[0080]

[0081] Table 5 shows the evaluation metrics of each method for correcting the real sequence frame data13.

[0082]

[0083] Table 6 shows the evaluation metrics of each method for correcting the real sequence frame data11.

[0084]

[0085] Table 7 shows the evaluation metrics of each method for correcting the real sequence frame data6.

[0086]

[0087] Table 8 shows the evaluation metrics of each method for correcting the real sequence frame data19.

[0088]

[0089] Table 9 shows the average evaluation metrics of each method for correcting five real sequence frames.

[0090]

[0091] Example 2

[0092] Embodiment 2 of the present invention provides a terminal device corresponding to Embodiment 1 above. The terminal device can be a processing device for a client, such as a mobile phone, a laptop, a tablet computer, a desktop computer, etc., to execute the method of the above embodiments.

[0093] The terminal device in this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in Embodiment 1 described above.

[0094] In some implementations, the memory may be high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0095] In other implementations, the processor can be any type of general-purpose processor, such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation here.

[0096] Example 3

[0097] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to Embodiment 1 above, on which a computer program / instructions are stored. When the computer program / instructions are executed by a processor, they implement the steps of the method of Embodiment 1 above.

[0098] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0100] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0103] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for pixel-by-pixel non-uniformity correction of multi-frame images based on 3D residual networks, characterized in that, Includes the following steps: S1. Train a multi-frame pixel-by-pixel non-uniformity correction network using the acquired multi-frame infrared images; the multi-frame pixel-by-pixel non-uniformity correction network includes a pixel-by-pixel gain correction sub-network and a pixel-by-pixel bias correction sub-network; the pixel-by-pixel gain correction sub-network includes multiple cascaded first 3D residual units. The last first 3D residual unit is connected to the first convolutional layer; the pixel-by-pixel bias correction subnetwork includes multiple cascaded second 3D residual units, the last second 3D residual unit being connected to the second convolutional layer; S2. Input the multi-frame infrared images acquired in real time into the trained pixel-by-pixel gain correction sub-network to obtain the pixel-by-pixel gain correction factor. S3. Correct the multi-frame infrared images using the pixel-by-pixel gain correction factor to obtain gain-corrected multi-frame images; S4. Input the gain-corrected multi-frame images into the trained pixel-by-pixel bias correction sub-network to obtain the pixel-by-pixel bias correction factor. Use the pixel-by-pixel bias correction factor to correct the gain-corrected multi-frame images to obtain non-uniformity-corrected multi-frame images.

2. The multi-frame image pixel-by-pixel non-uniformity correction method based on 3D residual networks according to claim 1, characterized in that, In step S2, the first 3D residual unit and the second 3D residual unit each include multiple cascaded 3D convolutional layers, and each 3D convolutional layer is followed by a batch normalization layer and an activation function in sequence.

3. The multi-frame image pixel-by-pixel non-uniformity correction method based on 3D residual networks according to claim 1, characterized in that, The number of the first 3D residual unit and the second 3D residual unit are both 8.

4. The multi-frame image pixel-by-pixel non-uniformity correction method based on 3D residual networks according to claim 2, characterized in that, The activation function is the ReLU activation function.

5. The multi-frame image pixel-by-pixel non-uniformity correction method based on 3D residual networks according to claim 1, characterized in that, In step S2, the process of obtaining the pixel-by-pixel gain correction factor includes: The maximum and minimum values ​​of the output of one channel of the first convolutional layer are removed from the frame-by-frame average, and the average value of the frame-by-frame average after removing the maximum and minimum values ​​is used as the pixel-by-pixel gain correction factor.

6. The multi-frame image pixel-by-pixel non-uniformity correction method based on 3D residual networks according to claim 1, characterized in that, In step S2, the pixel-by-pixel gain correction factor G ij The calculation formula is: G ij =1 / g ij Among them, g ij This represents the gain of the detector element in the i-th row and j-th column of the array detector element.

7. The multi-frame image pixel-by-pixel non-uniformity correction method based on 3D residual networks according to claim 1, characterized in that, In step S4, the process of obtaining the per-pixel bias correction factor includes: removing the maximum and minimum values ​​of the output of one channel of the second convolutional layer between frames, and using the mean value of the frames after removing the maximum and minimum values ​​as the per-pixel bias correction factor.

8. The multi-frame image pixel-by-pixel non-uniformity correction method based on 3D residual networks according to claim 1, characterized in that, In step S4, the per-pixel offset correction factor O ij The calculation formula is: O ij =-o ij / g ij Among them, g ij and o ij Let i and j represent the gain and bias of each detector element, respectively, and let i and j represent the i-th row and j-th column of the area array detector element, respectively.

9. The method for pixel-by-pixel non-uniformity correction of multi-frame images based on 3D residual networks according to any one of claims 1 to 8, characterized in that, In step S4, the element in the i-th row and j-th column of the f-th frame image after non-uniformity correction... The expression is: G ij y(f) is the pixel-by-pixel gain correction factor. ij =g ij ·x(f) ij +o ij , x(f) ij and y(f) ij Let g represent the actual response and observed value of each pixel in the f-th frame, respectively. ij and o ij Let i and j represent the gain and bias of each detector element, respectively, and let i and j represent the i-th row and j-th column of the area array detector element, respectively. ij This is the pixel-by-pixel offset correction factor.

Images