Infrared area array strip noise non-uniformity correction method and device based on multiple images, and electronic equipment

Abstract

The application discloses an infrared area array strip noise non-uniformity correction method and device based on multiple images, and electronic equipment, and the method comprises the following steps: inputting N strip noise images with the same size, stacking the N strip noise images along a time axis to obtain a three-dimensional image tensor; initializing a strip noise estimation value and a strip noise weight value; calculating the gradient of each strip noise image along a strip direction to obtain N gradient images; performing iterative calculation on each strip in sequence until a preset iteration termination condition is met, and obtaining a strip noise estimation result; subtracting the strip noise estimation result from each strip noise image respectively to obtain N images after strip noise non-uniformity correction, and outputting the corrected image result. The application can overcome the influence of the dependence of the noise reduction process on training data and the random noise distribution assumption, and realize accurate separation of all strip noise and image details.

Description

Technical Field

[0001] This invention relates to a method, apparatus, and electronic device for correcting non-uniformity of infrared array stripe noise based on multi-frame images, belonging to the field of focal plane infrared image denoising technology. Background Technology

[0002] Thermal imaging instruments achieve non-contact imaging by absorbing the infrared radiation of target objects. However, due to the small temperature difference between targets, the contrast of the imaging effect is relatively poor. Although a low-noise focal plane detector is used, the output signal is still generally affected by non-uniform noise. This non-uniformity mainly stems from imperfections in the detector manufacturing process and differences in sensor response characteristics.

[0003] Among various noise types, stripe noise is particularly common in images output by focal plane array detectors. Stripe noise is distributed along a specific direction, and its grayscale value gradually becomes more pronounced as the thermal imager runs. It has a banded characteristic of alternating bright and dark areas, superimposed on the background information, and disrupts the uniformity of the image. Therefore, the actual output signal of the infrared image from the terminal of an area array instrument can be considered as the input signal being affected by the combined effects of stripe noise and random noise.

[0004] Currently, the most effective method for removing stripe noise in infrared images is blackbody calibration. Before leaving the factory, thermal imagers are uniformly calibrated against a blackbody with uniform energy radiation, and the calibration matrix is ​​saved for future initialization. However, the calibration matrix remains effective within a small range centered on a temperature point; when ambient and internal temperatures change under extreme conditions, image quality deteriorates significantly. Scene-based non-uniformity correction techniques have therefore received widespread attention, mainly including the following three categories:

[0005] Statistical methods, such as Guided Filter (GF), Midway Histogram Equalization (MHE), Block-Matching and 3D Filtering (BM3D), and Non-Local Means (NLM), are based on the assumption that all columns in thermal imaging share similar response characteristics, such as mean, standard deviation, or grayscale distribution. By constructing a mapping function that correlates paired columns, stripe noise can be effectively reduced, but these methods exhibit instability when faced with complex stripe noise and rapid scene changes.

[0006] Optimization iterative methods: such as Low-Rank-based Single-Image Decomposition (LRSID), which usually focuses on the overall changes of the image while imposing constraints related to local smoothness and sparsity of stripe noise, effectively removing stripe noise in complex scenes, but current methods lack inter-frame consistency;

[0007] Deep learning methods, such as the Strip Noise Removal Wavelet Deep Neural Network (SNRWDNN) and the Deep-Learning-based Strip NUC method (DLS-NUC), remove strip noise by learning local features through convolutional layers. However, the training of these models mainly depends on the data and the paradigm of the strip noise, making it difficult to achieve the expected results in practical applications.

[0008] In summary, the field of infrared imaging still lacks a method that can fully utilize inter-frame consistency and accurately and efficiently remove stripe noise from area array thermal imagers. Summary of the Invention

[0009] In view of the many defects and deficiencies existing in the above-mentioned background technology, the present invention has made improvements and innovations, aiming to provide a method, device and electronic device for non-uniformity correction of infrared array strip noise based on multi-frame images, so as to solve the above-mentioned related technical defects, realize the accurate separation of strip noise and real background information, and achieve noise reduction of infrared image strip noise.

[0010] To solve the above problems and achieve the above-mentioned objectives, the present invention provides a method, apparatus, and electronic device for correcting non-uniformity of infrared array stripe noise based on multi-frame images, which is achieved by adopting the following design structure and the following technical solution:

[0011] A method for correcting non-uniformity of infrared array stripe noise based on multi-frame images includes:

[0012] Input N striped noise images of the same size, stack the N striped noise images along the time axis to obtain a 3D image tensor; at the same time, initialize the striped noise estimate and striped noise weight values, which are used as input parameters for subsequent iterative calculations;

[0013] Based on stacked 3D image tensors, the gradient of each striped noise image is calculated along the strip direction to obtain N corresponding gradient images.

[0014] Based on the gradient image, the initialized strip noise estimate and the strip noise weight, the strip removal sub-model is iteratively calculated sequentially until the preset iteration termination condition is met, and the strip noise estimation result is obtained.

[0015] Subtract the strip noise estimation result from each strip noise image to obtain N images after strip noise non-uniformity correction, and output the corrected image results.

[0016] In this invention, the output corrected image is a denoised image, which is used for subsequent algorithms such as image enhancement, target recognition and classification, to synergistically improve the accuracy of the subsequent algorithms.

[0017] Preferably, a normalization operation is performed on the N input strip noise images of the same size to obtain a normalized image with values ​​ranging from 0 to 1;

[0018] Each image with striped noise corresponds to a two-dimensional data matrix, and N is a positive integer greater than 1.

[0019] Preferably, the strip noise estimate is initialized as an all-zero matrix.

[0020] Preferably, the initialization of the strip noise weight values ​​includes: if the strip noise image has not been adjusted by the histogram equalization algorithm, then the strip noise weight values ​​are initialized to an all-1 matrix;

[0021] If the striped noise image has been adjusted using a histogram equalization algorithm, then the initial striped noise weight values ​​should be set according to the following formula:

[0022]

[0023] in, It is a pixel-by-pixel exponentialization operation. It is the nth image of a stacked 3D image tensor. and Let represent the variance and mean of all pixels in the nth image, respectively. These are normalized parameters;

[0024] Normalized parameters Calculate using the following formula:

[0025]

[0026] in, It is the number of pixels in the height direction of the image. It is the number of pixels in the width direction of the image. This indicates that the nth image is in the nth position. Line number The pixel values ​​of the column.

[0027] In this invention, the weights are initialized in two ways: without prior information (using an all-one matrix) and with prior information (implemented in the gradient calculation for each striped noise image). Different initialization methods will affect the calculation results but will not affect the convergence of the algorithm, i.e., the solution possesses local uniqueness. The striped noise estimation result refers to the striped noise matrix that needs to be estimated. The striped noise image refers to the superposition result of the real image with striped noise and random noise.

[0028] Preferably, the gradient of each input striped noise image is calculated along the strip direction, and this gradient calculation step is set to be executed outside the iterative loop calculation for the subsequent strip removal sub-model, so as to reduce redundant calculations and save the overall program running time.

[0029] Preferably, the iterative calculation of each strip removal sub-model is performed sequentially, including solving the strip direction optimization sub-model, the vertical strip direction optimization sub-model, the grouped sparse optimization sub-model, and the strip noise solution sub-model sequentially.

[0030] After the solution is completed, it is determined whether the convergence value of the iteration is less than the cutoff factor or whether the number of iterations is greater than a preset value. If either condition is met, the iteration loop ends and the strip noise estimation result is output. If not, a new Lagrange multiplier is calculated and the iteration continues.

[0031] In this invention, the preset threshold includes a cutoff factor and the number of iterations.

[0032] Preferably, the preset value of the cutoff factor is 0.0001, and the preset number of iterations is 50;

[0033] The solution process of the strip direction optimization sub-model includes: performing L1 sparse regularization optimization on the gradient of the current strip noise image along the strip direction, and solving the optimization model using the quadratic Lagrange method; in this solution process, the consistency of the strip noise along the strip direction is considered, and the gradient image is solved element by element. If the iteration value of a certain element is less than a preset threshold, the solution of that element is sparsified to zero; otherwise, the original iteration value of that element is retained.

[0034] The solution process of the vertical stripe direction optimization sub-model includes: subtracting the current stripe noise image from each of the N stripe noise images to obtain N denoised images; solving the L1 sparse regularization optimization model for each of the N vertical stripe noise directions using the quadratic Lagrange method; the sub-model aims to achieve smoothing of the denoised images in the vertical stripe noise direction by solving the gradient image element by element. If the iteration value of an element is less than a preset threshold, the solution of that element is sparsified to zero; otherwise, the original iteration value of the element is retained.

[0035] The solution process of the grouped sparse optimization sub-model includes: grouping the strip noise according to the strip direction; if the strip direction is vertical, grouping by each column; if the strip direction is horizontal, grouping by each row; performing grouped L1 sparse regularization optimization on each group; and solving the optimization model using the quadratic Lagrange method. The sub-model aims to take into account the sparsity of each column or row of the strip noise, and iterates the element-wise solution column by column or row by row. If the iteration value of an element is less than a preset threshold, the solution of that element is sparsified to zero; otherwise, the original iteration value of that element is retained.

[0036] Preferably, the solution process of the strip noise solution sub-model includes: combining the solutions of the strip direction optimization sub-model, the vertical strip direction optimization sub-model, and the grouped sparse optimization sub-model during the iteration process to construct and solve the target gradient equation; when solving the target gradient equation, the Euler-Lagrange formula is used as the core solution method, and the mathematical means of Fourier transform are used for solution calculation to directly output the model solution of the target gradient equation.

[0037] In this invention, the preset threshold of the cutoff factor and the preset number of iterations can be changed according to specific circumstances. Each sub-model characterizes the distribution features of strip noise in different image directions, which helps to comprehensively solve the strip noise image and achieve the goal of restoring the real image; it solves the defects of overestimation or underestimation in single-frame image denoising under strong noise conditions, realizes accurate separation of strip noise from real background information, and achieves denoising of strip noise in infrared images.

[0038] Preferably, an infrared array stripe noise non-uniformity correction device based on multi-frame images includes:

[0039] The image input processing unit is used to perform standardization operations on the input striped noise image and calculate the gradient image of each striped noise image in the strip direction;

[0040] The parameter input unit is used to receive external instructions to adjust the correction parameters and set the weight setting method for strip noise calculation;

[0041] The strip noise model calculation unit is used to construct a strip noise constraint model, which includes a strip direction optimization sub-model, a vertical strip direction optimization sub-model, a grouped sparse optimization sub-model, and a strip noise sub-model. The strip noise model calculation unit is also used to iteratively solve each sub-model in the strip noise constraint model and update the Lagrange parameters according to the weights set by the parameter input unit and the gradient image calculated by the image input processing unit, and output the strip noise estimation result.

[0042] The image denoising unit is used to subtract the stripe noise estimation result from the original image in the image input unit to obtain the corrected image result.

[0043] Preferably, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor: characterized in that the processor executes the program to implement the steps of the method according to any one of claims 1 to 8.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. The correction method of the present invention supports processing any number of neighboring frames and does not require complex inter-frame registration, which greatly reduces the computational complexity and implementation difficulty of the algorithm.

[0046] 2. This invention effectively avoids the problem of underestimation or overestimation of strip noise in local areas by integrating the redundant information of adjacent frames into a unified model, and has excellent temporal consistency.

[0047] 3. This invention overcomes the instability of traditional single-frame noise reduction algorithms during scene motion, ensuring smooth and continuous correction effects in dynamic scenes;

[0048] 4. This invention does not require training on a large dataset, thus avoiding the dependence of deep learning algorithms on specific noise paradigms and having stronger environmental adaptability and engineering practicality.

[0049] 5. This invention does not impose any prior assumptions regarding the random noise distribution, but only relies on the constraints of the image gradient in each direction, making the algorithm have good universality for images with different noise levels;

[0050] 6. This invention accurately characterizes the structural features of strip noise by introducing a first-order continuity constraint along the strip direction, thereby improving the accuracy of noise estimation.

[0051] 7. This invention utilizes the first-order continuity constraint of the denoised image in the vertical strip direction to remove noise while preserving the details of the original scene to the maximum extent and preventing image blurring.

[0052] 8. This invention combines the sparsity characteristics of strip noise in the strip direction, which can more accurately identify and separate non-uniform stripes and reduce the false exposure to background information.

[0053] 9. This invention uses the joint constraints of multi-dimensional sub-models to accurately characterize the strip noise from different directions, and exhibits superior noise reduction performance compared with existing baseline methods;

[0054] 10. The overall solution of this invention has extremely high robustness and can maintain high uniformity and high definition of the area array infrared image under extreme environmental temperature changes or complex scenarios. Attached Figure Description

[0055] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:

[0056] Figure 1 This is a flowchart illustrating the correction method of the present invention;

[0057] Figure 2 This is a schematic diagram of the iterative solution steps of the correction method of the present invention;

[0058] Figure 3 This is one of the comparison diagrams between the correction method of this invention and other algorithms;

[0059] Figure 4 This is the second comparison diagram between the correction method of this invention and other algorithms;

[0060] Figure 5 This is the third comparison diagram between the correction method of this invention and other algorithms;

[0061] Figure 6 This is a schematic diagram of the correction device structure of the present invention;

[0062] Figure 7 This is a schematic diagram of the calibration electronic device of the present invention;

[0063] Figure 8 This is an example of the calibration software interface of the present invention;

[0064] In the figure, the methods represented by the labels are as follows: A—NLM (Non-Local Mean Method), B—BM3D (Three-Dimensional Block Filtering Method), C—MHE (Median Histogram Balancing Method), D—GF (One-Dimensional Guided Filtering Method), E—LRSID (Low-Rank Single Image Decomposition), F—SNRWDNN (Strip Removal Based on Wavelet Transform Deep Neural Network), G—DLS-NUC (Deep Learning-Based Strip Non-Uniformity Correction Method), and H—The correction method of this invention. Detailed Implementation

[0065] To make the technical means, inventive features, objectives, and effects of this invention readily understandable, the technical solution of this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other. The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0066] like Figure 1As shown, an infrared array stripe noise non-uniformity correction method based on multi-frame images includes:

[0067] S1: Input N striped noise images of the same size, stack the N striped noise images along the time axis to obtain a three-dimensional image tensor; at the same time, initialize the striped noise estimate and striped noise weight values, which are used as input parameters for subsequent iterative calculations;

[0068] S2, based on stacked 3D image tensors, calculate the gradient of each striped noise image along the strip direction to obtain N corresponding gradient images;

[0069] S3, based on the gradient image, the initialized strip noise estimate and the strip noise weight value, performs iterative calculations on each strip removal sub-model in sequence until the preset iteration termination condition is met, and obtains the strip noise estimation result;

[0070] S4. Subtract the strip noise estimation result from each strip noise image to obtain N images after strip noise non-uniformity correction, and output the corrected image results.

[0071] In this invention, the output corrected image is a denoised image, which is used for subsequent algorithms such as image enhancement, target recognition and classification, to synergistically improve the accuracy of the subsequent algorithms.

[0072] Furthermore, a normalization operation is performed on the N input strip noise images of the same size to obtain a normalized image with values ​​ranging from 0 to 1;

[0073] Each image with striped noise corresponds to a two-dimensional data matrix, and N is a positive integer greater than 1;

[0074] Furthermore, the strip noise estimate is initialized as an all-zero matrix.

[0075] Furthermore, the initialization of the strip noise weight values ​​includes: if the strip noise image has not been adjusted by the histogram equalization algorithm, then the strip noise weight values ​​are initialized to an all-1 matrix;

[0076] If the striped noise image has been adjusted using a histogram equalization algorithm, then the initial striped noise weight values ​​should be set according to the following formula:

[0077]

[0078] in, It is a pixel-by-pixel exponentialization operation. It is the nth image of a stacked 3D image tensor. and Let represent the variance and mean of all pixels in the nth image, respectively. These are normalized parameters.

[0079] In this invention, the weights are initialized in two ways: without prior information (using an all-one matrix) and with prior information (implemented in the gradient calculation for each striped noise image). Different initialization methods will affect the calculation results but will not affect the convergence of the algorithm, i.e., the solution possesses local uniqueness. The striped noise estimation result refers to the striped noise matrix that needs to be estimated. The striped noise image refers to the superposition result of the real image with striped noise and random noise.

[0080] Specifically, the image in step S1 (denoted as...) ,high ,Width The quantity N can theoretically be any size (N is at least 1), and this application sets N to 5. This is an empirical value setting, and users can adjust it appropriately according to the actual image noise intensity.

[0081] Specifically, the initial values ​​of the striped noise image in step S1 (not generally extremely...) The setting of ) is usually set to an all-zero matrix when there is no prior information, that is Other values ​​can also be set.

[0082] Specifically, the strip noise weight setting in step S1 takes into account that the original image output by the focal plane detector is 14-bit. After adjustment by algorithms such as histogram equalization, the strip noise in the extremely dark or bright parts of the 8-bit image is not obvious. Therefore, the algorithm allows the weight of the strip noise solution to be adjusted according to the brightness of different regions in multiple images. Therefore, the recommended weight setting for actual use is:

[0083]

[0084] in, It is a pixel-by-pixel exponentialization operation. It is the nth image of a stacked 3D image tensor. and Let represent the variance and mean of all pixels in the nth image, respectively. These are the normalized parameters; meanwhile, if the image has not been adjusted by histogram equalization or similar operations, then... It can be set as an all-one matrix, that is .

[0085] Among them, the normalized parameters Calculate using the following formula:

[0086]

[0087] in, It is the number of pixels in the height direction of the image. It is the number of pixels in the width direction of the image. This indicates that the nth image is in the nth position. Line number The pixel values ​​of the column.

[0088] Furthermore, by calculating the gradient of each input striped noise image along the strip direction and setting this gradient calculation step to be executed outside the iterative loop calculation for the subsequent strip removal sub-model, redundant computation is reduced and the overall program running time is saved.

[0089] Specifically, step S2 involves pre-computing each image. The gradient along the strip direction is used to simplify the calculation steps and accelerate the algorithm in subsequent iterations.

[0090] Furthermore, the iterative calculation is performed sequentially on each strip removal sub-model, including sequentially solving the strip direction optimization sub-model, the vertical strip direction optimization sub-model, the grouped sparse optimization sub-model, and the strip noise solution sub-model;

[0091] After the solution is completed, it is determined whether the convergence value of the iteration is less than the cutoff factor or whether the number of iterations is greater than a preset value. If either condition is met, the iteration loop ends and the strip noise estimation result is output. If not, a new Lagrange multiplier is calculated and the iteration continues.

[0092] In this invention, the preset threshold includes a cutoff factor and the number of iterations.

[0093] Specifically, the preset value of the cutoff factor is 0.0001, and the preset number of iterations is 50;

[0094] The solution process of the strip direction optimization sub-model includes: performing L1 sparse regularization optimization on the gradient of the current strip noise image along the strip direction, and solving the optimization model using the quadratic Lagrange method; in this solution process, the consistency of the strip noise along the strip direction is considered, and the gradient image is solved element by element. If the iteration value of a certain element is less than a preset threshold, the solution of that element is sparsified to zero; otherwise, the original iteration value of that element is retained.

[0095] The solution process of the vertical stripe direction optimization sub-model includes: subtracting the current stripe noise image from each of the N stripe noise images to obtain N denoised images; solving the L1 sparse regularization optimization model for each of the N vertical stripe noise directions using the quadratic Lagrange method; the sub-model aims to achieve smoothing of the denoised images in the vertical stripe noise direction by solving the gradient image element by element. If the iteration value of an element is less than a preset threshold, the solution of that element is sparsified to zero; otherwise, the original iteration value of the element is retained.

[0096] The solution process of the grouped sparse optimization sub-model includes: grouping the strip noise according to the strip direction; if the strip direction is vertical, grouping by each column; if the strip direction is horizontal, grouping by each row; performing grouped L1 sparse regularization optimization on each group; and solving the optimization model using the quadratic Lagrange method. The sub-model aims to take into account the sparsity of each column or row of the strip noise, and iterates the element-wise solution column by column or row by row. If the iteration value of an element is less than a preset threshold, the solution of that element is sparsified to zero; otherwise, the original iteration value of the element is retained.

[0097] Specifically, the solution process of the strip noise solution sub-model includes: combining the solutions of the strip direction optimization sub-model, the vertical strip direction optimization sub-model, and the grouped sparse optimization sub-model during the iteration process to construct and solve the target gradient equation; when solving the target gradient equation, the Euler-Lagrange formula is used as the core solution method, and Fourier transform is used for solution calculation to directly output the model solution of the target gradient equation.

[0098] In this invention, the preset threshold of the cutoff factor and the preset number of iterations can be changed according to specific circumstances. Each sub-model characterizes the distribution features of strip noise in different image directions, which helps to comprehensively solve the strip noise image and achieve the goal of restoring the real image; it solves the defects of overestimation or underestimation in single-frame image denoising under strong noise conditions, realizes accurate separation of strip noise from real background information, and achieves denoising of strip noise in infrared images.

[0099] Specifically, step S4 involves the solution and iteration of four core subroutines; this invention integrates three characteristics of strip noise: strip noise has continuous characteristics in the strip direction, specifically for the first... No. Strip noise of the column In the strip direction ( gradient of direction The smaller the value, the better the continuity in the strip direction. To initialize the input stripe noise weights at the th No. The value of the column, For the current stripe noise estimation, the th No. The value of the column, To initialize the input stripe noise weights at the th No. The value of the column, For the current stripe noise estimation, the th No. The values ​​of the columns. The matrix form of the above constraints. The minimum value needs to be reached, where, These are the initial stripe noise weight values. The gradient operator represents the gradient of an image along the stripe direction. Represents the product of matrix elements pixel by pixel, if , then the corresponding , , and They are matrices , and In the Line number Column elements; This represents an L1 regularization constraint, specifically for Calculation The denoised image along the vertical strip noise direction has a smoothing feature. Specifically, for the nth denoised image... In its first No. Vertical strip direction of the column ( gradient of direction The smaller the value, the better its continuity in the direction perpendicular to the stripes. For the nth striped noise image at the th No. The value of the column, For the current stripe noise estimation, the th No. The value of the column, For the nth striped noise image at the th No. The value of the column, For the current stripe noise estimation, the th No. The values ​​of the columns. The above constraints apply to a matrix form of N images. To achieve the minimum value, where, Operators indicating the direction of vertical stripes, It is the nth image stacked on the time axis, which is the input noisy image; in addition, the strip noise has grouping sparsity. It needs to reach the minimum value, and This indicates that L1 regularization constraints are applied to image column groupings, i.e., for a matrix... , The three objective functions are weighted by the Lagrange algorithm. and The overall optimization model is obtained by balancing the equilibrium:

[0100] The overall optimization model is as follows:

[0101]

[0102] in, The gradient operator represents the gradient of the image along the strip direction, and correspondingly... Operators indicating the direction of vertical stripes; Represents the product of matrix elements pixel by pixel; and This represents the balance parameter between different objectives; This represents an L1 regularization constraint, and This indicates that L1 regularization constraints are applied to the grouping; using the second-order Lagrange multiplier optimization model, it can be transformed into:

[0103]

[0104]

[0105] By changing the model, the original problem is transformed into a problem concerning... The solution is given. To further illustrate this in detail, we provide... Figure 2 The process was further analyzed. It is important to note that there is no specific solution order for the strip direction optimization sub-model, the vertical strip direction optimization sub-model, and the grouped sparse optimization sub-model. Figure 2 This only provides one form; the solutions to these three sub-models are independent of each other, and can be obtained by following... Figure 2 The steps are performed in the order shown, and can also be executed in parallel, but all must be completed before solving the striped noise sub-model; without loss of generality, this application still follows the... Figure 2 This will explain the key steps in solving the sub-model.

[0106] like Figure 2 As shown, S301 refers to the sub-model for solving the strip direction optimization problem, which contains relevant parameters in the above model. The solution to this problem can be broken down into the following subproblems:

[0107]

[0108] in, The parameters to be solved are... and Denotes the parameters in a quadratic Lagrange optimization. It is the square of the L2 norm, and correspondingly, for a matrix... In other words, , The input strip noise weights, Represents the product of matrix elements pixel by pixel. The current strip noise estimate in the strip direction ( The gradient of the direction, and the iterative solution to the above subproblem are:

[0109]

[0110] in, Representing a symbolic function, correspondingly, This involves determining the sign of each element in matrix X, that is, for the first element... Line number Column elements , ,when ; ,when ; ,when ;

[0111] Representing the absolute value function, correspondingly, This involves performing an absolute value comparison on each element of matrix X, that is, for the ... Line number Column elements , ,when ; ,when ; ,when ;max represents the maximum operator, and the corresponding For matrices and For each element in the set, the largest value is determined; that is, for the first element... Line number Column elements and , ,if ; ,if ; This means performing calculations on each element of the matrix, specifically for the first element. No. Column values ,in, Is the current stripe noise at the th No. The direction of the stripes of the column pixels ( gradient of direction It is a parameter In the No. The value of the column; This means performing calculations on each element of the matrix; specifically, on the matrix... In the No. Column value elements ,in Is the strip noise weight in the th order? No. The value of the column pixels. Overall parameters The Line number The estimated value of the column is Thus, the iterative formula for the strip direction optimization sub-model is obtained.

[0112] like Figure 2 As shown, S302 refers to the sub-model for solving the vertical strip direction optimization problem, which contains relevant parameters in the above model. Without loss of generality, the solution is as follows: Let's take an example to explain; its sub-problems can be separated into:

[0113]

[0114] in, The parameters to be solved, and Denotes the parameters in a quadratic Lagrange optimization. It is the square of the L2 norm, and correspondingly, for a matrix... In other words, , For the current denoised image in the vertical strip direction ( The gradient of the direction, and the iterative solution to the above subproblem are:

[0115]

[0116] in, Representing a symbolic function, correspondingly, This involves determining the sign of each element in matrix X, that is, for the first element... Line number Column elements , ,when ; ,when ; ,when ;

[0117] Representing the absolute value function, correspondingly, This involves performing an absolute value comparison on each element of matrix X, that is, for the ... Line number Column elements , ,when ; ,when ; ,when ;max represents the maximum operator, and the corresponding For matrices and For each element in the set, the largest value is determined; that is, for the first element... Line number Column elements and , ,if ; ,if ; This means performing calculations on each element of the matrix, specifically for the first element. No. Column values ),in, For the nth striped noise image at the th No. The value of the column, For the current stripe noise at the th No. The value of the column; This means performing calculations on all elements of the matrix; specifically, , It is a parameter In the No. The values ​​of the column. Overall, the parameters... The Line number The estimated value of the column is Thus, the iterative formulas for optimizing the direction of N vertical stripes are obtained.

[0118] like Figure 2 As shown, S303 refers to the solution of the grouped sparse optimization sub-model, which contains relevant parameters in the above model. The solution; its subproblems can be separated into:

[0119]

[0120] in, The matrix to be solved No. A vector of columns; and This represents the parameters in a quadratic Lagrange optimization. It is a parameter The Column vector; This indicates that L1 regularization constraints are applied to image column groupings, i.e., for a matrix... , ; and Represent matrices respectively and of For column vectors, the iterative solution to the above subproblem is as follows:

[0121]

[0122] in, Representing a symbolic function, correspondingly, This involves determining the sign of each element in matrix X, that is, for the first element... Line number Column elements , ,when ; ,when ; ,when ;

[0123] Representing the absolute value function, correspondingly, This involves performing an absolute value comparison on each element of matrix X, that is, for the ... Line number Column elements , ,when ; ,when ; ,when ;max represents the maximum operator, and the corresponding For matrices and For each element in the set, the largest value is determined; that is, for the first element... Line number Column elements and , ,if ; ,if ; Represents the current estimation matrix for striped noise. The List Perform operations on all elements, for No. Each element according to Calculate, where, This is the current strip noise estimation. In the Line number Column elements, It is a parameter matrix In the Line number The elements of the column; parameters in general. The Line number The estimated value of the column can be calculated using the following formula: Therefore, the iterative formula for the grouped sparse optimization sub-model is obtained.

[0124] After steps S301, S302, and S303, it is necessary to separate the relevant strip noise from the original model. The optimization is S304:

[0125]

[0126] in, Optimize the solution of the sub-model for the strip direction; The solution for optimizing the sub-model in the direction of the nth vertical stripe; This is the solution for the grouped sparse optimization sub-model. Solving this requires the use of Fourier transform; specifically, the Euler-Lagrange method is applied to solving equations containing gradient information, such as a functional equation. exist There is a global minimum; therefore, the following formula should be used to solve it. In the strip solution sub-model: It is the optimization function of S304, for strip noise estimation. That is the formula above. After differentiation, the following equation is obtained:

[0127]

[0128] Unlike solving general equations, this requires the application of the Discrete Fourier Transform. .in, For two-dimensional coordinate points in the frequency domain, It is a two-dimensional image In the, It is an image of high, It is an image width, The imaginary unit satisfies Through discrete Fourier transform, the following solution is finally obtained:

[0129]

[0130] in, This indicates element-wise calculation; This represents the Fourier transform function described above; This represents the inverse Fourier transform, i.e., the transformation of an image. Spectrum In other words, its inverse Fourier transform is ; Along the strip direction ( The gradient operator (direction); The gradient transpose operator is used along the strip direction in this algorithm. ; Vertical strip direction ( The gradient operator (direction); The gradient transpose operator is used in the direction perpendicular to the stripe. In this algorithm, , , , , and It is a second-order Lagrange parameter matrix; It is the input stripe noise weight matrix; Optimize the solution of the sub-model for the strip direction; The solution for optimizing the sub-model in the direction of the nth vertical stripe; This is the solution for the grouped sparse optimization sub-model.

[0131] After the iterative calculation and update in S304, the program will enter the determination stage in S305. When two consecutive... Second and The strip noise estimate is: and Then, the convergence criterion is as follows: Condition 1:

[0132]

[0133] in, It is the first The L1 norm of the sub-strip noise estimation, i.e. , It is the first Sub-strip noise estimation in the th Line number Estimated values ​​of column elements; It is the first Second and The L1 norm of the difference between the estimated stripe noise values, i.e. . To prevent the denominator from being zero, this parameter is typically set to 1e-12. The convergence threshold is set to a value between 0 and 1, with 0.0001 being recommended.

[0134] Condition 2: This refers to the current iteration number k. If it is less than a set threshold, such as K, then... ,recommend ;

[0135] If both conditions 1 and 2 are met, proceed to the S306 Lagrange parameter update process; otherwise, proceed to... Figure 1 The S4 step outputs an iterative strip noise estimate. .

[0136] If the S306 Lagrange parameter update process is initiated, the parameters will be updated according to the following equation from the first... Updated to [number] Second-rate:

[0137]

[0138] in , , For the first The Lagrange parameter of the order of magnitude; , , For the first The Lagrange parameter of the order of 1. , and It is the Lagrange parameter that adjusts the iteration rate; Indicates element-wise calculation; Along the strip direction ( The gradient operator (direction); Vertical strip direction ( The gradient operator (direction); It is the input stripe noise weight matrix; Optimize the solution of the sub-model for the strip direction; The solution for optimizing the sub-model in the direction of the nth vertical stripe; This is the solution for the grouped sparse optimization sub-model.

[0139] Because the iteration process has not ended, the program will enter S301 to proceed to the next cycle of calculation.

[0140] If S305 determines that entry is possible Figure 1 In step S4, the current striped noise image is output. The following operation is performed on each original noisy image to obtain the denoised image:

[0141]

[0142] in, For the nth image with striped noise, For the strip noise estimation of the algorithm, To output the nth denoised image.

[0143] like Figures 3 to 5 The figures shown are three specific examples of the implementation of this method. The related technical methods include: A represents NLM, B represents BM3D, C represents MHE, D represents GF, E represents LRSID, F represents SNRWDNN, G represents DLS-NUC, and H represents the correction method of this application.

[0144] like Figure 3 The example shown compares the algorithms using simulated infrared images containing striped noise. NLM, BM3D, GF, and MHE all exhibit noticeable residual striped noise in their denoising results. LRSID, SNRWDNN, and DLS-NUC, however, show less noise. Figure 3 The shoulder area of ​​the figure in the previous image has artifacts, and the noise reduction image destroys the integrity of the original figure's edges; conversely, the method proposed in this application preserves the image details well while also removing stripe noise.

[0145] like Figure 4 The second example illustrates a comparison of noise reduction algorithms for infrared images of water heaters captured by the Kunming Institute of Physics' self-developed mid-wave micro-scanning area array infrared thermal imager. NLM excessively smooths local textures, damaging image details, particularly blurring the Chinese characters on the water heater in the denoised image. BM3D, GF, and MHE still exhibit noticeable residual stripe noise in their noise reduction results, especially in the background sky. LRSID shows excessive noise reduction in areas with antennas, particularly in the sky above the antennas, resulting in artifacts. SNRWDNN and DLS-NUC have good overall noise reduction effects, but still exhibit "ringing" artifacts around the antennas. Conversely, the method proposed in this application effectively preserves image details while also removing stripe noise.

[0146] like Figure 5 As an example three, this paper compares the noise reduction algorithms for infrared images of walls captured by the mid-wave micro-scanning area array infrared thermal imager developed by Kunming Institute of Physics. Among them, NLM excessively smooths local textures, damaging image details, especially blurring the edges of glass; BM3D and GF still exhibit visible striped noise residue on the front of the wall; MHE and LRSID show artifacts "intruding" into the wall below near the window frame in the upper right corner of the image; SNRWDNN and DLS-NUC have good overall noise reduction effects, but still exhibit "ringing" artifacts around the edges of the wall. Conversely, the method proposed in this application effectively preserves image details while also removing striped noise.

[0147] In summary, the method provided in this application significantly improves the signal-to-noise ratio (PSNR) and structural similarity (SSIM) indicators, while effectively removing stripe noise from infrared images of area array instruments. It also addresses issues such as grayscale distortion and loss of image detail information, outperforming currently disclosed benchmark stripe noise removal methods.

[0148] The following are embodiments of the apparatus of this application, which are used to execute the method of this application. For details not disclosed in the comparative software of this application, please refer to the method embodiments of this application.

[0149] An infrared array stripe noise non-uniformity correction device based on multi-frame images, comprising:

[0150] The image input processing unit is used to perform standardization operations on the input striped noise image and calculate the gradient image of each striped noise image in the strip direction;

[0151] The parameter input unit is used to receive external instructions to adjust the correction parameters and set the weight setting method for strip noise calculation;

[0152] The strip noise model calculation unit is used to construct a strip noise constraint model, which includes a strip direction optimization sub-model, a vertical strip direction optimization sub-model, a grouped sparse optimization sub-model, and a strip noise sub-model. The strip noise model calculation unit is also used to iteratively solve each sub-model in the strip noise constraint model and update the Lagrange parameters according to the weights set by the parameter input unit and the gradient image calculated by the image input processing unit, and output the strip noise estimation result.

[0153] The image denoising unit is used to subtract the stripe noise estimation result from the original image in the image input unit to obtain the corrected image result.

[0154] Specifically, such as Figure 6 As shown, this application also provides a schematic diagram of an infrared array stripe noise non-uniformity correction device based on multi-frame images. This device can be implemented in whole or in part as a terminal through a combination of software and hardware, and can also be integrated as an independent module into a computer or server. In this application example, the multi-frame infrared array stripe noise non-uniformity correction device can be applied to a terminal computer or cloud server. The device 60 includes an image input processing unit 601, a parameter input unit 602, a stripe noise model calculation unit 603, and an image denoising unit 604.

[0155] The image input processing unit 601 is used to perform standardization operations on the input image;

[0156] The image input processing unit 601 is also used to calculate the gradient image of each original image in the strip direction;

[0157] The parameter input unit 602 adjusts the parameters using the method of this application;

[0158] The parameter input unit 602 is also used to receive external instructions to set the weight setting method for strip noise calculation.

[0159] The strip noise model calculation unit 603 is used to construct the strip noise constraint model and separate the strip direction optimization sub-model, the vertical strip direction optimization sub-model, the grouped sparse optimization sub-model, and the strip noise sub-model.

[0160] The strip noise model calculation unit 603 is also used to iteratively solve each sub-model of the strip constraint and update the Lagrange parameters, and output the strip noise image.

[0161] The image noise reduction unit 604 is used to subtract the striped noise image from the original image in the image input unit 601 to obtain the corrected image result.

[0162] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described in the above embodiments of this application.

[0163] Specifically, such as Figure 7 As shown, the electronic device 700 includes a processor 701 and a memory 702.

[0164] In this embodiment, the processor 701 is the core control center of the computer system. It can be a physical processor or a processor in a virtual environment. 701 can include one or more computing processing cores, including but not limited to a 4-core processor or an 8-core processor. The processor 701 can also be a processor that processes data in a wake-up state, also known as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0165] The memory 702 may include at least one computer-readable storage medium, which may be non-transitory; the memory 702 may include high-speed random access memory, non-volatile memory, i.e., at least one mechanical hard disk storage or flash memory device. In the above embodiments of this application, the non-transitory memory 702 is used to store code instructions for implementing the process executed by the processor 701.

[0166] In some embodiments, device 700 further includes peripheral device interface 703 and at least one peripheral device 704; peripheral device 704 includes a display, a USB flash drive, hard disk drive, optical disk, DVD, CD-ROM or any other type of media or device that can be externally stored for visual output of data, and for connecting and inputting external data; peripheral device interface 703 can be connected to processor 701 and memory 702 via bus or signal lines.

[0167] In some embodiments of this application, the processor 701, memory 702, and peripheral device interface 703 are integrated on the same chip or circuit board; in other embodiments, any one or two of the processor 701, memory 702, and peripheral device interface 703 can be implemented on separate chips or circuit boards. This application does not specifically limit this.

[0168] The electronic device structural diagrams in the embodiments of this application do not limit the components constituting the electronic device 700. The device 700 may contain more or fewer components, or combinations and arrangements of the components.

[0169] The computer program used to implement the above-described example method steps in this application embodiment can be stored in the memory 702 inside the device 700 or loaded by connecting to an external device 704 through the external device interface 703.

[0170] To further explain, Figure 8 Embodiments of a method, apparatus, and electronic device for infrared array stripe noise non-uniformity correction based on multi-frame images are provided. The method of this application is designed with a visual operating program on a personal computer according to the above-described apparatus structure diagram. Those skilled in the art can interactively input images into the local memory 702 or upload images to the processor 701 via the peripheral device 704 through the peripheral device interface 703 for processing. The results are ultimately returned to the memory 702 and transmitted to the peripheral device display via the interface 703 to demonstrate the noise reduction effect.

[0171] Through the description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented using software and related hardware platforms. Based on this understanding, the essence of the above technical solutions, or the contributions made by related technologies, can be embodied in the form of a software product. This software product can be stored in a computer-readable storage medium, such as an optical disc or hard disk, and includes several instructions that cause a computing device (including a computer, server, etc.) to execute the various embodiments or parts thereof.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and have no other limitations. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the above embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for correcting non-uniformity of infrared array stripe noise based on multi-frame images, characterized in that: include: Input N striped noise images of the same size, stack the N striped noise images along the time axis to obtain a three-dimensional image tensor; Simultaneously initialize the strip noise estimate and strip noise weight values, which will be used as input parameters for subsequent iterative calculations; Based on stacked 3D image tensors, the gradient of each striped noise image is calculated along the strip direction to obtain N corresponding gradient images. Based on the gradient image, the initialized strip noise estimate and the strip noise weight, the strip removal sub-model is iteratively calculated sequentially until the preset iteration termination condition is met, and the strip noise estimation result is obtained. Subtract the strip noise estimation result from each strip noise image to obtain N images after strip noise non-uniformity correction, and output the corrected image results.

2. The correction method according to claim 1, characterized in that: Normalization is performed on the N input striped noise images of the same size to obtain normalized images with values ​​ranging from 0 to 1; Each image with striped noise corresponds to a two-dimensional data matrix, and N is a positive integer greater than 1.

3. The correction method according to claim 1, characterized in that: The strip noise estimate is initialized as an all-zero matrix.

4. The correction method according to claim 1, characterized in that: The initialization of the strip noise weight values ​​includes: if the strip noise image has not been adjusted by the histogram equalization algorithm, then the strip noise weight values ​​are initialized to an all-1 matrix; If the striped noise image has been adjusted using a histogram equalization algorithm, then the initial striped noise weight values ​​should be set according to the following formula: ， in, It is a pixel-by-pixel exponentialization operation. It is the nth image of a stacked 3D image tensor. and Let represent the variance and mean of all pixels in the nth image, respectively. These are normalized parameters; Normalized parameters Calculate using the following formula: ， in, It is the number of pixels in the height direction of the image. It is the number of pixels in the width direction of the image. This indicates that the nth image is in the nth position. Line number The pixel values ​​of the column.

5. The correction method according to claim 1, characterized in that: By calculating the gradient of each input striped noise image along the strip direction and setting this gradient calculation step to be executed outside the iterative loop calculation for the subsequent strip removal sub-model, redundant computation is reduced and the overall program running time is saved.

6. The correction method according to claim 1, characterized in that: The iterative calculation of each strip removal sub-model is performed sequentially, including solving the strip direction optimization sub-model, the vertical strip direction optimization sub-model, the grouped sparse optimization sub-model, and the strip noise solution sub-model sequentially. After the solution is completed, it is determined whether the convergence value of the iteration is less than the cutoff factor or whether the number of iterations is greater than a preset value. If either condition is met, the iteration loop ends and the strip noise estimation result is output. If not, a new Lagrange multiplier is calculated and the iteration continues.

7. The correction method according to claim 6, characterized in that: The preset value of the cutoff factor is 0.0001, and the preset number of iterations is 50; The solution process of the strip direction optimization sub-model includes: performing L1 sparse regularization optimization on the gradient of the current strip noise image along the strip direction, and solving the optimization model using the quadratic Lagrange method; in this solution process, the consistency of the strip noise along the strip direction is considered, and the gradient image is solved element by element. If the iteration value of a certain element is less than a preset threshold, the solution of that element is sparsified to zero; otherwise, the original iteration value of that element is retained. The solution process of the vertical stripe direction optimization sub-model includes: subtracting the current stripe noise image from each of the N stripe noise images to obtain N denoised images; solving the L1 sparse regularization optimization model for each of the N vertical stripe noise directions using the quadratic Lagrange method; the sub-model aims to achieve smoothing of the denoised images in the vertical stripe noise direction by solving the gradient image element by element. If the iteration value of an element is less than a preset threshold, the solution of that element is sparsified to zero; otherwise, the original iteration value of the element is retained. The solution process of the grouped sparse optimization sub-model includes: grouping the strip noise according to the strip direction; if the strip direction is vertical, grouping by each column; if the strip direction is horizontal, grouping by each row; performing grouped L1 sparse regularization optimization on each group; and solving the optimization model using the quadratic Lagrange method. The sub-model aims to balance the sparsity of each column or row of the strip noise, and iterates the element-wise for each column or row of the strip noise. If the iteration value of an element is less than a preset threshold, the solution of that element is sparsified to zero; otherwise, the original iteration value of that element is retained.

8. The correction method according to claim 6 or 7, characterized in that: The solution process of the strip noise solution sub-model includes: combining the solutions of the strip direction optimization sub-model, the vertical strip direction optimization sub-model, and the grouped sparse optimization sub-model during the iteration process to construct and solve the target gradient equation; when solving the target gradient equation, the Euler-Lagrange formula is used as the core solution method, and Fourier transform is used for solution calculation to directly output the model solution of the target gradient equation.

9. A device for correcting non-uniformity of infrared array stripe noise based on multi-frame images, characterized in that: include: The image input processing unit is used to perform standardization operations on the input striped noise image and calculate the gradient image of each striped noise image in the strip direction; The parameter input unit is used to receive external instructions to adjust the correction parameters and set the weight setting method for strip noise calculation; The strip noise model calculation unit is used to construct a strip noise constraint model, which includes a strip direction optimization sub-model, a vertical strip direction optimization sub-model, a grouped sparse optimization sub-model, and a strip noise sub-model. The strip noise model calculation unit is also used to iteratively solve each sub-model in the strip noise constraint model and update the Lagrange parameters according to the weights set by the parameter input unit and the gradient image calculated by the image input processing unit, and output the strip noise estimation result. The image denoising unit is used to subtract the stripe noise estimation result from the original image in the image input unit to obtain the corrected image result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor: characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 8.

Images