Infrared image dynamic non-uniformity correction method and device

Abstract

The application provides an infrared image dynamic non-uniformity correction method and device, the method comprising: acquiring an original infrared image and extracting a reference temperature characteristic value thereof; inputting the reference temperature characteristic value and a preset position encoding matrix into a trained solidification response fitting model for prediction processing to obtain a target background base map of the original infrared image under the reference temperature characteristic value; the solidification response fitting model comprising a static parameter setting layer, a dynamic parameter prediction layer and a background base map synthesis layer, the solidification static tensor being stored in the static parameter setting layer, the dynamic parameter prediction layer being constructed based on a preset multi-layer perception network, and the preset position encoding matrix being generated based on the column dimension of a focal plane array in an infrared detector; and correcting the original infrared image based on the target background base map to obtain a corrected infrared image. The scheme of the application can realize continuous and high-precision non-uniformity correction in a full temperature range, while reducing the calculation amount and storage amount.

Description

Technical Field

[0001] This invention relates to the field of infrared imaging and image processing technology, and in particular to a method and apparatus for correcting dynamic non-uniformity of infrared images. Background Technology

[0002] In infrared imaging systems, due to limitations in manufacturing processes and the temperature sensitivity of detector materials, individual pixels in an infrared focal plane array will exhibit different responses under the same environmental radiation, a phenomenon known as "non-uniformity." This typically manifests as fixed-pattern noise in the image (such as stripes, dead pixels, etc.). More seriously, this non-uniformity can also dynamically change as the detector's operating temperature drifts.

[0003] Existing non-uniformity correction methods mainly suffer from the following drawbacks:

[0004] 1. Traditional multi-point calibration methods (such as two-point method and multi-point method): These methods require collecting blackbody data at multiple fixed temperatures and generating large lookup tables for gain and bias coefficients. They consume a large amount of hardware storage space and can only perform simple linear interpolation between calibration temperature points, making it impossible to accurately fit the nonlinear response drift of the detector.

[0005] 2. Scene-based noise reduction (SBNUC): This method relies on scene motion to statistically eliminate noise. It is prone to producing "ghosting" when the camera is stationary or when observing a flat scene, resulting in blurred image edges.

[0006] 3. Traditional deep learning methods (such as CNN-based models): These methods typically input noisy images into deep convolutional neural networks and directly output corrected images. Such methods have extremely high computational complexity, making it difficult to achieve high frame rate real-time processing on edge computing platforms (such as low-power FPGAs or DSPs) for infrared sensors. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method and apparatus for dynamic non-uniformity correction of infrared images, so as to achieve continuous and high-precision correction of infrared images in the whole temperature range, while reducing the amount of computation.

[0008] To address the aforementioned technical problems, embodiments of the present invention first provide a method for correcting dynamic non-uniformity in infrared images, comprising:

[0009] Acquire the original infrared image to be corrected;

[0010] Extract reference temperature feature values ​​from the original infrared image. These reference temperature feature values ​​are used to characterize the temperature change characteristics of a preset pixel column in the focal plane array when the infrared detector acquires the original infrared image.

[0011] The reference temperature feature value and the preset position encoding matrix are input into the trained solidification response fitting model for prediction processing to obtain the target background map of the original infrared image under the reference temperature feature value. The solidification response fitting model is obtained by iteratively training the constructed hybrid response fitting model based on the blackbody radiation image set collected by the infrared detector at multiple temperature nodes. The solidification response fitting model includes a static parameter setting layer, a dynamic parameter prediction layer, and a background map synthesis layer. The static parameter setting layer stores the trained solidification static tensor. The dynamic parameter prediction layer is constructed based on a preset multilayer perceptron network. The preset position encoding matrix is ​​generated based on the column dimension of the focal plane array in the infrared detector.

[0012] The original infrared image is corrected based on the target background map to obtain a corrected infrared image.

[0013] In one embodiment, extracting reference temperature feature values ​​from the original infrared image includes:

[0014] Extract the average value of the first pixel in the first preset reference column of the original infrared image, where the first preset reference column corresponds to the preset pixel column in the focal plane array of the infrared detector.

[0015] The mean value of the first pixel is normalized to obtain the reference temperature feature value.

[0016] In one embodiment, the reference temperature feature value and a preset position encoding matrix are input into a trained curing response fitting model for prediction processing to obtain the target background map of the original infrared image under the reference temperature feature value, including:

[0017] The reference temperature feature value and the preset position encoding matrix are input into the dynamic parameter prediction layer for prediction processing to obtain the dynamic correction coefficient under the reference temperature feature value.

[0018] The reference temperature characteristic value, the dynamic correction coefficient, and the solidified static tensor in the static parameter setting layer are input into the background base map synthesis layer for synthesis processing to obtain the target background base map.

[0019] In one embodiment, the dynamic parameter prediction layer includes a gain parameter prediction sublayer and a bias parameter prediction sublayer;

[0020] The reference temperature feature value and the preset position encoding matrix are input into the dynamic parameter prediction layer for prediction processing to obtain the dynamic correction coefficient under the reference temperature feature value, including:

[0021] The reference temperature feature value and the preset position encoding matrix are concatenated, and the concatenation result is input into the gain parameter prediction sub-layer for weighting to obtain the column dynamic gain correction coefficient.

[0022] The reference temperature feature value is input into the bias parameter prediction sub-layer for weighted processing to obtain the global dynamic bias correction coefficient.

[0023] In one embodiment, the solidified static tensor includes a solidified gain matrix, a solidified bias matrix, and a solidified pixel offset weight matrix;

[0024] The reference temperature characteristic value, the dynamic correction coefficient, and the solidified static tensor in the static parameter setting layer are input into the background base map synthesis layer for synthesis processing to obtain the target background base map, including:

[0025] In the background image synthesis layer, the reference temperature feature value, the dynamic correction coefficient, and the solidified static tensor are weighted using the following formula to obtain the target pixel value of the target background image:

[0026] ;

[0027] in, This represents the target pixel value; This represents the curing gain matrix; This represents the curing bias matrix; This represents the solidified pixel offset weight matrix; This refers to the column-direction dynamic bias correction coefficient in the dynamic correction coefficients; This refers to the global dynamic gain correction coefficient in the dynamic correction coefficients; This represents the reference temperature characteristic value.

[0028] In one embodiment, the training process of the solidified response fitting model includes the following steps:

[0029] Obtain the training reference temperature set and training location encoding matrix, and define a global static learnable tensor in the static parameter setting layer of the hybrid response fitting model;

[0030] The training reference temperature set and the training location encoding matrix are input into the dynamic parameter prediction layer of the hybrid response fitting model for processing to obtain the training dynamic correction parameters corresponding to each training reference temperature.

[0031] The training reference temperature set, the training dynamic correction parameters, and the static learnable tensor are input into the background map synthesis layer of the hybrid response fitting model for processing to obtain the training background map set.

[0032] The parameters of the hybrid response fitting model are optimized based on a preset loss function and a gradient backpropagation algorithm, and the above process is repeated until the preset loss function converges to obtain the solidified response fitting model.

[0033] In one embodiment, obtaining the training reference temperature set and the training location encoding matrix includes:

[0034] The blackbody radiation image set is acquired by an infrared detector at multiple temperature nodes, and the blackbody radiation image set corresponds one-to-one with the temperature node.

[0035] At each temperature node, the mean value of the second pixel in the second preset reference column of the corresponding blackbody radiation image is extracted, and the second pixel value is normalized to obtain the training reference temperature set for model training. The second preset reference column is the ineffective imaging column or blind cell column in the focal plane array of the infrared detector.

[0036] A training position encoding matrix for model training is generated based on the column dimensions of the focal plane array in the infrared detector.

[0037] In one embodiment, correcting the original infrared image based on the target background map to obtain a corrected infrared image includes:

[0038] Based on the target background map, a background noise fluctuation map corresponding to the target background map is determined;

[0039] The original infrared image is corrected based on the background noise fluctuation map to obtain the corrected infrared image.

[0040] In one embodiment, determining a background noise fluctuation map corresponding to the target background map based on the target background map includes:

[0041] Determine the average pixel value of the entire target background image;

[0042] The background noise fluctuation map is determined based on the target background base map and the average pixel value of the entire image.

[0043] Embodiments of the present invention also provide an infrared image dynamic non-uniformity correction device, comprising:

[0044] The image acquisition module is used to acquire the original infrared image to be corrected;

[0045] The feature extraction module is used to extract reference temperature feature values ​​from the original infrared image. The reference temperature feature values ​​are used to characterize the temperature change characteristics of a preset pixel column in the focal plane array when the infrared detector acquires the original infrared image.

[0046] The background image synthesis module is used to input the reference temperature feature value and the preset position encoding matrix into the trained solidification response fitting model for prediction processing, so as to obtain the target background image of the original infrared image under the reference temperature feature value; the solidification response fitting model is obtained by iteratively training the constructed hybrid response fitting model based on the blackbody radiation image set collected by the infrared detector at multiple temperature nodes; the solidification response fitting model includes a static parameter setting layer, a dynamic parameter prediction layer and a background image synthesis layer, the static parameter setting layer stores the trained solidification static tensor, the dynamic parameter prediction layer is constructed based on a preset multilayer perceptron network, and the preset position encoding matrix is ​​generated based on the column dimension of the focal plane array in the infrared detector.

[0047] An image correction module is used to correct the original infrared image based on the target background map to obtain a corrected infrared image.

[0048] Embodiments of the present invention also provide a computing device, comprising: a memory for storing one or more programs; and one or more processors for executing the one or more programs to implement the method described above.

[0049] The above-described solution of the present invention has at least the following beneficial effects:

[0050] 1. This invention establishes a hybrid response fitting model of "static physical benchmark + dynamic multilayer perceptron network compensation" and learns blackbody radiation images under continuous temperature nodes to obtain a solidified response fitting model that can fit the continuous nonlinear mapping relationship between temperature and background response. This enables continuous and high-precision non-uniformity correction across the entire temperature range, eliminating the interpolation error of the traditional two-point method at non-calibrated temperature points.

[0051] 2. In practical application scenarios, this invention can significantly save memory resources by replacing the lookup table of multi-point calibration with the dynamic correction coefficients generated by the trained solidified response fitting model. At the same time, during the processing, only the matrix operation of the fully connected layer of the multilayer perceptron network in the solidified response fitting model needs to be calculated, which can reduce the amount of computation, thus facilitating hardware deployment and making it suitable for FPGA porting.

[0052] 3. The present invention makes predictions based on a trained solidified response fitting model, which can decouple from the scene motion state and completely eliminate the "ghosting" problem of scene-based algorithms; the mean-preserving logic when correcting based on the target background map ensures that the DC component of the scene is completely preserved while the original infrared image is denoised, thus guaranteeing the absolute brightness of the scene.

[0053] 4. The present invention introduces position coding (PE) into the column distribution characterization of the focal plane array of the infrared detector, enabling the model to learn the non-uniformity of the focal plane array of the infrared detector in the column direction more efficiently and accurately, thereby effectively overcoming the low-frequency column stripes and "U-shaped" non-uniformity unique to infrared images.

[0054] It should be understood that the implementation of any embodiment of the present invention does not mean that it will simultaneously possess or achieve multiple or all of the above-mentioned beneficial effects. Attached Figure Description

[0055] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0056] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0057] Figure 1 This is a schematic flowchart of the infrared image dynamic non-uniformity correction method provided in an embodiment of the present invention;

[0058] Figure 2 This is a flowchart illustrating the specific implementation of a correction method provided in an optional embodiment of the present invention;

[0059] Figure 3 This is a flowchart illustrating the specific implementation of the dynamic parameter prediction layer and background map synthesis layer in the solidification response fitting model provided in an optional embodiment of the present invention.

[0060] Figure 4 This is a schematic diagram illustrating the specific training process of an optional implementation of the hybrid response fitting model of the present invention;

[0061] Figure 5 This is an original infrared image provided in an optional embodiment of the present invention;

[0062] Figure 6 This is a corrected infrared image provided in an optional embodiment of the present invention;

[0063] Figure 7This is a schematic block diagram of an infrared image dynamic non-uniformity correction device provided in an embodiment of the present invention;

[0064] Figure 8 This is a schematic block diagram of an electronic device provided in an embodiment of the present invention; and

[0065] Figure 9 This is a schematic block diagram of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0066] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0067] It should be understood that the terms "comprising / including," "consisting of," or any other variations are intended to cover non-exclusive inclusion, such that a product, apparatus, process, or method that comprises a list of elements includes not only those elements but may also include, where necessary, other elements not expressly listed, or elements inherent to such a product, apparatus, process, or method. Without further limitation, an element defined by the phrase "comprising / including," "consisting of," does not exclude the presence of additional identical elements in the product, apparatus, process, or method that includes said element.

[0068] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0069] like Figure 1 As shown, an embodiment of the present invention proposes a method for dynamic non-uniformity correction of infrared images, which may include the following steps:

[0070] Step 11: Obtain the original infrared image to be corrected;

[0071] Step 12: Extract the reference temperature feature value of the original infrared image. The reference temperature feature value is used to characterize the temperature change characteristics of the preset pixel column in the focal plane array when the infrared detector acquires the original infrared image.

[0072] Step 13: Input the reference temperature feature value and the preset position encoding matrix into the trained solidification response fitting model for prediction processing to obtain the target background map of the original infrared image under the reference temperature feature value; the solidification response fitting model is obtained by iteratively training the constructed hybrid response fitting model based on the blackbody radiation image set collected by the infrared detector at multiple temperature nodes; the solidification response fitting model includes a static parameter setting layer, a dynamic parameter prediction layer and a background map synthesis layer. The static parameter setting layer stores the trained solidification static tensor, the dynamic parameter prediction layer is constructed based on the preset multilayer perceptron network, and the preset position encoding matrix is ​​generated based on the column dimension of the focal plane array in the infrared detector;

[0073] Step 14: Correct the original infrared image based on the target background map to obtain the corrected infrared image.

[0074] In this embodiment, the preset pixel columns are non-effective imaging columns or blind pixel columns in the focal plane array of the infrared detector. These pixel columns can only sense the temperature changes of the detector itself during imaging and are not affected by the radiation of the external scene. Their output directly reflects the temperature changes of the detector itself when acquiring images. By extracting the reference temperature feature value of each frame of the original infrared image and inputting it into the trained solidified response fitting model for processing, the correction coefficient generated during the model processing can always be consistent with the current thermal state of the detector, thereby generating a target background map corresponding to the reference temperature feature value (at this time, the target background map represents the response image output by the detector according to its own characteristics under the reference temperature feature value). Subsequently, the original infrared image is corrected based on the target background map to remove the detector's own response from the original infrared image and retain only the real radiation information under the real scene, thereby realizing continuous dynamic correction of the original infrared image.

[0075] The preset position encoding matrix is ​​generated based on the column dimension of the focal plane array in the infrared detector. Specifically, it can be generated by cross-encoding the columns in the focal plane array using sine and cosine functions combined with a preset exponential decay. This matrix characterizes the inherent low-frequency and high-frequency spatial structure distribution characteristics of the detector's focal plane array in the column direction. The position encoding matrix is ​​required for both model training and target background image prediction to provide the model with spatial context information of the focal plane array, thereby ensuring the accuracy of model training and prediction. It should be noted that due to the manufacturing process of infrared detectors, the temperature change characteristics of any two detectors are not completely consistent. Therefore, during model training, the infrared detector used to acquire the training blackbody image should be the same detector used to acquire the original infrared image in actual application. Therefore, the preset position encoding matrix in this embodiment can also be generated based on the column dimension of the focal plane array in the infrared detector and directly stored in the dynamic parameter prediction layer of the model.

[0076] In one feasible example of the present invention, an exponential decay of 10000 can be used to ensure that the low-frequency and high-frequency spatial distribution characteristics in the column dimension can be fully extracted by the neural network. For the first column in the focal plane array... Column, the first in the preset position encoding matrix peacekeeping The elements corresponding to the dimensions can be represented as follows:

[0077] ;

[0078] ;

[0079] in, The column index represents the focal plane array, D represents the total dimension of the preset position encoding matrix, and n represents the dimension index (the value can be from 0 to 9). The position encoding matrix generated through the above process can accurately map the U-shaped non-uniform slope distribution of the focal plane array caused by physical reasons such as edge effects, thereby ensuring that the generated target background map can accurately reflect the temperature response characteristics of the infrared detector itself, so as to ensure the accuracy of subsequent correction of the original infrared image.

[0080] Here, the solidification response fitting model is trained based on a blackbody radiation image set at multiple temperature nodes and the training position encoding matrix of the infrared detector that acquired the blackbody radiation image. The multiple temperature nodes are continuously changing temperature nodes. During model training, the reference temperature of the blackbody radiation image at each temperature node is extracted and used as the training reference temperature. The training reference temperature and the training position encoding matrix are used as training data so that the model can learn the continuous nonlinear relationship between temperature and the detector's own temperature response, realize the prediction of the target background map in the entire temperature range, and thus achieve continuous and high-precision non-uniformity correction in the entire temperature range, solving the problems of limited calibration points and inaccurate interpolation in traditional methods.

[0081] When predicting the background map, the solidified static tensor in the static parameter setting layer defines the reference state of the detector at a reference temperature, which is the inherent non-uniformity of the detector at a standard temperature. When the reference temperature feature value is input into the model, the dynamic parameter prediction layer generates dynamic correction coefficients based on the feature value. These dynamic correction coefficients are used to describe the offset of the detector at the reference temperature feature value. Here, the solidified static tensor is provided by the static parameter setting layer, and the dynamic correction coefficients are provided by the dynamic parameter prediction layer to decouple the inherent non-uniformity of the detector from the dynamic offset. This improves processing efficiency on the one hand, and ensures the accuracy of the subsequent target background map generation on the other hand, thereby ensuring the accuracy of subsequent image correction.

[0082] Furthermore, in the background image synthesis layer, the solidified static tensor is adjusted based on dynamic correction coefficients to generate the target background image under the current reference temperature characteristic value. By combining static parameters (solidified static tensor) with dynamic correction coefficients, the physical process of image generation is realistically reflected, ensuring the accuracy of the generated target background image and thus guaranteeing the accuracy of subsequent image correction. In addition, since the solidified response fitting model is built based on a multilayer perceptron network, high-density full-image convolution calculations are unnecessary when predicting the target background image. Only a small number of fully connected layers of the multilayer perceptron network need to be calculated and weighted, reducing computational load and storage space, which is beneficial for hardware deployment and suitable for FPGA porting.

[0083] See Figure 2 and Figure 3 In an optional embodiment of the present invention, step 12 above may include:

[0084] Step 121: Extract the mean value of the first pixel in the first preset reference column of the original infrared image. The first preset reference column corresponds to the preset pixel column in the focal plane array of the infrared detector.

[0085] Step 122: Normalize the mean value of the first pixel to obtain the reference temperature feature value.

[0086] In this embodiment, the first preset reference column is a preset pixel column in the original infrared image, and the corresponding pixel value is generated by the preset cell column in the focal plane array of the infrared detector. The first pixel mean is obtained by extracting the pixel grayscale value of the preset pixel column and calculating the arithmetic mean. Since the preset cell column in the focal plane array does not respond to scene radiation, the change of the first pixel mean is entirely caused by the temperature change of the detector itself.

[0087] Furthermore, the mean value of the first pixel is normalized to convert the original grayscale value into a reference temperature feature value in the range of 0 to 1. This reference temperature feature value serves as the dynamic input of the model, driving the model to generate dynamic correction coefficients (which are consistent with the current thermal state of the detector). Based on the dynamic correction coefficients, a target background map corresponding to the reference temperature feature value is generated to achieve subsequent continuous dynamic correction.

[0088] Continue reading Figure 2 and Figure 3 In an optional embodiment of the present invention, step 13 above may include:

[0089] Step 131: Input the reference temperature feature value and the preset position encoding matrix into the dynamic parameter prediction layer for prediction processing to obtain the dynamic correction coefficient under the reference temperature feature value.

[0090] Here, the dynamic parameter prediction layer can include a gain parameter prediction sublayer and a bias parameter prediction sublayer, both of which are constructed using a multilayer perceptron network. The gain parameter prediction sublayer generates a column-directed dynamic gain correction coefficient based on a reference temperature feature value and a preset position encoding matrix. This gain correction coefficient represents the column-directed offset of the gain response relative to the reference state at the reference temperature feature value. The bias parameter prediction sublayer generates a global dynamic bias correction coefficient based on the reference temperature feature value. This global bias correction coefficient represents the overall translation amount of the bias of all pixels relative to the reference state at the reference temperature feature value.

[0091] Continue reading Figure 2 and Figure 3 Step 131 above may specifically include:

[0092] Step 1311: The reference temperature feature value and the preset position encoding matrix are concatenated, and the concatenation result is input into the gain parameter prediction sub-layer for weighted processing to obtain the column dynamic gain correction coefficient.

[0093] Step 1312: Input the reference temperature feature value into the bias parameter prediction sub-layer for weighted processing to obtain the global dynamic bias correction coefficient.

[0094] In this embodiment, when generating the column-oriented dynamic gain correction coefficients, since the reference temperature feature value is a scalar, it is first necessary to expand the reference temperature feature value in spatial dimension to form a temperature vector aligned with the dimension of the preset position encoding matrix. Each element in this temperature vector is a reference temperature feature value. Further, the temperature vector and the preset position encoding matrix are concatenated in the feature channel dimension to form an input tensor that simultaneously describes spatial position and temperature state. Further, this input tensor is input into the input gain parameter prediction sublayer, and linear transformations and nonlinear activation processes are sequentially performed on the input tensor based on the multiple hidden layers of the multilayer perceptron network to output the column-oriented dynamic gain correction coefficients corresponding to each column. Since the column noise is highly consistent in the vertical direction, column-level output gain correction coefficients are used here, which can reduce the number of parameters and thus reduce the computational load.

[0095] When generating the global dynamic bias correction coefficient, the reference temperature feature value is directly input into the bias parameter prediction sub-layer, and the reference temperature feature value is sequentially subjected to linear transformation and nonlinear activation processing based on the multiple hidden layers of the multilayer perceptron network to capture the overall response drift caused by temperature (this drift is independent of the column position of the detector) and output the global dynamic bias correction coefficient.

[0096] Continue reading Figure 2 and Figure 3In an optional embodiment of the present invention, step 13 above may include:

[0097] Step 132: Input the reference temperature feature value, dynamic correction coefficient, and solidified static tensor in the static parameter setting layer into the background base map synthesis layer for synthesis processing to obtain the target background base map.

[0098] Here, the solidified static tensor includes the solidified gain matrix, the solidified bias matrix, and the solidified pixel offset weight matrix; wherein, the solidified gain matrix is ​​used to characterize the intrinsic response slope of the pixel; the solidified bias matrix is ​​used to characterize the fundamental bias matrix of the intrinsic dark current background of the pixel; and the solidified pixel offset weight matrix is ​​used to characterize the residual drift characteristics of the pixel.

[0099] In the background image synthesis layer, the solidified gain matrix, solidified bias matrix, and solidified pixel offset weight matrix are adjusted based on the column-directed dynamic gain correction coefficient and the global dynamic bias correction coefficient to generate the target background image under the current reference temperature feature value. Through the mutual cooperation of the column-directed dynamic gain correction coefficient, the global dynamic bias correction coefficient, the solidified gain matrix, the solidified bias matrix, and the solidified pixel offset weight matrix, the physical process of image generation is realistically reflected, ensuring the accuracy of the generated target background image and thus ensuring the accuracy of subsequent image correction.

[0100] Continue reading Figure 2 and Figure 3 In an optional embodiment of the present invention, step 132 may specifically include:

[0101] Step 1321: In the background image composite layer, the reference temperature feature value, dynamic correction coefficient, and solidified static tensor are weighted using the following formula to obtain the target pixel value of the target background image:

[0102] ;

[0103] in, Indicates the target pixel value; Represents the solidification gain matrix; Represents the solidification bias matrix; This represents the fixed pixel offset weight matrix; This refers to the column-direction dynamic bias correction coefficient in the dynamic correction coefficients; This represents the global dynamic gain correction coefficient in the dynamic correction coefficients; This represents the reference temperature characteristic value.

[0104] See Figure 4 In an optional embodiment of the present invention, the training process of the solidified response fitting model includes the following steps:

[0105] Step 21: Obtain the training reference temperature set and training location encoding matrix, and define a global static learnable tensor in the static parameter setting layer of the hybrid response fitting model.

[0106] Step 211: Collect a set of blackbody radiation images at multiple temperature nodes based on an infrared detector. The blackbody radiation images in the set correspond one-to-one with the temperature nodes.

[0107] Step 212: At each temperature node, extract the mean value of the second pixel in the second preset reference column of the corresponding blackbody radiation image, and normalize the second pixel value to obtain the training reference temperature set for model training. The second preset reference column is the ineffective imaging column or blind cell column in the focal plane array of the infrared detector.

[0108] Step 213: Generate a training position encoding matrix for model training based on the column dimension of the focal plane array in the infrared detector;

[0109] Step 22: Input the training reference temperature set and the training position encoding matrix into the dynamic parameter prediction layer of the hybrid response fitting model for processing to obtain the training dynamic correction parameters corresponding to each training reference temperature; the training dynamic correction parameters correspond one-to-one with the training reference temperatures and form a training dynamic correction parameter set.

[0110] Step 23: Process the background map synthesis layer of the training reference temperature set, the training dynamic correction parameters and the static learnable tensor input hybrid response fitting model to obtain the training background map set.

[0111] Step 24: Optimize the parameters of the hybrid response fitting model based on the preset loss function and gradient backpropagation algorithm, and repeat the above process until the preset loss function converges to obtain the solidified response fitting model.

[0112] In this embodiment, the hybrid response fitting model has the same architecture as the trained solidified response fitting model, both including a static parameter setting layer, a dynamic parameter prediction layer, and a background image synthesis layer. The dynamic parameter prediction layer may include a gain parameter prediction sub-layer and a bias parameter prediction sub-layer. The specific determination process of the training reference temperature set and the training position encoding matrix is ​​the same as the specific determination process of the reference temperature feature value and the preset position encoding matrix mentioned above, and will not be repeated here. The static parameter setting layer will define a learnable gain matrix, a learnable bias matrix, and a learnable pixel offset weight matrix covering the global resolution.

[0113] During the training phase, the training reference temperature set is concatenated with the training position encoding matrix and then input into the gain parameter prediction sublayer, outputting the training gain correction coefficient at the training reference temperature; the training reference temperature set is input into the bias parameter prediction sublayer, outputting the training bias correction coefficient at the training reference temperature; further, the corresponding training reference temperature, training gain correction coefficient, training bias correction coefficient, learnable gain matrix, learnable bias matrix, and learnable pixel offset weight matrix are input into the background image synthesis layer for training prediction, and the training background image is output.

[0114] Repeat the above training process, using mean squared error (MSE) as the preset loss function. Calculate the training background image obtained from the training and its corresponding blackbody radiation image using the gradient backpropagation algorithm, and update the model parameters (parameters of the static parameter setting layer and weights of the dynamic parameter prediction layer) using the optimizer until the model converges. During training, the number of iterations can be set to 10, and the initial learning rate can be set to 1×10⁻⁶. -3 It also works with the AdamW optimizer to optimize parameters; the model is solidified during the model parameter export stage, and the stripe removal and shadow removal processing of a single frame can be completed in a very short time when the actual original infrared image is corrected.

[0115] In an optional embodiment of the present invention, step 14 above may include:

[0116] Step 141: Based on the target background map, determine the background noise fluctuation map corresponding to the target background map.

[0117] Since the target background map output by the solidification response fitting model contains information in two dimensions: DC component (global reference brightness, i.e., the overall average radiation response of the blackbody under the reference temperature characteristic value) and AC component (spatial fluctuations, i.e., spatial non-uniformity such as stripes, shadows, and dead pixels in the target background map); the background noise fluctuation map here is the AC component separated from the target background map; the specific separation process is as follows:

[0118] Step 1411: Determine the average pixel value of the entire target background image;

[0119] Step 1412: Determine the background noise fluctuation map based on the target background base map and the average pixel value of the entire image.

[0120] Specifically, the global pixel average can be determined based on the target pixel values ​​of the target background image. Correspondingly, the global pixel matrix is ​​the DC component of the target background image. This DC component is a single scalar value representing the basic average brightness value that the entire image should possess under reference temperature characteristics. Preferably, the global pixel average can be calculated using the following formula:

[0121] ;

[0122] in, The value represents the average pixel value of the entire image; i represents the row index of the target background image; j represents the column index of the target background image; H represents the height of the target background image; W represents the width of the target background image. This represents the target pixel value at the i-th row and j-th column position in the target background image.

[0123] Furthermore, by subtracting the mean pixel value of the entire image from the target pixel value at each location in the target background image, the resulting image composed of the pixel differences at each location is the background noise fluctuation map that separates the DC environmental radiation.

[0124] In an optional embodiment of the present invention, step 14 above may include:

[0125] Step 142: Correct the original infrared image based on the background noise fluctuation map to obtain the corrected infrared image.

[0126] In this embodiment, the original infrared image is corrected using a background noise map, which means subtracting the background noise fluctuation map from the original infrared image (specifically, subtracting the pixel value at each position in the original infrared image from the pixel difference at the corresponding position in the background noise fluctuation map). This yields and outputs a corrected infrared image that retains the absolute average brightness of the original scene. Preferably, the specific correction process for the original infrared image can be expressed by the following formula:

[0127] ;

[0128] in, Represents the pixel values ​​of the corrected infrared image; This represents the pixel value of the original infrared image. The mean-preserving logic in the above correction formula ensures that the DC component (average ambient radiation information) of the scene is fully preserved while the image is denoised. This allows the corrected infrared image to avoid ghosting problems caused by scene-based algorithms and retain the absolute brightness of the scene.

[0129] Here, taking an infrared detector with a resolution of 512×640 as an example, we set the original infrared image it acquires (such as...) Figure 5 The height of the (shown) is H=512, and the effective width is W. active =640, and take the 644th column as the first preset reference column to extract its corresponding reference temperature feature value. The corresponding preset position encoding matrix adopts sine and cosine cross encoding, with a dimension set to 10. The total dimension of the stitched image is 20. The original infrared image is then corrected using the method provided in the above embodiment to obtain the following... Figure 6The corrected infrared image shown is from Figure 5 As can be seen, there are many obvious vertical stripes in the image before correction. After correction using the method described above, Figure 6 The vertical stripes have been significantly removed.

[0130] The correction method provided by the above embodiments of the present invention extracts the pixel mean of the reference column as the reference temperature value, and combines it with column position encoding. It uses a lightweight multilayer perceptron to dynamically predict the nonlinear compensation coefficients of gain and bias, and then reconstructs the background map at the current reference temperature for differential correction. On the one hand, it can reduce the amount of computation and storage, and on the other hand, it can achieve continuous and high-precision non-uniformity correction over the entire temperature range. It solves the problems of large storage occupation, easy ghosting and excessive computation in the prior art.

[0131] like Figure 7 As shown, an embodiment of the present invention also provides an infrared image dynamic non-uniformity correction device 30, comprising:

[0132] Image acquisition module 31 is used to acquire the original infrared image to be corrected;

[0133] The feature extraction module 32 is used to extract reference temperature feature values ​​from the original infrared image. The reference temperature feature values ​​are used to characterize the temperature change characteristics of the preset pixel column in the focal plane array when the infrared detector acquires the original infrared image.

[0134] The background image synthesis module 33 is used to input the reference temperature feature value and the preset position encoding matrix into the trained solidification response fitting model for prediction processing, so as to obtain the target background image of the original infrared image under the reference temperature feature value. The solidification response fitting model is obtained by iteratively training the constructed hybrid response fitting model based on the blackbody radiation image set collected by the infrared detector at multiple temperature nodes. The solidification response fitting model includes a static parameter setting layer, a dynamic parameter prediction layer and a background image synthesis layer. The static parameter setting layer stores the trained solidification static tensor. The dynamic parameter prediction layer is constructed based on the preset multilayer perceptron network. The preset position encoding matrix is ​​generated based on the column dimension of the focal plane array in the infrared detector.

[0135] The image correction module 34 is used to correct the original infrared image based on the target background map to obtain the corrected infrared image.

[0136] It should be noted that this device is a device corresponding to the above-mentioned infrared image dynamic non-uniformity correction method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0137] like Figure 8As shown, embodiments of the present invention also provide an electronic device 50, including: a memory 51 for storing one or more computer programs; and one or more processors 52 for executing the one or more computer programs, wherein the computer programs, when run by the processors, perform the methods described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects. The electronic device 50 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown in this invention, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0138] like Figure 9 As shown, the electronic device 50 is a computing device or computer system, which may include a computing unit 501 that can perform various appropriate actions and processes according to a computer program stored in ROM 502 (read-only memory) or a computer program loaded from storage unit 508 into random access RAM 503 (memory). RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. I / O interface 505 (input / output interface) is also connected to bus 504.

[0139] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0140] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) processing units (NPCs), various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the infrared image dynamic non-uniformity correction method can be implemented as a computer software program tangibly contained in a computer-readable storage medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the infrared image dynamic non-uniformity correction method described above can be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform an infrared image dynamic non-uniformity correction method by any other suitable means (e.g., by means of firmware).

[0141] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0142] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0143] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0144] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0146] Furthermore, it should be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of the present invention. Moreover, the steps performing the above series of processes can naturally be executed in the order described, but are not necessarily required to be executed in chronological order; some steps can be executed in parallel or independently of each other. Those skilled in the art will understand that all or any step or component of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in hardware, firmware, software, or a combination thereof. This is something that those skilled in the art can achieve by using their basic programming skills after reading the description of the present invention.

[0147] Therefore, the object of the present invention can also be achieved by running a program or a set of programs on any computing device. The computing device can be a known general-purpose device. Therefore, the object of the present invention can also be achieved simply by providing a program product containing program code for implementing the method or apparatus. That is, such a program product also constitutes the present invention, and the storage medium storing such a program product also constitutes the present invention. Obviously, the storage medium can be any known storage medium or any storage medium developed in the future. It should also be noted that in the apparatus and method of the present invention, it is obvious that the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent to the present invention. Furthermore, the steps for performing the above series of processes can naturally be performed in the order described, but are not necessarily required to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0148] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for correcting dynamic non-uniformity in infrared images, characterized in that, include: Acquire the original infrared image to be corrected; Extract reference temperature feature values ​​from the original infrared image. These reference temperature feature values ​​are used to characterize the temperature change characteristics of a preset pixel column in the focal plane array when the infrared detector acquires the original infrared image. The reference temperature feature value and the preset position encoding matrix are input into the trained solidification response fitting model for prediction processing to obtain the target background map of the original infrared image under the reference temperature feature value. The curing response fitting model is obtained by iteratively training the constructed hybrid response fitting model based on the blackbody radiation image set collected by the infrared detector at multiple temperature nodes. The solidified response fitting model includes a static parameter setting layer, a dynamic parameter prediction layer, and a background map synthesis layer. The static parameter setting layer stores a trained solidified static tensor. The dynamic parameter prediction layer is constructed based on a preset multilayer perceptron network. The preset position encoding matrix is ​​generated based on the column dimension of the focal plane array in the infrared detector. The original infrared image is corrected based on the target background map to obtain a corrected infrared image.

2. The infrared image dynamic non-uniformity correction method according to claim 1, characterized in that, Extracting reference temperature feature values ​​from the original infrared image includes: Extract the average value of the first pixel in the first preset reference column of the original infrared image, where the first preset reference column corresponds to the preset pixel column in the focal plane array of the infrared detector. The mean value of the first pixel is normalized to obtain the reference temperature feature value.

3. The infrared image dynamic non-uniformity correction method according to claim 1, characterized in that, The reference temperature feature value and the preset position encoding matrix are input into a trained curing response fitting model for prediction processing to obtain the target background map of the original infrared image under the reference temperature feature value, including: The reference temperature feature value and the preset position encoding matrix are input into the dynamic parameter prediction layer for prediction processing to obtain the dynamic correction coefficient under the reference temperature feature value. The reference temperature characteristic value, the dynamic correction coefficient, and the solidified static tensor in the static parameter setting layer are input into the background base map synthesis layer for synthesis processing to obtain the target background base map.

4. The infrared image dynamic non-uniformity correction method according to claim 3, characterized in that, The dynamic parameter prediction layer includes a gain parameter prediction sublayer and a bias parameter prediction sublayer. The reference temperature feature value and the preset position encoding matrix are input into the dynamic parameter prediction layer for prediction processing to obtain the dynamic correction coefficient under the reference temperature feature value, including: The reference temperature feature value and the preset position encoding matrix are concatenated, and the concatenation result is input into the gain parameter prediction sub-layer for weighting to obtain the column dynamic gain correction coefficient. The reference temperature feature value is input into the bias parameter prediction sub-layer for weighted processing to obtain the global dynamic bias correction coefficient.

5. The infrared image dynamic non-uniformity correction method according to claim 3, characterized in that, The solidified static tensor includes a solidified gain matrix, a solidified bias matrix, and a solidified pixel offset weight matrix; The reference temperature characteristic value, the dynamic correction coefficient, and the solidified static tensor in the static parameter setting layer are input into the background base map synthesis layer for synthesis processing to obtain the target background base map, including: In the background image synthesis layer, the reference temperature feature value, the dynamic correction coefficient, and the solidified static tensor are weighted using the following formula to obtain the target pixel value of the target background image: ； in, This represents the target pixel value; This represents the curing gain matrix; This represents the curing bias matrix; This represents the solidified pixel offset weight matrix; This refers to the column-direction dynamic bias correction coefficient in the dynamic correction coefficients; This refers to the global dynamic gain correction coefficient in the dynamic correction coefficients; This represents the reference temperature characteristic value.

6. The infrared image dynamic non-uniformity correction method according to claim 1, characterized in that, The training process of the solidification response fitting model includes the following steps: Obtain the training reference temperature set and training location encoding matrix, and define a global static learnable tensor in the static parameter setting layer of the hybrid response fitting model; The training reference temperature set and the training location encoding matrix are input into the dynamic parameter prediction layer of the hybrid response fitting model for processing to obtain the training dynamic correction parameters corresponding to each training reference temperature. The training reference temperature set, the training dynamic correction parameters, and the static learnable tensor are input into the background map synthesis layer of the hybrid response fitting model for processing to obtain the training background map set. The parameters of the hybrid response fitting model are optimized based on a preset loss function and a gradient backpropagation algorithm, and the above process is repeated until the preset loss function converges to obtain the solidified response fitting model.

7. The infrared image dynamic non-uniformity correction method according to claim 6, characterized in that, Obtain the training reference temperature set and the training location encoding matrix, including: Based on the blackbody radiation image set acquired by the infrared detector at multiple temperature nodes, the blackbody radiation image set corresponds one-to-one with the temperature node. At each temperature node, the mean value of the second pixel in the second preset reference column of the corresponding blackbody radiation image is extracted, and the second pixel value is normalized to obtain the training reference temperature set for model training. The second preset reference column is the ineffective imaging column or blind cell column in the focal plane array of the infrared detector. A training position encoding matrix for model training is generated based on the column dimensions of the focal plane array in the infrared detector.

8. The infrared image dynamic non-uniformity correction method according to claim 1, characterized in that, The original infrared image is corrected based on the target background map to obtain a corrected infrared image, including: Based on the target background map, a background noise fluctuation map corresponding to the target background map is determined; The original infrared image is corrected based on the background noise fluctuation map to obtain the corrected infrared image.

9. The infrared image dynamic non-uniformity correction method according to claim 8, characterized in that, Based on the target background base map, determine the background noise fluctuation map corresponding to the target background base map, including: Determine the average pixel value of the entire target background image; The background noise fluctuation map is determined based on the target background base map and the average pixel value of the entire image.

10. A device for correcting dynamic non-uniformity of infrared images, characterized in that, include: The image acquisition module is used to acquire the original infrared image to be corrected; The feature extraction module is used to extract reference temperature feature values ​​from the original infrared image. The reference temperature feature values ​​are used to characterize the temperature change characteristics of a preset pixel column in the focal plane array when the infrared detector acquires the original infrared image. The base map synthesis module is used to input the reference temperature feature value and the preset position encoding matrix into the trained solidification response fitting model for prediction processing, so as to obtain the target background base map of the original infrared image under the reference temperature feature value. The curing response fitting model is obtained by iteratively training the constructed hybrid response fitting model based on the blackbody radiation image set collected by the infrared detector at multiple temperature nodes. The solidified response fitting model includes a static parameter setting layer, a dynamic parameter prediction layer, and a background map synthesis layer. The static parameter setting layer stores a trained solidified static tensor. The dynamic parameter prediction layer is constructed based on a preset multilayer perceptron network. The preset position encoding matrix is ​​generated based on the column dimension of the focal plane array in the infrared detector. An image correction module is used to correct the original infrared image based on the target background map to obtain a corrected infrared image.

Images