A Wide-Temperature and Large-Dynamic Infrared Image Correction Method Based on Blackbody Data Enhancement

Abstract

The present invention discloses a wide-temperature large-dynamic infrared image correction method based on blackbody data enhancement, which includes: for each of the H temperatures, N blackbody images at this temperature are collected by the infrared detector to be corrected, and the blackbody images obtained from the H temperatures constitute blackbody data; calculating the effective blackbody image and the paired image at each temperature as an image group; determining the sliding window adopted in the current cycle based on the maximum scale to be enhanced; performing the local region map acquisition step until the preset number of times is reached to obtain a fusion-enhanced image, and separating it to obtain the enhanced dirty image and the target image to form the enhanced image pair of the current cycle; after the number of cycles reaches G times, all the enhanced image pairs obtained are used to form an enhanced data set, and a preset deep neural network is trained to obtain a trained infrared image correction model for correcting the infrared image to be corrected. The present invention can effectively correct infrared images in a wide temperature range and a large dynamic range.

Description

Technical Field

[0001] The present invention belongs to the technical field of infrared image processing, and particularly relates to a wide-temperature large-dynamic infrared image correction method based on blackbody data enhancement. Background Art

[0002] In an infrared imaging system, even when receiving the same intensity of infrared radiation, the detection units on the focal plane array of an infrared detector still generate inconsistent output responses. This non-uniformity caused by the response of detection pixels will degrade the infrared image, affecting the visual effect of the infrared image and the accuracy of subsequent detection tasks. Therefore, it is necessary to perform non-uniformity correction on the original data collected by the infrared detector.

[0003] In practical applications, the most commonly used correction method is the temperature calibration-based method. This method aligns the focal plane array of the infrared detector with a blackbody with a uniform temperature distribution, collects the response values at one or more temperatures to calculate the gain and bias coefficients of each pixel on the focal plane array, and then compensates the original infrared image with the calculated parameters to complete the non-uniformity correction. However, since the response curve of the infrared detector is completely non-linear, it is difficult for the one-point, two-point, and multi-point correction methods to accurately fit the real correction curve; in addition, traditional calibration algorithms calculate more accurate correction coefficients by collecting data in a narrow temperature range. In this mode, when there are objects with a wide-temperature large-dynamic distribution in the scene, there will be non-uniformity noise residues in the corrected infrared image, affecting the actual use. Summary of the Invention

[0004] To solve the above problems existing in the prior art, embodiments of the present invention provide a wide-temperature large-dynamic infrared image correction method, device, electronic device, and storage medium based on blackbody data enhancement. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0005] In a first aspect, embodiments of the present invention provide a wide-temperature large-dynamic infrared image correction method based on blackbody data enhancement, the method comprising:

[0006] Step 1, for each of the preset H temperatures, use the infrared detector to be corrected to collect N blackbody images at this temperature, and the blackbody images obtained from the H temperatures constitute blackbody data; wherein, the H temperatures change continuously, H is at least 10, and the temperature step includes 5°C or 10°C;

[0007] Step 2, calculate the effective blackbody image and the corresponding paired image at each temperature in the blackbody data as an image group at the corresponding temperature;

[0008] Step 3: Determine the sliding window used in the current loop based on the maximum scale to be enhanced; perform the local region map acquisition step, including obtaining a fused image from the random image group and processing it using the sliding window to obtain a local region map of the fused enhanced image; repeat the local region map acquisition step and splice the obtained local region maps until the local region map acquisition step is executed a preset number of times to obtain a fused enhanced image, and separate it to obtain the enhanced dirty image and the target image, forming the enhanced image pair for the current loop.

[0009] Step 4: Determine whether the number of loops has reached G times; if not, execute Step 3, if so, execute Step 5.

[0010] Step 5: Form an enhanced data set from all the obtained enhanced image pairs; use the enhanced data set to train a preset deep neural network in a supervised manner to obtain a trained infrared image correction model for outputting a non-uniformity corrected image for the input infrared image to be corrected.

[0011] In an embodiment of the present invention, in Step 2, calculating the effective blackbody image and the corresponding paired image at each temperature in the blackbody data includes:

[0012] For each temperature, obtain the effective blackbody image at that temperature by performing pixel averaging on all the blackbody images at that temperature.

[0013] For each temperature, use the effective blackbody image at that temperature to determine the uniform response image of the focal plane array for that temperature, as the paired image corresponding to the effective blackbody image at that temperature.

[0014] In an embodiment of the present invention, for each temperature, the calculation formula used to obtain the effective blackbody image at that temperature by performing pixel averaging on all the blackbody images at that temperature includes:

[0015]

[0016] Wherein, is the effective blackbody image at temperature T g ; is the k-th blackbody image at temperature T g ; (i, j) is the position of the pixel in the image; g ∈ [1, H].

[0017] In an embodiment of the present invention, for each temperature, the calculation formula used to determine the uniform response image of the focal plane array for that temperature using the effective blackbody image at that temperature as the paired image corresponding to the effective blackbody image at that temperature includes:

[0018]

[0019] in, is the temperature T g The paired image corresponding to the effective blackbody image below; m×n is the total number of infrared focal plane array pixels of the infrared detector to be corrected, which is also the number of pixels of each blackbody image obtained.

[0020] In one embodiment of the present invention, in step 3, determining the sliding window used in the current cycle based on the maximum scale to be enhanced includes:

[0021] Generate a random number s according to the preset maximum scale l to be enhanced, where s = random (1, l);

[0022] Define a sliding window W with size and step size s s The sliding window used in the current loop.

[0023] In one embodiment of the present invention, in step 3, a local area map acquisition step is performed, including obtaining a fused image based on a random image group and using the sliding window to process to obtain a local area map of the fused enhanced image; repeating the local area map acquisition step and splicing the obtained local area maps until the local area map acquisition step is performed a preset number of times to obtain a fused enhanced image, and separating the fused enhanced image to obtain an enhanced dirty image and a target image, forming an enhanced image pair of the current cycle, including:

[0024] Step a1, generate a random number r;

[0025] r = random(1,H)

[0026] Step a2: Select a group of images with a temperature of random number r from all the image groups corresponding to H temperatures. and And splice the two along the image channel dimension to get the corresponding fused image

[0027]

[0028] Among them, Cat represents splicing, dim=1 represents the first dimension, and the first dimension is the image channel dimension;

[0029] Step a3, using the sliding window W used in the current cycle s For fused images Draw a fusion feature once to obtain a local area map of the fusion enhanced image of the current cycle;

[0030] Step a4: determine whether the local area map acquisition step has been executed for a preset number of times Otherwise, step a1 is executed again; and starting from the second execution of the local region map acquisition step, after each execution, the latest obtained local region map is spliced with the local region map obtained in the previous time; if so, the currently spliced local region map is used as the fused and enhanced image of the current cycle, and step 5 is executed;

[0031] Step a5: Separate the fused and enhanced image along the image channel dimension to obtain the enhanced dirty image and the target image, forming the enhanced image pair of the current cycle.

[0032] In an embodiment of the present invention, in step 5, using the enhanced data set to train a preset deep neural network in a supervised manner to obtain a trained infrared image correction model, including:

[0033] Taking the dirty images in the enhanced data set as the input data of the preset deep neural network, taking the corresponding target images as label data, using the Adam optimizer to optimize the preset loss function for iterative training until the convergence condition is reached to obtain a weight file, and configuring the preset deep neural network with the weight file to obtain a trained infrared image correction model; wherein, the preset deep neural network includes a U-Net neural network.

[0034] In a second aspect, an embodiment of the present invention provides a wide-temperature large-dynamic infrared image correction device based on blackbody data enhancement, and the device includes:

[0035] A blackbody data acquisition module, configured to, for each of the preset H temperatures, use an infrared detector to be corrected to collect N blackbody images at this temperature, and the blackbody images obtained from the H temperatures constitute blackbody data; wherein, the H temperatures change continuously, H is at least 10, and the temperature step includes 5°C or 10°C;

[0036] Each temperature image group acquisition module is configured to calculate the effective blackbody image and the corresponding paired image at each temperature in the blackbody data as an image group at the corresponding temperature;

[0037] An enhanced image pair acquisition module, configured to determine the sliding window adopted in the current cycle based on the maximum scale to be enhanced; execute the local region map acquisition step, including obtaining a fused image based on a random image group and using the sliding window for processing to obtain a local region map of the fused and enhanced image; repeat the local region map acquisition step and splice the obtained local region maps until the local region map acquisition step is executed a preset number of times to obtain a fused and enhanced image, and separating it to obtain the enhanced dirty image and the target image, forming the enhanced image pair of the current cycle;

[0038] A judgment module, configured to judge whether the number of loops reaches G times; if not, execute the processing procedure of the enhanced image pair acquisition module, and if so, execute the processing procedure of the network training and calibration module;

[0039] A network training and calibration module, configured to form an enhanced data set from all the obtained enhanced image pairs; use the enhanced data set to train a preset deep neural network in a supervised manner to obtain a trained infrared image calibration model, which is used to output a non-uniformity corrected image for the input infrared image to be corrected.

[0040] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0041] The memory is used to store a computer program;

[0042] When the processor is configured to execute the program stored on the memory, it implements the steps of the method for calibrating a wide-temperature large-dynamic infrared image based on blackbody data enhancement provided by the embodiment of the present invention.

[0043] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the method for calibrating a wide-temperature large-dynamic infrared image based on blackbody data enhancement provided by the embodiment of the present invention.

[0044] Advantages of the present invention:

[0045] Aiming at the problem that the existing temperature calibration method has poor calibration effect in a wide temperature range, the embodiment of the present invention combines the temperature calibration method with the deep learning method, and provides a method for calibrating a wide-temperature large-dynamic infrared image based on blackbody data enhancement. By randomly sampling wide-temperature blackbody data with a multi-scale sliding window, the spatial dependence of different response pixels on the focal plane array can be broken, which enables this solution to randomly replace the response temperature of pixels at the same position on the infrared detector focal plane array and implement blackbody data enhancement. The enhanced data can force the blackbody data with a large-dynamic wide-temperature distribution to be aggregated into an infrared image, establish an association between the wide-temperature large-dynamic response of the detection pixels and their spatial response, help the deep learning algorithm extract the pixel response features of the wide-temperature large-dynamic, provide data support for the deep learning algorithm to learn a continuous and accurate calibration curve, and achieve effective non-uniformity correction of infrared images. Description of the Drawings

[0046] Figure 1Schematic flowchart of a wide-temperature large-dynamic infrared image correction method based on blackbody data augmentation provided by an embodiment of the present invention;

[0047] Figure 2 Schematic flowchart of step 3 of an embodiment of the present invention;

[0048] Figures 3(a) to 3(c) Data augmentation results of the method according to an embodiment of the present invention, Figures 3(a) to 3(c) respectively representing the output results when the sliding step size s is 2, 4, and 8;

[0049] Figs. 4(a) and 4(b) show the correction results of the infrared images of two scenes using the trained U-Net network and the effect of two-point correction when the data is augmented to 50,000 pairs by the method according to an embodiment of the present invention;

[0050] Figure 5 (a) and Figure 5 (b) are the column response mean curves corresponding to the two scenes in Figs. 4(a) and 4(b);

[0051] Figure 6 Schematic structural diagram of a wide-temperature large-dynamic infrared image correction device based on blackbody data augmentation provided by an embodiment of the present invention;

[0052] Figure 7 Schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0053] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0054] It should be noted that the execution subject of a wide-temperature large-dynamic infrared image correction method based on blackbody data augmentation provided by an embodiment of the present invention can be a wide-temperature large-dynamic infrared image correction device, and the device can run in an electronic device. Among them, the electronic device can be a server or a terminal device, and of course, it is not limited thereto.

[0055] In a first aspect, an embodiment of the present invention provides a wide-temperature large-dynamic infrared image correction method based on blackbody data augmentation, as Figure 1 shown, the method may include the following steps:

[0056] Step 1, for each of the preset H temperatures, use the infrared detector to be calibrated to collect N blackbody images at this temperature. The blackbody images obtained from the H temperatures constitute blackbody data;

[0057] Traditional one-point, two-point, and multi-point calibration methods based on temperature mostly use local temperature changes to finely fit the calibration curve of the focal plane array. This results in the above calibration coefficients having the phenomenon of temperature threshold overflow in a wide temperature range and large dynamic range. For example, the calibration coefficients calculated from the temperature range of 20°C to 25°C cannot effectively calibrate an object at 50°C in the input image, which will cause local blurring of the image and residual non-uniformity noise.

[0058] In the embodiments of the present invention, the H temperatures change continuously. And to ensure the infrared image calibration performance, H is at least 10, and the temperature step size includes 5°C or 10°C. Compared with the prior art, the temperature range set in the embodiments of the present invention is a relatively wide range;

[0059] Traditional two-point or multi-point calibration both use two adjacent temperatures with a very narrow interval to estimate the response of the infrared detector within this temperature range. In this method, the narrower the temperature range, the higher the calibration accuracy. While the method in the embodiments of the present invention combines the learning ability of deep learning and uses the proposed data enhancement method to fit an accurate wide-temperature response curve.

[0060] Taking the temperature T1 of 20°C as an example, the process of collecting N blackbody images at this temperature includes:

[0061] First, adjust the blackbody to the temperature T1, and aim the infrared detector to be calibrated at the blackbody to take N blackbody images; since the infrared focal plane array of the infrared detector to be calibrated consists of m×n pixels, the pixels of each collected blackbody image are m×n.

[0062] Next, raise the blackbody temperature to the temperature T2 of 25°C, and repeat the above image collection process to obtain N blackbody images at the temperature T2. And so on, until the blackbody images at the H temperatures are collected, jointly constituting the blackbody data. It can be understood that there are H×N blackbody images in the blackbody data. Wherein, H, N, m, and n are all integers greater than 0.

[0063] Step 2, calculate the effective blackbody image and the corresponding paired image at each temperature in the blackbody data as an image group at the corresponding temperature;

[0064] Among them, calculating the effective blackbody image and the corresponding paired image at each temperature in the blackbody data includes:

[0065] 1) For each temperature, obtain the effective blackbody image at this temperature by pixel-averaging all the blackbody images at this temperature;

[0066] The calculation formulas used in this step include:

[0067]

[0068] Among them, is the effective blackbody image at temperature T g under; is the k-th blackbody image at temperature T g under; (i, j) is the position of the pixel in the image; g ∈ [1, H]. It can be understood that the size of the effective blackbody image is still m × n. Through the above formula (1), H effective blackbody images at H temperatures {T1, T2,..., T H} can be calculated

[0069] 2) For each temperature, use the effective blackbody image at this temperature to determine the uniform response image that the focal plane array has for this temperature, as the paired image corresponding to the effective blackbody image at this temperature.

[0070] The paired image at one temperature is a uniform image with invariant gray scale and size of m × n, used to reflect the uniform response that the focal plane array should have for this temperature.

[0071] The calculation formulas used in this step include:

[0072]

[0073] Among them, is the paired image corresponding to the effective blackbody image at temperature T g under; m × n is the total number of pixels of the infrared focal plane array of the infrared detector to be corrected, and also the number of pixels of each obtained blackbody image.

[0074] Through the above processing, H effective blackbody images for blackbody data in a wide temperature range can be obtained and the corresponding paired images

[0075] Step 3, determine the sliding window used in the current iteration based on the maximum scale to be enhanced; perform the local region map acquisition step, including obtaining a fused image based on a random image group and using the sliding window for processing to obtain a local region map of the fused enhanced image; repeat the local region map acquisition step and splice the obtained local region maps until the local region map acquisition step is executed a preset number of times to obtain a fused enhanced image, and separate it to obtain the enhanced dirty image and the target image, forming the enhanced image pair in the current iteration;

[0076] Among them, one execution of step 3 corresponds to one cycle, and an enhanced image pair for the current cycle can be obtained to achieve the purpose of data augmentation. For the process of one cycle corresponding to step 3, please refer to Figure 2 Understand, Figure 2 Take the first execution of step 3 as an example.

[0077] Before data augmentation in step 3, it is necessary to first determine the sliding window adopted in the current cycle. Specifically, in step 3, based on the maximum scale to be enhanced, the sliding window adopted in the current cycle is determined, including:

[0078] (1) According to the preset maximum scale l to be enhanced, generate a random number s, where,

[0079] s = random(1, l) (3)

[0080] (2) Define a sliding window W with both size and step length being s s as the sliding window adopted in the current cycle.

[0081] It can be seen that in the embodiment of the present invention, the sliding window adopted for each cycle is a randomly generated sliding window W s of the sampling scale s to obtain multi-scale enhanced data. Among them, the maximum scale l to be enhanced can be set according to needs. The larger the l is set, the faster the image enhancement speed of the embodiment of the present invention, the larger the local receptive field, but the mapping of detailed information is not fine; the smaller the l is set, the slower the image enhancement speed, but the mapping of the focal plane array pixels is more accurate.

[0082] In an optional embodiment, in step 3, the local region map acquisition step is executed, including obtaining a fused image based on a random image group and using the sliding window to process to obtain a local region map of the fused enhanced image; repeating the local region map acquisition step and splicing the obtained local region maps until the local region map acquisition step is executed a preset number of times to obtain a fused enhanced image, and separating it to obtain an enhanced dirty image and a target image, constituting the enhanced image pair for the current cycle, including:

[0083] Step a1, generate a random number r;

[0084] r = random(1, H) (4)

[0085] Among them, both the above random numbers s and r are integers. For the convenience of distinction, when step 3 has different cycle times, r can be represented by r1, r2, etc. in turn.

[0086] Step a2, select a group of image groups with temperature being the random number r from all image groups corresponding to H temperatures and And splice the two along the image channel dimension to obtain the corresponding fused image

[0087]

[0088] Among them, all the image groups corresponding to the H temperatures include H valid blackbody images and the corresponding paired images The valid blackbody image and the paired image corresponding to the position form a group of images, and select from them and Cat represents splicing, dim = 1 represents the first dimension, and the first dimension is the image channel dimension;

[0089] Step a3, use the sliding window W adopted in the current loop s to extract the fused features from the fused image once to obtain the local region map of the fused enhanced image in the current loop;

[0090] In the embodiment of the present invention, each execution of step 3 corresponds to one loop. In one loop, the local region map acquisition step will be executed a preset number of times. For the sake of distinction, each execution of the local region map acquisition step in the embodiment of the present invention is not called a loop. After each execution of the local region map acquisition step, a local region map will be obtained. The local region maps obtained by the preset number of times in one loop are continuously spliced and filled to complete the fused enhanced image corresponding to this loop.

[0091] Since the scale of the sliding window in each loop is random, then in multiple loops, by randomly sampling the wide-temperature blackbody data through the multi-scale sliding window, the spatial dependence of different response pixels on the focal plane array can be broken, which enables this solution to randomly replace the response temperatures of the pixels at the same position on the infrared detector focal plane array and implement blackbody data enhancement.

[0092] Among them, taking the first loop as an example, the obtained fused enhanced image can be represented by A1, and the local region map after the first execution of the local region map acquisition step can be represented as A1(s, s); where:

[0093]

[0094] So far, the first local region map acquisition step in the first loop corresponding to step a3 is completed.

[0095] Step a4, judge whether the execution of the local region map acquisition step has reached the preset number of times If not, execute step a1 again; and starting from the second execution of the local region map acquisition step, splice the latest obtained local region map with the previously obtained local region map after each execution; if so, use the currently spliced local region map as the fusion-enhanced image of the current loop, and execute step 5;

[0096] Taking the second execution of the local region map acquisition step as an example, use the formula (4) to regenerate the random number r2, and use the formula (5) to extract the effective blackbody images from all image groups corresponding to H temperatures again and paired images and splice them to obtain a new fused image Then, use the sliding window W according to formula (6) s for the fused image to perform a second sliding to fill, and obtain the local region map A1(s, 2s) of the fusion-enhanced image A1. And so on, until the local region map acquisition step is executed a preset number of times After that, the fusion-enhanced image A1 of the first loop is obtained, and its size is m×n.

[0097] Step a5, separate the fusion-enhanced image along the image channel dimension to obtain the enhanced dirty image and the target image, and form the enhanced image pair of the current loop.

[0098] The above process can be expressed by the formula as:

[0099] A N ,A C =Split dim＝1 (A) (7)

[0100] where Split represents the separation operation; A represents the fusion-enhanced image obtained in the current loop. For example, for the first loop, A is A1, and the enhanced dirty image A N is the target image A C is

[0101] Step 4, determine whether the number of loops reaches G times; if not, execute step 3, if so, execute step 5;

[0102] In the embodiments of the present invention, G is an integer greater than 0, representing the number of enhanced image pairs, which can be set according to the required number, such as 5000, etc.

[0103] Step 5: Compose the enhanced dataset from all the obtained enhanced image pairs; use the enhanced dataset to train a preset deep neural network in a supervised manner to obtain a trained infrared image correction model, which is used to output a non-uniformity corrected image for the input infrared image to be corrected.

[0104] It can be understood that after G times of image enhancement, that is, after Step 3 is executed G times, the enhanced dataset D can be obtained from all the enhanced image pairs, specifically composed of and components.

[0105] Using the enhanced dataset to train a preset deep neural network in a supervised manner to obtain a trained infrared image correction model includes:

[0106] Taking the dirty images in the enhanced dataset as the input data of the preset deep neural network, taking the corresponding target images as label data, using the Adam optimizer to optimize the preset loss function for iterative training until the convergence condition is reached to obtain a weight file, and using the weight file to configure the preset deep neural network to obtain a trained infrared image correction model; among them, the preset deep neural network includes a U-Net neural network.

[0107] Specifically, the training data is the enhanced dataset D. Taking the standard 3-layer U-Net as an example, for example, given a dirty image The target image input to the network is the corresponding Then the loss function L in the training process can be expressed as:

[0108]

[0109] where SSIM(·) is the structural similarity function; f(·) represents the U-Net neural network. The batch size of the network input can be 16. The Adam optimizer can be used to optimize the loss function L. The initial learning rate of training can be set to 0.001, and then gradually decay to 1×10 -6 , the number of iteration rounds is 100, and the training can be completed when the loss converges to less than 0.005 to obtain a weight file UNet.pth. Using the weight file UNet.pth to configure the preset deep neural network, a trained infrared image correction model can be obtained. Then, after inputting the infrared image to be corrected into the infrared image correction model, the corresponding non-uniformity corrected image can be output.

[0110] Aiming at the problem that the existing temperature calibration method has poor calibration effect in a wide temperature range, the embodiment of the present invention combines the temperature calibration method with the deep learning method, and provides a wide-temperature large-dynamic infrared image calibration method based on blackbody data enhancement. By randomly sampling wide-temperature blackbody data through a multi-scale sliding window, the spatial dependence of different response pixels on the focal plane array can be broken, which enables this solution to randomly replace the response temperature of pixels at the same position on the focal plane array of the infrared detector, and implement blackbody data enhancement. The enhanced data can force the blackbody data with a large-dynamic wide-temperature distribution to be aggregated into an infrared image, establish an association between the wide-temperature large-dynamic response of the detection pixels and their spatial response, help the deep learning algorithm extract the pixel response characteristics of wide-temperature large-dynamic, provide data support for the deep learning algorithm to learn a continuous and accurate calibration curve, and achieve effective non-uniformity correction of infrared images.

[0111] To verify the effect of the method in the embodiment of the present invention, the following experimental data are given and described.

[0112] Among them, Figures 3(a) to 3(c) is the data enhancement result of the method in the embodiment of the present invention, Figures 3(a) to 3(c) which respectively represent the output results when the sliding step s is 2, 4, and 8; Figures 3(a) to 3(c) In, the left and right images in each row are the enhanced dirty image and the target image respectively.

[0113] Figs. 4(a) and 4(b) are the calibration results of the infrared images of two scenes using the trained U-Net network and the effect of two-point calibration when the data in the embodiment of the present invention is enhanced to 50,000 pairs. The three images from left to right in Figs. 4(a) and 4(b) are the uncalibrated image, the image after two-point calibration, and the image after calibration by the method of the present invention. It can be seen that there are still relatively light stripes remaining after two-point calibration, while the processing result of the method in the embodiment of the present invention is more uniform in the flat area of the image and retains details such as building edges at the same time.

[0114] Figure 5 (a) and Figure 5 (b) are the column response mean curves corresponding to the two scenes in Figs. 4(a) and 4(b) respectively. It can be seen that the column response of the method in the embodiment of the present invention is smoother, while the column response curve of two-point calibration shows fluctuations due to the remaining stripe noise.

[0115] In the second aspect, corresponding to the above method embodiment, the embodiment of the present invention also provides a wide-temperature large-dynamic infrared image calibration method device based on blackbody data enhancement, as Figure 6 shown, the device includes:

[0116] The blackbody data acquisition module 601 is configured to collect, for each of the preset H temperatures, N blackbody images at that temperature by using an infrared detector to be calibrated, and the blackbody images obtained from the H temperatures constitute blackbody data; wherein, the H temperatures vary continuously, H is at least 10, and the temperature step includes 5°C or 10°C;

[0117] Each temperature image group acquisition module 602 is configured to calculate the effective blackbody image and the corresponding paired image at each temperature in the blackbody data as an image group at the corresponding temperature;

[0118] The enhanced image pair acquisition module 603 is configured to determine the sliding window adopted in the current cycle based on the maximum scale to be enhanced; execute the local region map acquisition step, including obtaining a fused image based on a random image group and processing it with the sliding window to obtain a local region map of the fused enhanced image; repeat the local region map acquisition step and splice the obtained local region maps until the local region map acquisition step is executed a preset number of times to obtain a fused enhanced image, and separate it to obtain the enhanced dirty image and the target image, constituting the enhanced image pair in the current cycle;

[0119] The judgment module 604 is configured to judge whether the number of cycles reaches G times; if not, execute the processing process of the enhanced image pair acquisition module, and if so, execute the processing process of the network training and calibration module;

[0120] The network training and calibration module 605 is configured to constitute an enhanced data set from all the obtained enhanced image pairs; use the enhanced data set to train a preset deep neural network in a supervised manner to obtain a trained infrared image calibration model for outputting a non-uniformity corrected image for the input infrared image to be calibrated.

[0121] Optionally, when calculating the effective blackbody image and the corresponding paired image at each temperature in the blackbody data, the each temperature image group acquisition module 602 is specifically configured to:

[0122] For each temperature, obtain the effective blackbody image at that temperature by performing pixel averaging on all the blackbody images at that temperature;

[0123] For each temperature, determine the uniform response image of the focal plane array for that temperature by using the effective blackbody image at that temperature as the paired image corresponding to the effective blackbody image at that temperature.

[0124] Optionally, when obtaining the effective blackbody image at each temperature by performing pixel averaging on all the blackbody images at that temperature for each temperature, the calculation formula adopted by the each temperature image group acquisition module 602 includes:

[0125]

[0126] Among them, is the effective blackbody image at temperature T g ; is the k-th blackbody image at temperature T g ; (i, j) is the position of the pixel in the image; g ∈ [1, H].

[0127] Optionally, when the temperature image group acquisition module 602 determines, for each temperature, the uniform response image of the focal plane array for the temperature by using the effective blackbody image at the temperature as the paired image corresponding to the effective blackbody image at the temperature, the calculation formula adopted includes:

[0128]

[0129] Among them, is the paired image corresponding to the effective blackbody image at temperature T g ; m×n is the total number of pixels of the infrared focal plane array of the infrared detector to be corrected, and is also the number of pixels of each obtained blackbody image.

[0130] Optionally, when the enhanced image pair acquisition module 603 determines the sliding window adopted in the current loop based on the maximum scale to be enhanced, it is specifically used for:

[0131] Generating a random number s according to the preset maximum scale l to be enhanced, where s = random(1, l);

[0132] Defining a sliding window W with both the size and the step size being s s as the sliding window adopted in the current loop.

[0133] Optionally, when the enhanced image pair acquisition module 603 executes the local area map acquisition step, including obtaining a fused image based on a random image group and using the sliding window to process it to obtain a local area map of the fused enhanced image; repeating the local area map acquisition step and stitching the obtained local area maps until the local area map acquisition step is executed a preset number of times to obtain a fused enhanced image, and separating it to obtain a dirty image and a target image after enhancement, to form the enhanced image pair of the current loop, it is specifically used to execute the following steps:

[0134] Step a1, generating a random number r;

[0135] r = random(1, H)

[0136] Step a2, selecting a group of image groups with the temperature being the random number r from all the image groups corresponding to H temperatures and And splice the two along the image channel dimension to obtain the corresponding fused image

[0137]

[0138] Among them, Cat represents splicing, dim = 1 represents the first dimension, and the first dimension is the image channel dimension;

[0139] Step a3, use the sliding window W adopted in the current loop s to extract the fused features from the fused image once to obtain the local region map of the fused enhanced image in the current loop;

[0140] Step a4, determine whether the number of executions of the local region map acquisition step reaches a preset number If not, execute step a1 again; and starting from the second execution of the local region map acquisition step, splice the latest obtained local region map with the previously obtained local region map after each execution; if so, use the currently spliced local region map as the fused enhanced image in the current loop and execute step 5;

[0141] Step a5, separate the fused enhanced image along the image channel dimension to obtain the enhanced dirty image and the target image, forming the enhanced image pair in the current loop.

[0142] Optionally, when the network training and calibration module 605 trains a preset deep neural network in a supervised manner using the enhanced data set to obtain a trained infrared image calibration model, it is specifically used for:

[0143] Use the dirty images in the enhanced data set as the input data of the preset deep neural network, use the corresponding target images as label data, optimize the preset loss function using the Adam optimizer for iterative training until the convergence condition is reached to obtain a weight file, and configure the preset deep neural network using the weight file to obtain a trained infrared image calibration model; among them, the preset deep neural network includes a U-Net neural network.

[0144] For the specific processing procedures of each module of the device, please refer to the relevant content in the first aspect and will not be elaborated here.

[0145] In a third aspect, an embodiment of the present invention further provides an electronic device, as Figure 7 shown, including a processor 701, a communication interface 702, a memory 703, and a communication bus 704. Among them, the processor 701, the communication interface 702, and the memory 703 complete communication with each other through the communication bus 704,

[0146] The memory is used to store a computer program;

[0147] When the processor is used to execute the program stored in the memory, the steps of any method for correcting a wide-temperature large-dynamic infrared image based on blackbody data enhancement provided in the first aspect of the embodiments of the present invention are implemented.

[0148] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0149] The communication interface is used for communication between the above electronic device and other devices.

[0150] The memory may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0151] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0152] The method provided by the embodiments of the present invention can be applied to an electronic device. Specifically, the electronic device may be: a desktop computer, a portable computer, a smart mobile terminal, a server, etc. There is no limitation here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0153] For the device / electronic device / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, please refer to the partial description of the method embodiments.

[0154] It should be noted that the device, electronic device, and storage medium in the embodiments of the present invention are respectively the device, electronic device, and storage medium applying the above-mentioned wide-temperature large-dynamic infrared image correction based on blackbody data enhancement. Then all the embodiments of the above-mentioned wide-temperature large-dynamic infrared image correction based on blackbody data enhancement are applicable to the device, electronic device, and storage medium, and can achieve the same or similar beneficial effects.

[0155] The above description is only for the preferred embodiments of the present invention and is not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A wide-temperature large-dynamic infrared image correction method based on blackbody data enhancement, characterized in that Including: Step 1: For each of the preset H temperatures, use the infrared detector to be calibrated to collect N blackbody images at each temperature. The blackbody images obtained from the H temperatures form blackbody data. Among them, the H temperatures vary continuously, H is at least 10, and the temperature step includes 5°C or 10°C. Step 2: Calculate the effective blackbody image and the corresponding paired image at each temperature in the blackbody data as an image group at the corresponding temperature. Step 3: Determine the sliding window used in the current cycle based on the maximum scale to be enhanced. Execute the local region map acquisition step, including obtaining a fused image based on a random image group and using the sliding window to process it to obtain a local region map of the fused enhanced image. Repeat the local region map acquisition step and splice the obtained local region maps until the local region map acquisition step is executed a preset number of times to obtain a fused enhanced image, and separate it to obtain the enhanced dirty image and the target image, forming the enhanced image pair of the current cycle. Step 4: Determine whether the number of cycles has reached G times. If not, execute Step 3; if so, execute Step 5. Step 5: The enhanced data set is composed of all the obtained enhanced image pairs. Use the enhanced data set to train a preset deep neural network in a supervised manner to obtain a trained infrared image correction model for outputting a non-uniformity corrected image for the input infrared image to be corrected.

2. The wide-temperature large-dynamic infrared image correction method based on blackbody data enhancement according to claim 1, characterized in that In Step 2, calculating the effective blackbody image and the corresponding paired image at each temperature in the blackbody data includes: For each temperature, obtain the effective blackbody image at that temperature by averaging the pixels of all the blackbody images at that temperature. For each temperature, use the effective blackbody image at that temperature to determine the uniform response image of the focal plane array at that temperature as the paired image corresponding to the effective blackbody image at that temperature.

3. The method for correcting wide-temperature and large-dynamic infrared images based on blackbody data enhancement according to claim 2, characterized in that For each temperature, the calculation formula for obtaining the effective blackbody image at that temperature by averaging the pixels of all the blackbody images at that temperature includes: in, is the temperature T g The effective black body image below; is the temperature T g The kth black body image under the image; (i, j) is the position of the pixel in the image; g∈[1,H].

4. The wide-temperature large-dynamic infrared image correction method based on blackbody data enhancement according to claim 3, wherein For each temperature, the calculation formula for using the effective blackbody image at that temperature to determine the uniform response image of the focal plane array at that temperature as the paired image corresponding to the effective blackbody image at that temperature includes: Among them, is the paired image corresponding to the effective blackbody image at temperature T; m×n is the total number of pixels in the infrared focal plane array of the infrared detector to be corrected, and is also the number of pixels in each obtained blackbody image. g ​ 5. The wide-temperature large-dynamic infrared image correction method based on blackbody data enhancement according to claim 4, wherein In Step 3, determining the sliding window used in the current cycle based on the maximum scale to be enhanced includes: According to the preset maximum scale l to be enhanced, generate a random number s, where s = random(1, l). Define a sliding window W with both size and step length being s s as the sliding window adopted in the current loop.

6. The wide-temperature large-dynamic infrared image correction method based on blackbody data enhancement according to claim 5, wherein In Step 3, executing the local region map acquisition step, including obtaining a fused image based on a random image group and using the sliding window to process it to obtain a local region map of the fused enhanced image. Repeat the local region map acquisition step and splice the obtained local region maps until the local region map acquisition step is executed a preset number of times to obtain a fused enhanced image, and separate it to obtain the enhanced dirty image and the target image, forming the enhanced image pair of the current cycle, includes: Step a1: Generate a random number r. r = random(1, H) Step a2, select a group of images corresponding to the random number r from all groups of images corresponding to H temperatures and concatenate the two along the image channel dimension to obtain the corresponding fused image Among them, Cat represents concatenation, dim = 1 represents the first dimension, and the first dimension is the image channel dimension; Step a3, using the sliding window W adopted in the current loop s to extract once the fused features from the fused image to obtain a local region map of the fused enhanced image for the current loop; Step a4: Determine whether the number of executions of the local area map acquisition step has reached a preset number If not, execute step a1 again; and starting from the second execution of the local area map acquisition step, splice the latest obtained local area map with the previously obtained local area map after each execution; if so, use the currently spliced local area map as the fused and enhanced image for the current loop, and execute step 5; Step a5: Separate the fused enhanced image along the image channel dimension to obtain the enhanced dirty image and the target image, forming the enhanced image pair for the current iteration.

7. The method for correcting wide-temperature and large-dynamic infrared images based on blackbody data enhancement according to claim 6, wherein In step 5, use the enhanced dataset to train a preset deep neural network in a supervised manner to obtain a trained infrared image correction model, including: Use the dirty images in the enhanced dataset as the input data of the preset deep neural network, use the corresponding target images as label data, optimize the preset loss function using the Adam optimizer for iterative training until the convergence condition is reached to obtain a weight file, and configure the preset deep neural network using the weight file to obtain a trained infrared image correction model; among them, the preset deep neural network includes a U-Net neural network.

8. An infrared image correction device with wide temperature and large dynamic range based on blackbody data enhancement, characterized in that, Including: Blackbody data acquisition module, for each of the preset H temperatures, use the infrared detector to be corrected to collect N blackbody images at this temperature, and the blackbody images obtained from the H temperatures constitute blackbody data; among them, the H temperatures change continuously, H is at least 10, and the temperature step size includes 5°C or 10°C; Each temperature image group acquisition module, for calculating the effective blackbody image and the corresponding paired image at each temperature in the blackbody data as an image group at the corresponding temperature; Enhanced image pair acquisition module, for determining the sliding window used in the current iteration based on the maximum scale to be enhanced; execute the local region map acquisition step, including obtaining a fused image based on a random image group and processing it using the sliding window to obtain a local region map of the fused enhanced image; repeat the local region map acquisition step and splice the obtained local region maps until the local region map acquisition step is executed a preset number of times to obtain a fused enhanced image, and separate it to obtain the enhanced dirty image and the target image, forming the enhanced image pair for the current iteration; Judgment module, for judging whether the number of iterations reaches G times; if not, execute the processing process of the enhanced image pair acquisition module, if so, execute the processing process of the network training and correction module; Network training and correction module, for forming an enhanced dataset from all the obtained enhanced image pairs; use the enhanced dataset to train a preset deep neural network in a supervised manner to obtain a trained infrared image correction model for outputting a non-uniformity corrected image for the input infrared image to be corrected.

9. An electronic device, characterized in that, Including a processor, a communication interface, a memory, and a communication bus, among which, the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used for storing a computer program; When the processor executes the program stored on the memory, it implements the method steps of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method steps of any one of claims 1-7.

Images