An infrared image processing method, device, medium and equipment

Abstract

This application relates to the field of image processing technology, specifically disclosing an infrared image processing method, apparatus, medium, and device. The method includes: performing photometric parameter separation processing on a current raw infrared image captured by an infrared camera to obtain an initial detail layer and an initial base layer at the current moment; performing denoising processing on the initial detail layer based on the target fixed-pattern noise at the current moment to obtain a target detail layer at the current moment; determining the target base layer at the current moment using an iterative optimization method based on the initial base layer from the previous moment, the maximum response value from the previous moment, the minimum response value from the previous moment, the initial maximum response value from the current moment, the initial minimum response value from the current moment, the initial base layer at the current moment, and a feature matching set; and performing image fusion processing based on the target detail layer and the target base layer to obtain a target infrared image. This application can improve the processing effect of infrared images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an infrared image processing method, apparatus, medium and device. Background Technology

[0002] Infrared cameras, due to their night vision capabilities and strong adaptability to environmental changes, have been increasingly adopted in the field of Simultaneous Localization and Mapping (SLAM). Compared to visible light images, infrared cameras operate in the 7-14µm spectral band, are insensitive to visible light, are less affected by illumination interference, and can achieve scene recognition under different lighting conditions and even at all times of day. Although infrared images and visible light images share some similarities in reflecting scene structure information, making their visual positioning principles similar, infrared images generally suffer from problems such as less information, lower quality, lower signal-to-noise ratio, and high-temperature drift. This results in infrared SLAM systems typically having lower robustness and accuracy than visible light solutions. Therefore, improving the robustness and positioning accuracy of infrared SLAM is a key issue that must be addressed for its practical application.

[0003] Infrared images face challenges of photometric variation in the spatiotemporal dimensions: 1) When fast adaptive gain directly converts infrared images into a common 8-bit format, the entry and exit of hot / cold objects in the field of view will cause a wide range of pixel saturation, resulting in photometric variations in the thermal image over time; 2) Infrared cameras perform non-uniformity correction to reduce noise during operation, during which the camera needs to stop working briefly, resulting in photometric differences between frames; 3) The non-uniformity of the focal plane microbolometer array causes a fixed thermal radiating object to exhibit different responses at different focal plane positions, which manifests as fixed pattern noise (FPN) in the infrared image space.

[0004] In infrared image processing, the most classic method is based on histogram equalization (HE). The core idea of ​​histogram equalization is to non-linearly stretch and remap the pixel value probability distribution (i.e., the histogram) of the entire image to make it approximately uniformly distributed in the output image. However, it has several drawbacks: Excessive detail magnification and background noise enhancement: While stretching image contrast, HE indiscriminately amplifies the differences across all gray levels. This means that while enhancing useful target signals, useless background noise such as sensor background noise and fixed pattern noise (FPN) is also synchronously, or even excessively, amplified. This results in the output image signal-to-noise ratio not being effectively improved, and sometimes even decreasing, producing "salt and pepper" or blocky noise that severely interferes with subsequent feature extraction and recognition.

[0005] Gray-level merging and detail loss, gray-level saturation: When the gray-level distribution of the original image is concentrated in a narrow range, the global stretching mapping of HE may cause multiple adjacent input gray-levels to be mapped to the same output gray-level, resulting in the merging and loss of gray-level information. Although this improves the overall contrast, it may sacrifice subtle local temperature difference details, causing important but small thermal targets to become blurred or disappear after enhancement.

[0006] Poor adaptability to dynamic scenes and temporal instability: Heat sources in infrared scenes, such as vehicles, pedestrians, and flames, are dynamically changing. Their appearance or disappearance drastically alters the grayscale statistical characteristics (maximum value Imax and minimum value Imin) of the image. HE relies on the global statistics of the entire frame, causing drastic jumps in the grayscale mapping function between adjacent frames. This temporal inconsistency in the mapping relationship leads to unpredictable changes in the pixel values ​​of the same physical heat source in adjacent frames, severely disrupting the photometric consistency of the image sequence and directly causing the failure of tasks such as feature tracking and optical flow calculation based on appearance matching.

[0007] Ignoring spatial structure information: HE is a pixel-level operation based entirely on grayscale value distribution, completely ignoring the spatial relationships and local context information between pixels. It cannot distinguish whether a grayscale region is a flat background or an object containing texture. Therefore, this method lacks the ability to suppress spatial fixed-pattern noise (FPN), which is one of the core factors affecting image quality in infrared imaging.

[0008] Therefore, there is an urgent need for an infrared image processing method to solve the problems of deviation in the processing effect of infrared images, insufficient precision in the detail enhancement processing of infrared images, and unstable processing effect in the existing technology. Summary of the Invention

[0009] In view of this, the present invention provides an infrared image processing method, apparatus, medium and device, the main purpose of which is to solve the problems of deviation in the current infrared image processing effect, insufficient precision in infrared image detail enhancement processing, and unstable processing effect.

[0010] To address the above problems, this application provides an infrared image processing method, comprising: The photometric parameters of the current raw infrared image captured by the infrared camera are separated to obtain the initial detail layer and the initial base layer at the current moment. Based on the target fixed pattern noise corresponding to the current moment, the initial detail layer at the current moment is denoised to obtain the target detail layer at the current moment; Based on the initial base layer of the previous time step, the maximum response value of the previous time step, the minimum response value of the previous time step, the initial maximum response value of the current time step, the initial minimum response value of the current time step, the initial base layer of the current time step, and the feature matching set between the initial base layer of the previous time step and the initial base layer of the current time step, the target base layer of the current time step is determined by iterative optimization. Based on the target detail layer and the target base layer at the current moment, image fusion processing is performed to obtain the processed target infrared image.

[0011] Optionally, the step of performing photometric parameter separation processing on the current raw infrared image captured by the infrared camera to obtain the initial detail layer and the initial base layer at the current moment specifically includes: The current original infrared image is subjected to photometric separation processing based on a nonlinear filter to obtain the initial detail layer and the initial base layer at the current moment.

[0012] Optionally, before performing denoising processing on the initial detail layer at the current moment based on the target fixed-mode noise corresponding to the current moment, the method further includes: determining the target fixed-mode noise corresponding to the current moment based on the state type of the infrared camera at the current moment, specifically including: When the infrared camera is in a non-motion state, the historical fixed pattern noise from the previous moment is determined as the target fixed pattern noise at the current moment. When the infrared camera is in motion mode, the initial fixed-mode noise at the current moment is determined based on the initial detail layer at the current moment and the initial detail layer at the previous moment.

[0013] Optionally, before determining the target base layer at the current moment, the method further includes: determining the maximum response value and the minimum response value at the previous moment, specifically including: Based on the predetermined sliding window corresponding to the previous moment, the response value is smoothed by using the historical maximum and minimum response values ​​corresponding to each historical moment within the predetermined sliding window, so as to obtain the maximum response value and the minimum response value of the previous moment.

[0014] Optionally, before determining the target base layer at the current moment, the method further includes determining the feature matching set, specifically including: Edge extraction is performed on the initial base layer of the previous time step to obtain the edge map of the previous time step; For each pixel in the initial base layer of the previous time step, the distance field is extracted using the edge image of the previous time step to obtain the nearest edge distance corresponding to each pixel, so as to obtain the proxy texture map of the previous time step. Edge extraction is performed on the initial base layer at the current time to obtain the edge map at the current time; For each pixel in the initial base layer at the current time, the distance field is extracted using the edge image at the current time to obtain the nearest edge distance corresponding to each pixel, so as to obtain the proxy texture map at the current time. Based on the proxy texture map of the previous time step and the proxy texture map of the current time step, feature matching is performed to obtain the feature matching set.

[0015] Optionally, after performing distance field extraction to obtain the nearest edge distance corresponding to each pixel, the method further includes: The nearest edge distance of each pixel is constrained based on a predetermined distance threshold, so that the nearest edge distance is less than or equal to the predetermined distance threshold.

[0016] Optionally, the target base layer at the current moment is determined using an iterative optimization method based on the initial base layer of the previous moment, the maximum response value of the previous moment, the minimum response value of the previous moment, the initial maximum response value of the current moment, the initial minimum response value of the current moment, the initial base layer of the current moment, and the feature matching set between the initial base layer of the previous moment and the initial base layer of the current moment. Specifically, this includes: Based on the initial base layer, the maximum response value, and the minimum response value of the previous time step, determine the target base layer of the previous time step. The current base layer is determined based on the initial base layer, the initial maximum response value, and the initial minimum response value at the current moment. Based on the current base layer at the current time, the target base layer at the previous time, and the feature matching set, the initial maximum response value and the initial minimum response value at the current time are optimized to obtain the current maximum response value and the current minimum response value at the current time, and to determine whether the predetermined optimization conditions are met. If the optimization conditions are not met, the current base layer is redefined based on the initial base layer, the current maximum response value, and the current minimum response value at the current moment. Then, based on the redefined base layer, the target base layer from the previous moment, and the feature matching set, the current maximum response value and the current minimum response value are optimized until the predetermined optimization conditions are met. At that point, the current maximum response value is taken as the target maximum response value and the current minimum response value is taken as the target minimum response value. The target base layer is determined based on the initial base layer at the current moment, the target maximum response value at the current moment, and the target minimum response value at the current moment.

[0017] To address the above problems, this application provides an infrared image processing apparatus, comprising: The separation module is used to perform photometric parameter separation processing on the current raw infrared image captured by the infrared camera to obtain the initial detail layer and the initial base layer at the current moment. The denoising module is used to denoise the initial detail layer at the current moment based on the target fixed pattern noise corresponding to the current moment, so as to obtain the target detail layer at the current moment. The determination module is used to determine the target base layer at the current moment based on the initial base layer at the previous moment, the maximum response value at the previous moment, the minimum response value at the previous moment, the initial maximum response value at the current moment, the initial minimum response value at the current moment, the initial base layer at the current moment, and the feature matching set between the initial base layer at the previous moment and the initial base layer at the current moment, using an iterative optimization method. The fusion module is used to perform image fusion processing based on the target detail layer and the target base layer at the current moment to obtain the processed target infrared image.

[0018] To address the aforementioned problems, this application provides a storage medium storing a computer program that, when executed by a processor, implements the steps of any of the infrared image processing methods described above.

[0019] To address the aforementioned problems, this application provides an electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program in the memory, implements the steps of any of the infrared image processing methods described above.

[0020] The infrared image processing method, apparatus, medium, and device in this application decompose the original infrared image into an initial detail layer and an initial base layer by performing photometric parameter separation processing, thereby effectively decoupling detail information from photometric information. By performing target fixed-mode noise denoising processing on the initial detail layer, noise interference can be suppressed while preserving effective detail features, thus enhancing the detail of the infrared image. By introducing the base layer, response extrema, and feature matching set from the previous and current moments, and using an iterative optimization method to determine the target base layer at the current moment, the temporal information and global photometric features can be fully utilized to ensure that the base layer at the current moment maintains a stable, continuous, and consistent photometric response with historical frames, avoiding inter-frame brightness abrupt changes and distortion, thereby ensuring the photometric consistency of the infrared image sequence. Finally, the target detail layer and the target base layer are fused, which can significantly enhance image details while maintaining overall photometric stability and inter-frame consistency, so that the processed infrared image has both high detail performance and high photometric fidelity, improving the imaging quality and reliability of the infrared imaging system in complex scenes. This invention achieves enhanced detail while suppressing noise (especially FPN), ensuring high photometric consistency in infrared image sequences. This provides stable and reliable data input for infrared visual SLAM, improving its positioning accuracy and environmental adaptability. It also enhances detail while ensuring photometric consistency in infrared images, reducing processing deviations and improving image detail, resulting in more stable final processed infrared images. The above description is merely an overview of the invention's technical solution. To better understand the technical means of this invention and to facilitate its implementation according to the specification, and to make the above and other objects, features, and advantages of this invention more apparent, specific embodiments of the invention are described below. Attached Figure Description

[0021] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an infrared image processing method according to an embodiment of this application; Figure 2(a) is a schematic diagram of the initial detail layer in an embodiment of this application; Figure 2(b) is a schematic diagram of the target detail layer in an embodiment of this application; Figure 3(a) is a schematic diagram of the initial base layer in an embodiment of this application; Figure 3(b) is a schematic diagram of the target base layer in an embodiment of this application; Figure 4 This is a structural block diagram of an infrared image processing device according to another embodiment of this application; Figure 5 This is a structural block diagram of an electronic device according to another embodiment of this application. Detailed Implementation

[0022] Various embodiments and features of this application are described herein with reference to the accompanying drawings.

[0023] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.

[0024] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0025] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.

[0026] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.

[0027] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.

[0028] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely to serve as a representative basis for teaching those skilled in the art to use this application in a variety of substantially any suitable detailed structures.

[0029] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.

[0030] This application provides an infrared image processing method, such as... Figure 1 As shown, it includes the following steps: Step S101: Perform photometric parameter separation processing on the current raw infrared image captured by the infrared camera to obtain the initial detail layer and the initial base layer at the current moment. In practice, this step can be performed on the original infrared image using a nonlinear filter to obtain the initial detail layer D at the current time t. t and the initial base layer B at the current moment. t Nonlinear filters can be, for example, bilateral filters.

[0031] In this step, the current raw infrared image can be decomposed into a 14-bit base layer (low-frequency spatial component) and a 14-bit detail layer (high-frequency spatial component) using a bilateral filter.

[0032] Step S102: Based on the target fixed pattern noise corresponding to the current moment, perform denoising processing on the initial detail layer at the current moment to obtain the target detail layer at the current moment; In the specific implementation process of this step, the target fixed pattern noise (FPN) corresponding to the current moment can be determined first based on the state type of the infrared camera at the current moment. Then, the target fixed pattern noise FPN is used to apply the initial detail layer D at the current moment. t Denoising is performed to obtain the target detail layer D at the current moment. t '.in, .

[0033] Step S103: Based on the initial base layer of the previous time step, the maximum response value of the previous time step, the minimum response value of the previous time step, the initial maximum response value of the current time step, the initial minimum response value of the current time step, the initial base layer of the current time step, and the feature matching set between the initial base layer of the previous time step and the initial base layer of the current time step, the target base layer of the current time step is determined by iterative optimization. In the specific implementation process of this step, the feature matching set C can be determined first. t Then, based on the initial base layer of the previous time step... The maximum response value at the previous time step, the minimum response value at the previous time step, the initial maximum response value at the current time step, the initial minimum response value at the current time step, and the initial base layer at the current time step. and the initial base layer of the previous time step. The initial base layer at the current moment The feature matching set between the two is used to determine the target base layer B at the current time through iterative optimization. t '.

[0034] Step S104: Based on the target detail layer and the target base layer at the current moment, perform image fusion processing to obtain the processed target infrared image.

[0035] In this step, the target detail layer D at the current time t is obtained. t'and target base layer B' t After that, the target detail layer D can be... t 'and target base layer B' t The images are fused together to obtain a processed infrared image of the target.

[0036] The method in this embodiment separates the photometric parameters of the original infrared image into an initial detail layer and an initial base layer, effectively decoupling detail information from photometric information. By performing target fixed-mode noise denoising on the initial detail layer, noise interference can be suppressed while preserving effective detail features, thus enhancing the detail of the infrared image. By introducing the base layer, response extrema, and feature matching set from the previous and current moments, and using an iterative optimization method to determine the target base layer at the current moment, the method can fully utilize temporal information and global photometric features, ensuring that the base layer at the current moment maintains a stable, continuous, and consistent photometric response with historical frames, avoiding abrupt changes and distortions in brightness between frames, thereby ensuring the photometric consistency of the infrared image sequence. Finally, the target detail layer and the target base layer are fused, which can significantly enhance image details while maintaining overall photometric stability and inter-frame consistency. This results in an infrared image with both high detail and high photometric fidelity, improving the imaging quality and reliability of the infrared imaging system in complex scenes. This approach achieves enhanced detail while suppressing noise (especially FPN), ensuring high photometric consistency in infrared image sequences. This provides stable and reliable data input for infrared visual SLAM, improving its positioning accuracy and environmental adaptability. Furthermore, by enhancing detail and ensuring photometric consistency in infrared images, it reduces processing deviations, improves image detail, and ultimately results in more stable infrared image processing.

[0037] Based on the above embodiments, another embodiment of this application provides an infrared image processing method, including the following steps: Step S201: Perform photometric separation processing on the current original infrared image based on a nonlinear filter to obtain the initial detail layer and the initial base layer at the current moment.

[0038] In this embodiment, photometric separation processing facilitates subsequent estimation of the obtained temporal parameters (base layer) and spatial parameters (detail layer), thereby effectively reducing the complexity of correction. Specifically, a bilateral filter decomposes the 14-bit infrared image into a 14-bit base layer (low-frequency spatial component) and a 14-bit detail layer (high-frequency spatial component).

[0039] Step S202: Based on the state type of the infrared camera at the current moment, determine the target fixed pattern noise corresponding to the current moment; In the specific implementation process of this step, the state types include both non-motion state and motion state, that is, determining the target fixed pattern noise is divided into the following two cases: Case 1: When the infrared camera is in a non-motion state, the historical fixed pattern noise from the previous moment is determined as the target fixed pattern noise at the current moment; Scenario 2: When the infrared camera's state type is in motion, based on the initial detail layer D at the current moment... t And the initial detail layer D from the previous moment. t-1 Determine the initial fixed-mode noise FPN at the current time. t Based on the initial fixed-mode noise at the current moment and historical fixed-mode noise FPN T-1 Determine the target fixed-mode noise FPN at the current moment. T .

[0040] Wherein, the initial fixed-mode noise FPN at the current time t t (i.e., FPN) t The formula for calculating (i, j) is: .

[0041] Target Fixed-Mode Noise FPN at current time t T The calculation formula is: .

[0042] Among them, FPN T It was obtained through multi-frame estimation. The current time step determines the estimation of FPN. T-1 The weighting of FPN. After convergence over multiple frames, FPN... T While closer to the true value, this approach may introduce interfering ghosting. The main reason is that when the infrared camera is stationary, detailed textures are misidentified as FPN; furthermore, directly using the above formula for initialization leads to excessive reliance on the first frame's data for FPN. Therefore, this application analyzes the angular velocity and acceleration data of the IMU at the current moment to determine whether the camera is in motion, thus ensuring that FPN statistics and updates are only performed when the camera is moving. Simultaneously, initialization is performed by calculating the average FPN estimate of the first n moving frames to avoid excessive reliance on specific frames. Experience shows that approximately 100 frames of data can effectively suppress ghosting and ensure the accuracy of fixed-pattern noise estimation.

[0043] Step S203: Based on the target fixed pattern noise corresponding to the current moment, perform denoising processing on the initial detail layer at the current moment to obtain the target detail layer at the current moment.

[0044] In this step, the calculation formula for the target detail layer is: D t =Dt -FPN T .

[0045] Among them, D t 'This is the target detail layer at the current moment, which is an 8-bit image, meaning the pixel value range is 8 bits; FPN T The target noise at the current moment is a fixed pattern. (D) t The initial detail layer at the current moment is shown in Figure 2(a), which is the detail layer before spatial parameter calibration; Figure 2(b) is the target detail layer at the current moment, which is the detail layer after spatial parameter calibration.

[0046] Step S204: Determine the maximum response value and the minimum response value of the previous time step; Step S205: Based on the initial base layer of the previous time step and the initial base layer at the current moment Determine the feature matching set C t ; Step 6: Based on the initial base layer of the previous time step The maximum response value at the previous moment The minimum response value at the previous time step Determine the target base layer of the previous moment. ; The formula for calculating the target base layer at the previous moment is:

[0047] in, This represents the initial base layer at the previous time step / t-1, which contains 14-bit raw pixel values. This indicates the pixel position at time t-1; This represents the pixel position at time t; This represents spatial deviation.

[0048] Step S206: Initial base layer based on the current time. The initial maximum response value at the current moment. and the initial minimum response value at the current moment. Determine the current base layer B at the current moment. t '; The formula for calculating the current base layer at the current moment is:

[0049] Among them, B t 'Right now This is the current detail layer at the current moment, which is an 8-bit image, meaning the pixel values ​​range from 8 bits. This represents the initial base layer at the current moment, which contains 14-bit raw pixel values. This indicates the pixel position at time t-1; This represents the pixel position at time t; This represents spatial bias. In other words, by using the above formula, the pixel values ​​of a 14-bit infrared image can be mapped to 8 bits.

[0050] Step S207: Based on the current base layer B at the current moment t 'The target base layer B at the previous moment' t-1 'and feature matching set C t The initial maximum response value and the initial minimum response value at the current time are optimized to obtain the current maximum response value and the current minimum response value at the current time, and it is determined whether the predetermined optimization conditions are met. If the optimization conditions are not met, the current base layer is redefined based on the initial base layer, the current maximum response value, and the current minimum response value at the current moment. Then, based on the redefined base layer, the target base layer from the previous moment, and the feature matching set, the current maximum response value and the current minimum response value are optimized until the predetermined optimization conditions are met. At that point, the current maximum response value is taken as the target maximum response value and the current minimum response value is taken as the target minimum response value. In this embodiment, since the FPN has been separated and individually calibrated in the detail layer, therefore The negligible influence during the photometric calibration of the base layer necessitates only the estimation of the maximum and minimum response parameters. The maximum likelihood estimation corresponds to the solution of the least squares objective, and the solution formula is as follows:

[0051] in This represents the set of feature matches between frame t-1 and frame t. Good matching can improve the accuracy of the solution. Various advanced methods can be used to obtain feature matches.

[0052] Step S208: Initial base layer based on the current time. The target's maximum and minimum response values ​​at the current moment are used to determine the target's base layer B at the current moment. t '.

[0053] In this embodiment, the initial base layer before time parameter calibration and the target base layer after time parameter calibration can be shown in Figures 3(a) and 3(b). That is, Figure 3(a) is the base layer / initial base layer before photometric correction, and Figure 3(b) is the base layer / target base layer after photometric correction. By comparison, it can be seen that the method in this application can effectively avoid interference from high-temperature objects.

[0054] In this embodiment, the detail layer contains most of the FPN and is less affected by hot and cold objects in the scene. This is because the bilateral filter, as a nonlinear filter, can suppress the leakage of sharp edges caused by hot and cold objects into the detail layer. Correspondingly, the base layer is significantly affected by hot and cold objects but is not sensitive to FPN. Based on this, this embodiment estimates the FPN in the detail layer and estimates appropriate minimum / maximum response values ​​in the base layer, finally generating a photometrically corrected infrared image by fusing the two layers. In addition, this layered approach also brings the additional advantage of significantly enhancing the detail texture of the infrared image.

[0055] Based on the above embodiments, when determining the maximum and minimum response values ​​of the previous moment, that is, when executing step S204, the following method can be adopted: Based on the predetermined sliding window corresponding to the previous moment, the response value smoothing process is performed using the historical maximum and historical minimum response values ​​corresponding to each historical moment within the predetermined sliding window to obtain the maximum and minimum response values ​​of the previous moment. Specifically, the formula for calculating the maximum response value of the previous moment is:

[0056] in, Max represents the maximum response value of the base layer at time t-1 after smoothing. n-1 equal to Max t-h-1 , representing the historical maximum response value within the sliding window time period; The sliding window size (number of frames / number of moments) is set to half the frame rate of the thermal camera by default, and can be adjusted in this embodiment according to the actual application scenario.

[0057] In this embodiment, if only the calibration between consecutive frames / times t-1 and t is considered, and the estimated value of the next frame is always based on the estimated result of the previous frame, the least squares estimation error will gradually accumulate during iterative optimization, and the algorithm may produce unacceptable deviations after long-term operation. Therefore, this application chooses to use the maximum / minimum response value obtained from the mathematical statistics of frame t-1 as a benchmark, and estimates the parameters at time t based on this, so that the least squares estimation is always performed sequentially and progressively within a sliding window of size 2, ensuring that the minimum luminance change is always maintained between adjacent frames. However, directly using the mathematical statistics of the previous frame also brings problems, such as outliers potentially causing excessively low image contrast. Therefore, a sliding window method is adopted: although this method cannot eliminate luminance changes, it can ensure that the contrast of the base layer of the previous frame is always maintained at a usable level and supports feature point extraction. The specific process includes: first, removing outlier pixels according to the 3σ criterion, statistically calculating the maximum / minimum response value of the remaining pixel values, so that the outlier values ​​enter a saturated state to enhance image contrast; then, smoothing the maximum / minimum response value based on the sliding window to further stabilize the image contrast.

[0058] Based on the above embodiments, in determining the feature matching set C t In other words, when executing step S205, the following method can be used: For the initial base layer of the previous time step... Edge extraction is performed to obtain the edge map from the previous time step; for each pixel in the initial base layer from the previous time step, distance field extraction is performed using the edge image from the previous time step to obtain the nearest edge distance corresponding to each pixel, thus obtaining the proxy texture map from the previous time step; the initial base layer at the current time step... Edge extraction is performed to obtain the edge map at the current time step; for each pixel in the initial base layer at the current time step, distance field extraction is performed using the edge image at the current time step to obtain the nearest edge distance corresponding to each pixel, so as to obtain the proxy texture map at the current time step; based on the proxy texture map at the previous time step and the proxy texture map at the current time step, feature matching is performed to obtain the feature matching set.

[0059] In other words, the specific process of determining the feature matching set is as follows: 1) Edge extraction.

[0060] High-quality edge extraction algorithms are crucial for robust edge flow tracking, requiring the following characteristics: high positioning accuracy, smooth and glitch-free edges, inter-frame repeatability, and high real-time performance. To meet these requirements, this application employs the TEED edge detector. This detector can extract smooth, continuous edges from an image with high accuracy, while also being robust to changes in image luminance, ensuring the repeatability of extracted edges across frames. It should be noted that other edge extraction methods can also be used; any edge extraction method that meets the above requirements is applicable. By performing edge extraction on the base layer, the corresponding edge image can be obtained. .

[0061] 2) LDT-KLT tracker.

[0062] Despite the good stability and repeatability of edges under illumination variations, edge flow tracking remains challenging. This is because edge images are binary content; the areas outside the edges are flat, blank, and lack texture. This characteristic easily causes feature trackers to get trapped in local optima, leading to tracking errors. To address this issue, this application proposes an LDT-KLT tracker, which achieves edge flow tracking based on the geometric texture provided by the edges.

[0063] By applying a distance transform, the distance from each pixel in the base layer to the nearest edge in the edge image is calculated, generating a distance field with the same size as the image. The value of each pixel in the distance field represents the distance from that pixel to the nearest edge, and its texture information is richer than the original binary image. Therefore, this application applies a distance transform to the binary edge image to construct an edge distance field as a proxy texture, encoding the image edge structure. This proxy texture inherits the robustness of edges to changes in luminance and expands the convergence window of edge flow tracing compared to the edge image. The formula for calculating the proxy texture map / distance field is:

[0064] in, Represents the edge image. This represents the Euclidean distance between pixel p in the base image and edge point q in the edge image.

[0065] Furthermore, because distance transformation is highly sensitive to edge distribution in the scene, especially when the thermal camera rotates, edges within the field of view move in and out rapidly, causing drastic changes in the distance field. This inter-frame texture inconsistency ultimately leads to edge flow tracking errors. These texture variations primarily originate from random region growth across the entire image; that is, changes at some edges propagate to the global region until they are blocked by other edges.

[0066] Therefore, to avoid explicit repeated searches, this application employs a random region growing technique to efficiently calculate the Euclidean distance field, thereby generating a DT image. In other words, after extracting the distance field and obtaining the nearest edge distance corresponding to each pixel, this application obtains an initial proxy texture image containing the nearest edge distances corresponding to each pixel. Subsequently, the method further includes: constraining the nearest edge distance of each pixel based on a predetermined distance threshold, so that the nearest edge distance is less than or equal to the predetermined distance threshold, thereby obtaining the final proxy texture map. The constraint formula is:

[0067] in, This represents a predetermined distance threshold used to limit the growth of random regions. If the threshold is too small, the texture information of the distance field will be insufficient; if the threshold is too large, the limiting effect will fail. Therefore, a reasonable threshold needs to be selected while ensuring the richness and stability of the distance field texture. Specifically, the predetermined distance threshold can be set to 20, which exhibits good and universal performance.

[0068] Subsequently, this application employs the Shi-Tomasi algorithm to select edge features within a constrained range field and utilizes LDT-KLT to achieve edge flow tracking. For the range field / proxy texture map of the previous time step... A certain feature The target of the LDT-KLT tracker is the distance field at the current time. Find pixels This minimizes the error of the constrained distance field, thereby determining the feature matching set C. t Among them, displacement The solution can be found by minimizing the error function, and the specific calculation formula is as follows:

[0069] The above formula minimizes a pixel-centered area with a size of [missing information]. The tracking is achieved by using the errors of image patches in two range fields. Larger image patches can adapt to large movements but reduce the accuracy of edge flow tracking; conversely, smaller image patches can achieve finer tracking but are prone to failure under large movements. Therefore, this method uses smaller patch sizes in the image pyramid for tracking, thus balancing accuracy and robustness. Finally, the edge feature matching results are filtered and verified using an essential matrix model.

[0070] The infrared image processing method in this embodiment improves robustness and accuracy: it provides infrared SLAM with a photometrically consistent infrared image sequence, significantly enhancing the stability of feature tracking and the accuracy of pose estimation, enabling reliable operation in dynamic thermal environments and all-weather conditions. This application achieves collaborative optimization of image quality: it effectively suppresses fixed pattern noise (FPN) at the detail layer, dynamically optimizes contrast at the base layer, and finally achieves a balance between noise reduction and detail enhancement after fusion, improving image signal-to-noise ratio and usability. This application overcomes circular dependencies: the innovative edge flow tracking based on constrained distance transform (LDT-KLT) method utilizes geometric edges insensitive to photometric changes for robust matching, providing reliable input for time parameter calibration and breaking the "chicken and egg" problem between photometric calibration and feature matching. This application enhances environmental adaptability: it enables the vision system to overcome extreme conditions such as strong light changes and darkness at night, significantly expanding the practical application range of visual SLAM in complex environments such as satellite denial.

[0071] Another embodiment of this application provides an infrared image processing device, such as... Figure 4 As shown, it includes: The separation module 11 is used to perform photometric parameter separation processing on the current raw infrared image captured by the infrared camera to obtain the initial detail layer and the initial base layer at the current moment. The denoising module 12 is used to perform denoising processing on the initial detail layer at the current time based on the target fixed pattern noise corresponding to the current time, so as to obtain the target detail layer at the current time. The determination module 13 is used to determine the target base layer at the current moment based on the initial base layer at the previous moment, the maximum response value at the previous moment, the minimum response value at the previous moment, the initial maximum response value at the current moment, the initial minimum response value at the current moment, the initial base layer at the current moment, and the feature matching set between the initial base layer at the previous moment and the initial base layer at the current moment, using an iterative optimization method. The fusion module 14 is used to perform image fusion processing based on the target detail layer and the target base layer at the current time to obtain the processed target infrared image.

[0072] In this embodiment, the separation module is specifically used to: perform photometric separation processing on the current original infrared image based on a nonlinear filter to obtain the initial detail layer and the initial base layer at the current moment.

[0073] In this embodiment, the infrared image processing device further includes a noise determination module. The noise determination module is used to: determine the target fixed pattern noise corresponding to the current moment based on the state type of the infrared camera at the current moment; specifically, the noise determination module is used to: determine the historical fixed pattern noise of the previous moment as the target fixed pattern noise at the current moment when the state type of the infrared camera is non-motion state; and determine the initial fixed pattern noise at the current moment based on the initial detail layer of the current moment and the initial detail layer of the previous moment when the state type of the infrared camera is motion state.

[0074] In this embodiment, the infrared image processing device further includes a historical response value determination module. The historical response value determination module is used to perform response value smoothing processing based on the predetermined sliding window corresponding to the previous moment, using the historical maximum response value and historical minimum response value corresponding to each historical moment within the predetermined sliding window, to obtain the maximum response value and the minimum response value of the previous moment.

[0075] In this embodiment, the infrared image processing device further includes a feature set determination module, which is used to: extract edges from the initial base layer at the previous time step to obtain the edge map at the previous time step; extract the distance field for each pixel in the initial base layer at the previous time step using the edge image at the previous time step to obtain the nearest edge distance corresponding to each pixel, thereby obtaining the proxy texture map at the previous time step; extract edges from the initial base layer at the current time step to obtain the edge map at the current time step; extract the distance field for each pixel in the initial base layer at the current time step using the edge image at the current time step to obtain the nearest edge distance corresponding to each pixel, thereby obtaining the proxy texture map at the current time step; and perform feature matching based on the proxy texture map at the previous time step and the proxy texture map at the current time step to obtain the feature matching set.

[0076] In this embodiment, the infrared imaging device further includes a constraint module, which is used to constrain the nearest edge distance of each pixel based on a predetermined distance threshold, so that the nearest edge distance is less than or equal to the predetermined distance threshold.

[0077] In this embodiment, the determination module is specifically used for: determining the target base layer at the previous time step based on the initial base layer, the maximum response value, and the minimum response value at the previous time step; determining the current base layer at the current time step based on the initial base layer, the initial maximum response value, and the initial minimum response value at the current time step; optimizing the initial maximum response value and the initial minimum response value at the current time step based on the current base layer, the target base layer at the previous time step, and the feature matching set to obtain the current maximum response value and the current minimum response value at the current time step, and determining whether a predetermined optimization condition is met; if the optimization condition is not met, based on... Given the initial base layer, the current maximum response value, and the current minimum response value at the current moment, the current base layer is redefined. Based on the redefined current base layer, the target base layer from the previous moment, and the feature matching set, the current maximum response value and the current minimum response value are optimized until a predetermined optimization condition is met. At this point, the current maximum response value is taken as the target maximum response value, and the current minimum response value is taken as the target minimum response value. Based on the initial base layer, the target maximum response value, and the target minimum response value at the current moment, the target base layer at the current moment is determined.

[0078] The apparatus in this application decomposes the original infrared image into an initial detail layer and an initial base layer by performing photometric parameter separation processing, thereby effectively decoupling detail information from photometric information. By performing target fixed-mode noise denoising processing on the initial detail layer, noise interference can be suppressed while preserving effective detail features, thus enhancing the detail of the infrared image. By introducing the base layer, response extrema, and feature matching set from the previous and current time moments, and using an iterative optimization method to determine the target base layer at the current time moment, the apparatus can fully utilize temporal information and global photometric features, ensuring that the base layer at the current time moment maintains a stable, continuous, and consistent photometric response with historical frames, avoiding inter-frame brightness abrupt changes and distortion, thereby ensuring the photometric consistency of the infrared image sequence. Finally, the target detail layer and the target base layer are fused, which can significantly enhance image details while maintaining overall photometric stability and inter-frame consistency, so that the processed infrared image has both high detail performance and high photometric fidelity, improving the imaging quality and reliability of the infrared imaging system in complex scenes. This approach achieves enhanced detail while suppressing noise (especially FPN), ensuring high photometric consistency in infrared image sequences. This provides stable and reliable data input for infrared visual SLAM, improving its positioning accuracy and environmental adaptability. Furthermore, by enhancing detail and ensuring photometric consistency in infrared images, it reduces processing deviations, improves image detail, and ultimately results in more stable infrared image processing.

[0079] Another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, implements the following method steps: Step 1: Perform photometric parameter separation processing on the current raw infrared image captured by the infrared camera to obtain the initial detail layer and the initial base layer at the current moment. Step 2: Based on the target fixed pattern noise corresponding to the current moment, perform denoising processing on the initial detail layer at the current moment to obtain the target detail layer at the current moment; Step 3: Based on the initial base layer of the previous time step, the maximum response value of the previous time step, the minimum response value of the previous time step, the initial maximum response value of the current time step, the initial minimum response value of the current time step, the initial base layer of the current time step, and the feature matching set between the initial base layer of the previous time step and the initial base layer of the current time step, the target base layer of the current time step is determined by iterative optimization. Step 4: Based on the target detail layer and the target base layer at the current moment, perform image fusion processing to obtain the processed target infrared image.

[0080] The specific implementation process of the above method steps can be found in the embodiments of any of the above infrared image processing methods, and will not be repeated here.

[0081] This application separates the photometric parameters of the original infrared image into an initial detail layer and an initial base layer, effectively decoupling detail and photometric information. By performing target-fixed-mode noise denoising on the initial detail layer, noise interference is suppressed while preserving effective detail features, thus enhancing the detail of the infrared image. By introducing the base layer from the previous and current timeframes, response extrema, and feature matching sets, and using an iterative optimization method to determine the target base layer at the current timeframe, temporal information and global photometric features are fully utilized. This ensures that the base layer at the current timeframe maintains a stable, continuous, and consistent photometric response with historical frames, avoiding abrupt brightness changes and distortion between frames, thereby guaranteeing photometric consistency in the infrared image sequence. Finally, image fusion of the target detail layer and the target base layer significantly enhances image detail while maintaining overall photometric stability and inter-frame consistency. This results in a processed infrared image with both high detail and high photometric fidelity, improving the imaging quality and reliability of the infrared imaging system in complex scenes. This approach achieves enhanced detail while suppressing noise (especially FPN), ensuring high photometric consistency in infrared image sequences. This provides stable and reliable data input for infrared visual SLAM, improving its positioning accuracy and environmental adaptability. Furthermore, by enhancing detail and ensuring photometric consistency in infrared images, it reduces processing deviations, improves image detail, and ultimately results in more stable infrared image processing.

[0082] Another embodiment of this application provides an electronic device, such as... Figure 5 As shown, it includes at least a memory 1 and a processor 2. The memory 1 stores a computer program, and the processor 2 performs the following method steps when executing the computer program in the memory 1: Step 1: Perform photometric parameter separation processing on the current raw infrared image captured by the infrared camera to obtain the initial detail layer and the initial base layer at the current moment. Step 2: Based on the target fixed pattern noise corresponding to the current moment, perform denoising processing on the initial detail layer at the current moment to obtain the target detail layer at the current moment; Step 3: Based on the initial base layer of the previous time step, the maximum response value of the previous time step, the minimum response value of the previous time step, the initial maximum response value of the current time step, the initial minimum response value of the current time step, the initial base layer of the current time step, and the feature matching set between the initial base layer of the previous time step and the initial base layer of the current time step, the target base layer of the current time step is determined by iterative optimization. Step 4: Based on the target detail layer and the target base layer at the current moment, perform image fusion processing to obtain the processed target infrared image.

[0083] The specific implementation process of the above method steps can be found in the embodiments of any of the above infrared image processing methods, and will not be repeated here.

[0084] This application separates the photometric parameters of the original infrared image into an initial detail layer and an initial base layer, effectively decoupling detail and photometric information. By performing target-fixed-mode noise denoising on the initial detail layer, noise interference is suppressed while preserving effective detail features, thus enhancing the detail of the infrared image. By introducing the base layer from the previous and current timeframes, response extrema, and feature matching sets, and using an iterative optimization method to determine the target base layer at the current timeframe, temporal information and global photometric features are fully utilized. This ensures that the base layer at the current timeframe maintains a stable, continuous, and consistent photometric response with historical frames, avoiding abrupt brightness changes and distortion between frames, thereby guaranteeing photometric consistency in the infrared image sequence. Finally, image fusion of the target detail layer and the target base layer significantly enhances image detail while maintaining overall photometric stability and inter-frame consistency. This results in a processed infrared image with both high detail and high photometric fidelity, improving the imaging quality and reliability of the infrared imaging system in complex scenes. This approach achieves enhanced detail while suppressing noise (especially FPN), ensuring high photometric consistency in infrared image sequences. This provides stable and reliable data input for infrared visual SLAM, improving its positioning accuracy and environmental adaptability. Furthermore, by enhancing detail and ensuring photometric consistency in infrared images, it reduces processing deviations, improves image detail, and ultimately results in more stable infrared image processing.

[0085] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. Those skilled in the art can make various modifications or equivalent substitutions to this application within the scope and nature of this application, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. An infrared image processing method, characterized in that, include: The photometric parameters of the current raw infrared image captured by the infrared camera are separated to obtain the initial detail layer and the initial base layer at the current moment. Based on the target fixed pattern noise corresponding to the current moment, the initial detail layer at the current moment is denoised to obtain the target detail layer at the current moment; Based on the initial base layer of the previous time step, the maximum response value of the previous time step, the minimum response value of the previous time step, the initial maximum response value of the current time step, the initial minimum response value of the current time step, the initial base layer of the current time step, and the feature matching set between the initial base layer of the previous time step and the initial base layer of the current time step, the target base layer of the current time step is determined by iterative optimization. Based on the target detail layer and the target base layer at the current moment, image fusion processing is performed to obtain the processed target infrared image.

2. The method as described in claim 1, characterized in that, The process of performing photometric parameter separation processing on the current raw infrared image captured by the infrared camera to obtain the initial detail layer and the initial base layer at the current moment specifically includes: The current original infrared image is subjected to photometric separation processing based on a nonlinear filter to obtain the initial detail layer and the initial base layer at the current moment.

3. The method as described in claim 1, characterized in that, Before performing denoising processing on the initial detail layer at the current moment based on the target fixed-mode noise corresponding to the current moment, the method further includes: determining the target fixed-mode noise corresponding to the current moment based on the state type of the infrared camera at the current moment, specifically including: When the infrared camera is in a non-motion state, the historical fixed pattern noise from the previous moment is determined as the target fixed pattern noise at the current moment. When the infrared camera is in motion mode, the initial fixed-mode noise at the current moment is determined based on the initial detail layer at the current moment and the initial detail layer at the previous moment.

4. The method as described in claim 1, characterized in that, Before determining the target base layer at the current moment, the method further includes: determining the maximum response value and the minimum response value at the previous moment, specifically including: Based on the predetermined sliding window corresponding to the previous moment, the response value is smoothed by using the historical maximum and minimum response values ​​corresponding to each historical moment within the predetermined sliding window, so as to obtain the maximum response value and the minimum response value of the previous moment.

5. The method as described in claim 1, characterized in that, Before determining the target base layer at the current moment, the method further includes determining the feature matching set, specifically including: Edge extraction is performed on the initial base layer of the previous time step to obtain the edge map of the previous time step; For each pixel in the initial base layer of the previous time step, the distance field is extracted using the edge image of the previous time step to obtain the nearest edge distance corresponding to each pixel, so as to obtain the proxy texture map of the previous time step. Edge extraction is performed on the initial base layer at the current time to obtain the edge map at the current time; For each pixel in the initial base layer at the current time, the distance field is extracted using the edge image at the current time to obtain the nearest edge distance corresponding to each pixel, so as to obtain the proxy texture map at the current time. Based on the proxy texture map of the previous time step and the proxy texture map of the current time step, feature matching is performed to obtain the feature matching set.

6. The method as described in claim 5, characterized in that, After performing distance field extraction to obtain the nearest edge distance corresponding to each pixel, the method further includes: The nearest edge distance of each pixel is constrained based on a predetermined distance threshold, so that the nearest edge distance is less than or equal to the predetermined distance threshold.

7. The method according to any one of claims 1-6, characterized in that, The target base layer at the current moment is determined using an iterative optimization approach, based on the initial base layer of the previous moment, the maximum response value of the previous moment, the minimum response value of the previous moment, the initial maximum response value of the current moment, the initial minimum response value of the current moment, the initial base layer of the current moment, and the feature matching set between the initial base layer of the previous moment and the initial base layer of the current moment. Specifically, this includes: Based on the initial base layer, the maximum response value, and the minimum response value of the previous time step, determine the target base layer of the previous time step. The current base layer is determined based on the initial base layer, the initial maximum response value, and the initial minimum response value at the current moment. Based on the current base layer at the current time, the target base layer at the previous time, and the feature matching set, the initial maximum response value and the initial minimum response value at the current time are optimized to obtain the current maximum response value and the current minimum response value at the current time, and to determine whether the predetermined optimization conditions are met. If the optimization conditions are not met, the current base layer is redefined based on the initial base layer, the current maximum response value, and the current minimum response value at the current moment. Then, based on the redefined base layer, the target base layer from the previous moment, and the feature matching set, the current maximum response value and the current minimum response value are optimized until the predetermined optimization conditions are met. At that point, the current maximum response value is taken as the target maximum response value and the current minimum response value is taken as the target minimum response value. The target base layer is determined based on the initial base layer at the current moment, the target maximum response value at the current moment, and the target minimum response value at the current moment.

8. An infrared image processing device, characterized in that, include: The separation module is used to perform photometric parameter separation processing on the current raw infrared image captured by the infrared camera to obtain the initial detail layer and the initial base layer at the current moment. The denoising module is used to denoise the initial detail layer at the current moment based on the target fixed pattern noise corresponding to the current moment, so as to obtain the target detail layer at the current moment. The determination module is used to determine the target base layer at the current moment based on the initial base layer at the previous moment, the maximum response value at the previous moment, the minimum response value at the previous moment, the initial maximum response value at the current moment, the initial minimum response value at the current moment, the initial base layer at the current moment, and the feature matching set between the initial base layer at the previous moment and the initial base layer at the current moment, using an iterative optimization method. The fusion module is used to perform image fusion processing based on the target detail layer and the target base layer at the current moment to obtain the processed target infrared image.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the infrared image processing method according to any one of claims 1-7.

10. An electronic device, characterized in that, It includes at least a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the infrared image processing method according to any one of claims 1-7 when executing the computer program in the memory.

Images