Infrared multi-mode composite imaging system based on APD detector

Abstract

The invention discloses an infrared multi-mode composite imaging system based on an APD detector, and the system achieves the quick conversion of three modes through a switchable optical assembly: in a passive imaging mode, the detector receives a thermal radiation signal emitted by a target; in the active gating mode, a laser is utilized to irradiate a target, and set target depth-of-field imaging is realized by controlling a time window of a detector; in the active 3D imaging mode, a time flight method is used, and distance information of a target is obtained by emitting laser and receiving reflection echoes of the laser. Meanwhile, a double-layer fusion framework based on adaptive weight is designed, spatial distance information in an active 3D imaging mode is coded to a chrominance channel in a YCbCr color space, heat radiation information and object contour details in a passive infrared mode are coded to a brightness channel, and an image fusing two-dimensional heat information and three-dimensional space information is output.

Description

Technical Field

[0001] This invention belongs to the field of infrared photoelectric detection and imaging technology, specifically involving a multi-mode, multi-dimensional information fusion imaging system based on an avalanche photodiode (APD) detector. It integrates and coordinates three technologies: passive infrared imaging, active gating imaging, and active 3D imaging, to achieve synchronous or selective acquisition of target radiation intensity, set depth of field details, and spatial distance information. Background Technology

[0002] Infrared imaging technology is a crucial means of target perception at night and in harsh environments. Passive infrared imaging systems, in particular, achieve imaging by detecting the infrared radiation emitted by the target itself, offering a unique advantage in terms of stealth. However, such systems have inherent limitations: firstly, they can only provide two-dimensional intensity information of the target, failing to acquire crucial distance and three-dimensional spatial information; secondly, under conditions of minimal temperature difference between the environment and the target, or in complete darkness, the imaging contrast is low, making it difficult to discern details. In complex real-world applications, two-dimensional thermal radiation intensity images alone are far from sufficient. For example, in the presence of smoke, rain, snow, or partial obstructions, there is an urgent need for a capability that can penetrate interference and clearly observe targets behind a set depth of field; simultaneously, in many applications, accurate three-dimensional ranging and spatial modeling of the target are key to achieving autonomous decision-making and precise operation.

[0003] To compensate for the shortcomings of passive imaging, active imaging technologies (such as laser-gated imaging and lidar) provide an effective supplement. Gated imaging, by controlling the laser illumination and the detector receiving window, can achieve selective imaging of targets within a set distance range; lidar can directly acquire high-precision 3D point cloud data. However, existing technical solutions are mostly single-function independent systems, or only possess one of the above capabilities. This leads to problems such as fragmented system functions, incomplete information dimensions, and difficulty in data fusion, preventing users from obtaining comprehensive perception of targets under a unified spatiotemporal reference.

[0004] Therefore, there is an urgent need in this field for a multifunctional integrated imaging system that can achieve the intrinsic fusion and collaborative operation of active and passive imaging modes on the same hardware platform, thereby simultaneously acquiring the radiation characteristics of the target, setting depth details and precise distance information, and fundamentally solving the problem that existing imaging schemes have a single information dimension and are difficult to adapt to complex application needs. Summary of the Invention

[0005] The purpose of this invention is to provide an infrared multi-mode composite imaging system based on an APD detector to address the problems of limited functionality and insufficient information dimensions in existing infrared imaging systems. This system not only integrates passive infrared imaging, active gating imaging, and active 3D imaging functions, but more importantly, it achieves deep fusion and synergy of information between different modes, thereby enhancing target detection and recognition capabilities in complex scenarios.

[0006] The solution to achieve the present invention is: an infrared multi-mode composite imaging system based on an APD detector, characterized in that it includes:

[0007] APD detector array is used to receive optical signals and convert them into electrical signals, which are then used to obtain digital signals.

[0008] The laser emitting unit is used to emit pulsed laser light towards the target;

[0009] And an FPGA control and processing unit connected to the APD detector array and the laser emitting unit, respectively;

[0010] The FPGA control and processing unit is configured to switch the system to one of the following three operating modes:

[0011] 1) Passive imaging mode: Control the APD detector array to receive the thermal radiation signal of the target itself and process it to generate a thermal radiation intensity image.

[0012] 2) Active Gated Imaging Mode: Controls the laser emitting unit to emit laser light, and adjusts the exposure delay time of the APD detector array. and points time Establish the distance between the foreground and the depth of field in the imaging process. and depth range The linear control relationship enables the detector window to be precisely matched in time and space with the laser echo within the set depth of field, thereby achieving gated imaging of the aforementioned set depth of field slice.

[0013] 3) Active 3D Imaging Mode: Control the laser emitting unit to emit laser, and based on the Time-of-Flight (TOF) method, convert the laser flight time into a linear ramp voltage through a high linearity integral circuit for quantization, thereby achieving high-precision target distance measurement and three-dimensional image generation to obtain a distance image.

[0014] Compared with the prior art, the present invention has the following significant advantages:

[0015] By integrating three imaging modes on a single system platform, a high degree of functional synergy and complementarity is achieved.

[0016] 1. Innovative Mode: The APD detector system platform enables the reuse and collaborative operation of three modes: passive infrared imaging, active gating imaging, and active 3D imaging, fundamentally solving the problem of functional limitation.

[0017] 2. Technological advancement: In active 3D imaging mode, an innovative ranging scheme with high linear ramp voltage quantization of time of flight is adopted, and by limiting the sampling voltage to the linear range (1V~3V), the circuit cost is optimized while ensuring high accuracy.

[0018] 3. Powerful processing capabilities: It not only integrates an adaptive image processing pipeline to improve the quality of single-mode images, but also introduces an information fusion module, which can intuitively fuse the thermal radiation characteristics of the target with its three-dimensional spatial position, generating composite images with richer information dimensions that are easier for humans to interpret or machines to recognize. Attached Figure Description

[0019] Figure 1 This is a system block diagram of the present invention.

[0020] Figure 2 This is a hardware block diagram of the present invention.

[0021] Figure 3 This is a schematic diagram illustrating the timing principle of the active gating imaging mode in this invention.

[0022] Figure 4 This is a schematic diagram illustrating the timing working principle of the active 3D imaging mode in this invention.

[0023] Figure 5 This is a preferred embodiment of an integral circuit design for generating ramp voltage in the active 3D imaging mode of the present invention.

[0024] Figure 6 This is a schematic diagram of the algorithm framework of the active-passive fusion module of the present invention.

[0025] Figure 7 Figure 2 shows a comparison of the target processed images received by the APD detector in passive imaging mode in Embodiment 2 of the present invention. Figure (a) is a comparison of the original image captured by the camera and the image after contrast-limited adaptive histogram equalization (CLAHE) processing. Figure (b) is a comparison of the effect before and after single-point correction processing. Figure (c) is a comparison of the effect before and after two-point correction. Figure (d) is a comparison of the effect before and after blind pixel replacement. Figure (e) is a comparison of the white-hot / black-hot and pseudo-color effect.

[0026] Figure 8 The images shown are from an embodiment 3 of the present invention, illustrating the imaging test of the APD detector in active gating mode. Figures (a) and (b) show different integration times. Comparison of imaging depth of field range: Figure (a) Integration time Figure (b) shows the integration time. Figures (c) and (d) show different exposure delay times. The following is a comparison of the imaging foreground distance: The laser emission time in Figure (c) is earlier than that in Figure (d). .

[0027] Figure 9 The images shown are the imaging test images of the APD detector in active 3D imaging mode in Embodiment 4 of the present invention. Figures (a), (b), and (c) respectively show the passive infrared grayscale image, active 3D distance grayscale image, and pseudo-color conversion image in the same scene.

[0028] Figure 10 Figure 5 shows the fusion information diagrams of the system in active-passive fusion mode in Embodiment 5 of the present invention. Figure (a) is a feature-guided adaptive weight map, divided into a low-frequency structure weight map and a high-frequency detail weight map. Figure (b) is the image after the passive infrared image is layered, divided into a low-frequency layer and a high-frequency layer. Figure (c) is the image after the active 3D range map is layered, also divided into low-frequency and high-frequency layers. Figure (d) is the final fused image. Figure (e) is the final color fused image after color space synthesis of the fused image and the pseudo-color conversion map. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the accompanying drawings.

[0030] Combination Figure 1 An infrared multi-mode composite imaging system based on an APD detector, comprising:

[0031] APD detector array is used to receive optical signals and convert them into electrical signals, which are then used to obtain digital signals.

[0032] The laser emitting unit is used to emit pulsed laser light towards the target;

[0033] In addition, there is an FPGA control and processing unit that is connected to the APD detector array and the laser emission unit, respectively.

[0034] The FPGA control and processing unit is configured to switch the system to one of the following three operating modes:

[0035] 1) Passive Imaging Mode: The APD detector array is controlled to receive the thermal radiation signal of the target itself and process it to generate a thermal radiation intensity image, as follows:

[0036] The thermal radiation emitted by the target itself is received by the lens of the APD detector array and the signal is focused onto the surface of the APD detector. Based on the photoelectric conversion characteristics of mercury cadmium telluride material, the detector converts the received light signal into an electrical signal, and the electrical signal is amplified and amplified to obtain an analog electrical signal. The analog signal is then converted into a digital signal, which enters the signal processing module for processing to obtain a thermal radiation intensity image, which is finally displayed on the screen in real time.

[0037] 2) Active Gated Imaging Mode: Controls the laser emitting unit to emit pulsed laser light to illuminate the target, and adjusts the detector exposure delay time. and points time Establish the distance between the foreground and the depth of field in the imaging process. and depth range The linear control relationship enables the detector window to be precisely matched in time and space with the laser echo within the set depth of field, thereby achieving gated imaging of the aforementioned set depth of field slice.

[0038] 3) Active 3D Imaging Mode: The laser emitting unit is controlled to emit pulsed lasers to illuminate the target. Based on the Time-of-Flight (TOF) method, the laser flight time is converted into a linear ramp voltage through a high-linearity integral circuit for quantization, thereby achieving high-precision target distance measurement and three-dimensional image generation to obtain a distance image.

[0039] Combination Figure 2 An infrared multi-mode composite imaging system based on an APD detector, the specific hardware composition of which is as follows:

[0040] The hardware design mainly consists of a motherboard and an interface board. The motherboard comprises three main modules: a power supply module, a signal processing module, and a driver module. The power supply module is responsible for powering the entire system. The driver module provides drive signals to the detector and controls the pulsed laser; it primarily powers the detector and performs analog-to-digital conversion on the detector's output image. The signal processing module processes the image. The interface board is installed on the corresponding Dewar pins of the APD detector to connect the detector to the motherboard.

[0041] Combination Figure 3 The working principle of the active gating imaging mode in the infrared multi-mode composite imaging system based on the APD detector is as follows:

[0042] When the system is switched to active gating imaging mode, the laser emitting unit is controlled to emit a laser, and the detector exposure delay time is adjusted. and points time Establish the distance between the foreground and the depth of field in the imaging process. and depth range The linear control relationship enables precise spatiotemporal matching between the detector window and the laser echo within the set depth of field, thereby achieving gated imaging of the aforementioned set depth of field slice, as detailed below:

[0043] The FPGA control and processing unit sends a trigger signal to the APD detector array, which indicates that the APD detector array should begin exposure integration after a fixed internal delay in order to receive the echo signal of the pulsed laser.

[0044] Controlling the exposure delay time of the laser before the APD detector array begins exposure Emit pulsed laser to target; control detector integration time To determine the depth range of the target being detected.

[0045] Through precise adjustment and These two timing parameters are used to select and control the imaging depth of field.

[0046] Furthermore, adjust the exposure delay time. Foreground distance of the selected target depth of field Adjusting the integral time To determine the depth range of the target depth of field. It satisfies the formula:

[0047] ,

[0048] ,

[0049] in, This is the speed of light propagation.

[0050] Combination Figure 4 The working principle of the active 3D imaging mode in the infrared multi-mode composite imaging system based on the APD detector is as follows:

[0051] When the system is switched to active 3D imaging mode, the laser emitting unit is controlled to emit a laser. Based on the Time-of-Flight (TOF) method, a high-linearity integral circuit converts the laser flight time into a linear ramp voltage for quantization, achieving high-precision target distance measurement and 3D image generation, resulting in a distance image, as detailed below:

[0052] 1) Control the laser to emit pulsed laser light toward the target.

[0053] 2) A high-linearity integrator circuit is started simultaneously. This circuit charges or discharges the capacitor through a constant current source, thereby generating a definite slew rate that varies linearly with time. ramp voltage It converts time quantities into voltage quantities for high-precision measurement.

[0054] 3) When the APD detector array receives the reflected echo of the pulsed laser, sample the current voltage and record it as the sampled voltage. To ensure the accuracy of flight time measurement, the linear ramp voltage generated by the integrated voltage is used. The effective operating range needs to have good linearity, therefore the ramp voltage... The sampling range is limited to a high linearity range of 1V to 3V to minimize nonlinearity error at the hardware level.

[0055] 4) Based on the sampling voltage With ramp voltage The linear proportional relationship is used to calculate the time of flight of the laser pulse. And based on this, the distance to the target is obtained. Flight time The formula is as follows:

[0056] ,

[0057] in, The initial value for the ramp voltage sampling. This indicates the slewing rate.

[0058] The formula for TOF distance measurement is:

[0059] ,

[0060] in, It is the speed of light.

[0061] Combination Figure 5 The preferred embodiment of the integrator circuit design for generating the ramp voltage in the active 3D imaging mode in the infrared multi-mode composite imaging system based on the APD detector is as follows:

[0062] The high linearity integrator circuit includes an N-channel enhancement-mode MOSFET and a capacitor. ,resistance First capacitor Reference voltage source, operational amplifier, second capacitor Switching diode and first resistor .

[0063] A constant voltage passes through the first resistor The switching diode is connected in series with the input terminal of the reference voltage source, and the output terminal of the reference voltage source is connected to the resistor. ,capacitance Series grounding, capacitor Simultaneously, an N-channel enhancement-mode MOSFET and the non-inverting input of an operational amplifier are connected in parallel, and a first capacitor is connected between the output and input pins of the reference voltage source and its own ground terminal. Second capacitor These two capacitors are connected in parallel with the output of the operational amplifier and are connected to ground together.

[0064] The operational amplifier is configured as a voltage follower, and a capacitor is connected to its non-inverting input. The output terminal provides a low-impedance voltage to ensure the capacitor... The voltage across the capacitor is the same as the output voltage of the operational amplifier; One end is connected to ground, and the other end is connected to a resistor. Connected in series to the output of the reference voltage source; a constant current flows through the resistor. Flow to capacitor This makes the capacitor The voltage gradually increases, i.e., the ramp voltage; the charging current flows through the resistor. The magnitude of the charging current is determined by the resistance. The operational amplifier's output voltage is determined by its reference voltage; the operational amplifier's output voltage follows the capacitor. The operational amplifier controls the output voltage to ensure the capacitor remains constant, and drives the ground terminal of the reference voltage source. The relationship between the voltage across the terminals and the reference voltage is stable.

[0065] The slope of the ramp voltage is calculated using the following formula, which determines the values ​​of the resistor and capacitor based on the integration time:

[0066] Because the current is constant, the capacitor Voltage over time Linear increase:

[0067] ,

[0068] In the above formula, Indicates capacitance The voltage at both ends, Indicates resistance The resistance value, Indicates capacitance The capacitance value.

[0069] Principle of slope slewing rate calculation:

[0070] ,

[0071] in, Indicates the reference voltage.

[0072] time constant for:

[0073] ,

[0074] Current flowing through the circuit for:

[0075] ,

[0076] Then the slew rate for:

[0077] .

[0078] Combination Figure 6 The infrared multi-mode composite imaging system based on the APD detector includes an image processing module and an information fusion module in its FPGA control and processing unit, as detailed below:

[0079] 1. The image processing module performs non-uniformity correction, blind pixel detection and replacement, contrast enhancement, and pseudo-color processing on the digital signals acquired by the APD detector array to improve image quality and obtain an infrared image, as detailed below:

[0080] 1) In the image processing module, non-uniformity correction includes single-point correction and two-point correction. Single-point correction compensates for image offset by acquiring uniform reference scene data; two-point correction performs precise linear correction of the detector response based on the gain and offset coefficient obtained from high and low temperature calibration.

[0081] 2) In the image processing module, the two steps of blind pixel detection and replacement are as follows: in the blind pixel detection stage, the faulty pixels are identified by the neighborhood difference method; in the blind pixel compensation stage, median filtering (flat area) or directional interpolation (edge ​​area) is adaptively selected to repair the data based on the gradient characteristics of the region where the blind pixel is located.

[0082] 3) In the image processing module, the contrast enhancement uses the Contrast-Limited Adaptive Histogram Equalization (CLAHE) algorithm: the image is divided into multiple continuous image blocks, and the histograms of the local image blocks are cropped, restricted, and redistributed to effectively suppress noise amplification while enhancing image details.

[0083] 4) The image processing module can also perform display enhancement processing on the generated grayscale image to adapt to different observation needs. The FPGA control and processing unit integrates a pseudo-color mapping function, which performs real-time color mapping through a lookup table and supports black-and-white thermal and various pseudo-color display schemes.

[0084] 2. The information fusion module designs a two-layer fusion framework based on adaptive weights, calling infrared images and distance images for layered fusion to obtain a fused image. Finally, the fused image is synthesized in the YCbCr color space, outputting a multi-information fused image that can simultaneously display target spatial dimensional information and thermal information, realizing deep fusion of two-dimensional thermal information and three-dimensional spatial information, as detailed below:

[0085] 1) Perform feature analysis and weight generation, extracting physically meaningful features from the range image and infrared image respectively, namely the salience of the structure and the importance of details, and calculate the spatial weight map to guide the fusion accordingly, as follows:

[0086] 1-1) To effectively fuse depth information and spatial structure features in active 3D images, this invention employs a structure weight map generation method based on multi-scale local consistency. To quantify the reliability of each pixel, a depth confidence model based on local consistency is first established for any pixel. Its distance confidence for:

[0087] ,

[0088] in Represents pixels The original distance value at that location, Indicates The minimum distance value within the local neighborhood of the center. This represents the maximum distance value within the neighborhood, used as a normalization benchmark. The variance of the distance values ​​within the neighborhood reflects the stability of the distance measurement. This is the noise suppression coefficient, which controls the strength of the influence of the variance term on the confidence level.

[0089] Distance confidence It not only considers the absolute magnitude of the distance value, but also introduces a local consistency constraint, which reduces the confidence level in regions where the distance value changes drastically, while increasing the confidence level in regions where the distance value is smooth and continuous.

[0090] In the original active 3D image, near-field targets exhibit lower grayscale values, while background areas exhibit higher grayscale values. To generate a weight distribution that conforms to visual saliency, the depth values ​​need to be inverted.

[0091] ,

[0092] in, This represents the significance weight value after reversal.

[0093] Calculating confidence at a single scale is difficult to simultaneously preserve details and suppress noise. To address this, this invention proposes a multi-scale adaptive fusion method that comprehensively utilizes local structural information at different scales.

[0094] For scale parameters Scale parameter set , Represent the Kth scale, define the scale. Depth confidence for:

[0095] ,

[0096] in, For scale The local depth variance below, corresponding to a neighborhood size of Scale-adaptive attenuation coefficient ,in The constant decay factor It is an exponential consistency constraint function that provides smooth transition characteristics.

[0097] Attenuation coefficient With scale Inversely proportional, that is This design ensures that For small scale ( hour, Larger size, sensitive to noise, suitable for preserving details. For large scale ( )hour, Smaller size, robust to noise, suitable for smooth regions.

[0098] Final structure weight diagram The following is obtained through weighted fusion of multi-scale confidence levels:

[0099] ,

[0100] In the formula For the fusion weights at each scale, satisfy And usually set This is to emphasize the contribution of small-scale information.

[0101] To ensure that the weight values ​​are within the standard range and have good numerical characteristics, final normalization is performed:

[0102] ,

[0103] Normalized weight map It possesses three core characteristics. First, target enhancement, with the average weight of the foreground region significantly higher than that of the background; second, continuous gradation, with smooth spatial changes in weight values, avoiding hard boundaries; and third, noise suppression, where weight values ​​automatically decrease in regions where depth measurement is unstable. This makes it a reliable structural representation to guide subsequent processing.

[0104] 1-2) To comprehensively capture thermal boundary features, we employ a multi-directional gradient analysis method. First, we calculate the infrared image... gradient vector :

[0105] ,

[0106] Discrete approximation using the Sobel operator:

[0107] ,

[0108] ,

[0109] in , All are Sobel operators. , This represents the relative coordinate offset of the element in the convolution. This represents an approximate gradient of the image in the horizontal direction. This represents an approximate gradient of the image in the vertical direction.

[0110] ,

[0111] gradient magnitude for:

[0112] ,

[0113] Gradient direction angle for:

[0114] ,

[0115] Since gradient magnitude alone cannot distinguish between the true thermal boundary and the noise response, this invention proposes a gradient significance metric. Taking into account gradient magnitude, directional consistency, and local contrast:

[0116] ,

[0117] in, Local contrast is represented by the ratio of the standard deviation to the mean of the gradient magnitude within a local window, reflecting the local contrast. , Represent it as a constant approaching zero to avoid division by zero. Indicates local window centered The standard deviation of the internal gradient magnitude.

[0118] ,

[0119] in, They represent respectively with local window centered The mean of the inner gradient magnitude, To avoid division by zero for constants that approach zero.

[0120] This indicates consistency in local gradient directions:

[0121] ,

[0122] This value represents the number of pixels within the window. It is close to 1 when the gradient directions are consistent and close to 0 when the directions are random, effectively distinguishing between organized edge structures and random noise.

[0123] The mapping from gradient significance to weights employs a nonlinear function to enhance the weights of significant hot boundaries and suppress insignificant regions:

[0124]

[0125] in, This is a steepness control parameter used to control the steepness of the mapping curve; For significance measurement The global mean is used to adaptively adjust the threshold. This represents the detail weights of layer 0 (original scale), i.e., the detail weights of non-significant regions.

[0126] Considering that thermal boundaries may exist at different scales, this invention employs a multi-scale fusion strategy. At the original scale... and downsampling scale (Usually a 2:1 downsampling) weights are calculated separately. and Then Upsampled to the original size, the final detail weights are obtained:

[0127]

[0128] in, This is the scale fusion coefficient, typically set to 0.6-0.8, giving more weight to the original scale.

[0129] 2) Perform multi-scale decomposition and hierarchical fusion on the images in the two modes, decompose the images into different frequency bands, and use weight maps to implement adaptive fusion rules in the low-frequency layer and high-frequency layer. The low-frequency layer contains most of the basic structure of the image, and the high-frequency layer contains most of the detailed texture of the image.

[0130] 2-1) The two-level decomposition of an image is based on multi-resolution analysis theory, which decomposes the image into a low-frequency component carrying the main energy distribution and a high-frequency component containing detailed textures. For the input image... Its decomposition process can be represented as:

[0131]

[0132] Among them, low frequency components High-frequency components reflect the overall structure and illumination changes of the image. It reflects details such as the edges and textures of the image.

[0133] Guided filtering is used to extract low-frequency components while preserving edges. For the input image... If this is also used as the guiding image, then the guiding filter will be applied in each local window. Internal Hypothesis Output It is a guide image Linear transformation:

[0134]

[0135] in and All are in the window The linear coefficients remain constant within the window. These coefficients are obtained by minimizing the reconstruction error and regularization term within the window, with the cost function being:

[0136]

[0137] here, For regularization parameters, used To prevent excessive smoothness, the smoothness of the filter needs to be controlled. By solving the above optimization problem, the coefficients can be obtained. and Analytical solution: The high-frequency components are obtained by subtracting the low-frequency components from the original image.

[0138]

[0139]

[0140] in, and Guide images In the window Mean and variance within, For the input image The mean within the window, This represents the number of pixels within the window.

[0141] Since each pixel is covered by multiple windows, its final output value The mean of the results calculated for all windows containing that pixel:

[0142]

[0143] During filtering, the window radius controls the range of spatial smoothing, and the regularization parameter... The strength of edge retention control: smaller Values ​​make the filter more inclined to preserve the edges of the guiding image; larger values... The value produces a stronger smoothing effect.

[0144] Obtaining low-frequency components with edge preservation After that, high-frequency components The result is obtained by calculating the residual between the original input image and the low-frequency component:

[0145]

[0146] 2-2) Low-frequency fusion method

[0147] The low-frequency component contains the main energy and structural information of the image, and its fusion strategy needs to incorporate the background information of the infrared image while maintaining the structural integrity of the laser range image.

[0148] Let the low-frequency component of the laser range image be... The low-frequency component of the infrared image is The low-frequency structure weight is Low-frequency fusion employs a weighted average model, resulting in a low-frequency fused image. for:

[0149]

[0150] To further improve the fusion quality and enhance the structural contrast of the fused image, local contrast enhancement is introduced. First, the local window is calculated. mean within and standard deviation :

[0151]

[0152]

[0153] Enhanced low-frequency components for:

[0154]

[0155] in and The target contrast parameter.

[0156] 2-3) High-frequency fusion method

[0157] High-frequency components contain edge and texture details of the image. The fusion objective is to highlight significant thermal boundaries and detail features. This is based on a high-frequency detail weight map. A feature selection fusion strategy is adopted:

[0158]

[0159] The high-frequency components of the laser distance image are: The high-frequency components of the infrared image are , The threshold for thermal boundary detection is set. To enhance the fusion effect, a local energy maximization selection mechanism is introduced. The local energies of the two high-frequency components are calculated:

[0160]

[0161]

[0162] in, This indicates that the high-frequency components of the laser range map are in the range of... local window centered energy, This represents the local energy of the high-frequency components in the neighborhood of an infrared image. This represents the relative coordinate offset.

[0163] The improved fusion rules are as follows:

[0164]

[0165] in, This represents the enhanced high-frequency components. This selection mechanism ensures that infrared high-frequency details are preferentially used in regions with significant thermal boundaries, while richer detail information is selected in other regions based on local energy.

[0166] 2-4) Image Fusion Generation

[0167] After fusing the low-frequency and high-frequency components, the structure of each layer of the image remains unchanged. The final fused image is reconstructed by adding the two layers together. :

[0168]

[0169] 3) The system performs a channel synthesis step based on the YCbCr color space. The fused image is used as the luminance component (Y), and the chromaticity component (Cb, Cr) extracted from the pseudo-color 3D distance image after conversion to the YCbCr space is used as the color information. A new YCbCr image is generated by directly replacing channels: its luminance is entirely determined by the fused image, ensuring clear visibility of thermal targets and outlines. Its chromaticity is entirely determined by the distance information, forming a continuous color code from red to purple. Finally, the image is converted back to the RGB space for output on the host computer.

[0170] Example 1

[0171] The original infrared image was taken indoors and its size is [size missing]. The image is input into the algorithm module, and the host computer sends functions such as single-point correction, two-point correction, blind pixel replacement, and contrast enhancement to process the image.

[0172] Figure 7 (a) shows the original infrared image and the image after CLAHE processing. It can be seen that the original image has low contrast and blurred details, while the output image processed by the CLAHE algorithm has significantly improved contrast, enhanced details in key areas such as object outlines, and a more distinct sense of image layering.

[0173] Single-point correction of the image is mainly used to compensate for detector offset noise. A comparison of the effects before and after correction is shown below. Figure 7 As shown in (b), the overall uniformity of the image has been initially improved, especially the fixed pattern noise in the edge region has been effectively suppressed.

[0174] Based on single-point correction, a further two-point correction is performed to correct the gain differences of each pixel. The comparison of the effects before and after correction is as follows: Figure 7 As shown in (c), after two-point correction, the non-uniformity of the image is completely eliminated, and the image is clean and uniform, providing a high-quality base image for subsequent processing.

[0175] Blind pixel processing is performed on the image after non-uniformity correction. A comparison of the effects before and after processing is shown below. Figure 7 As shown in (d), the blind pixel replacement algorithm of this invention can accurately identify and compensate for faulty pixels in an image. While smoothing flat areas, it can effectively preserve detailed structures such as object edges, avoiding edge blurring or loss of detail that may occur with traditional methods.

[0176] Meanwhile, the system can perform pseudo-color mapping on the processed high-quality grayscale images. Figure 7 (e) Demonstrates the display effects of the same scene in white-hot, black-hot, and pseudo-color modes to adapt to different observation habits and application scenarios, enhance the intuitiveness of the images, and provide users with diverse visual analysis tools.

[0177] Example 2

[0178] The system's gated imaging functionality was verified under indoor conditions. Timing parameters were precisely controlled using an FPGA to verify its ability to control imaging depth of field.

[0179] Controlling different integration times To control the depth of field range of the image Maintain exposure delay time. The width of the detector's receiving window remains unchanged; however, the integration time is adjusted to control its width. Figure 8 (a) is the imaging result when the integration time is set to 1TMC. At this time, the depth of field is narrow and the system can only clearly image targets within the set distance range. Figure 8 (b) Imaging results with the integration time increased to 2 TMC show a significantly wider depth of field, with background details at greater distances being received and clearly rendered by the system. This result verifies the improved depth of field. With integration time The property of being directly proportional, that is .

[0180] Controlling different exposure delay times To control the foreground distance Maintain points for a certain period of time. The exposure delay time remains unchanged; the activation time of the detector is controlled by altering this delay. The step size is adjusted to control the gating distance, enabling imaging of the target at different gating distances. Figure 8 (c) and Figure 8 (d) presents the comparison results. Among them, Figure 8 (c) The laser emission time compared to Figure 8 (d) brought forward As can be seen from the imaging results, Figure 8 (c) The entire sharp imaging depth window relative to Figure 8 (d) The shift forward indicates a change in the depth of field of the primary target observed by the system. This result verifies the target foreground distance. With exposure delay time The property of being directly proportional, that is .

[0181] This embodiment fully verifies that the present invention can precisely adjust the exposure delay time via FPGA in active gating imaging mode. and points time This allows for the measurement of the foreground distance in the imaging depth of field. and depth range The flexible and effective control achieved the expected technical results.

[0182] Example 3

[0183] The system's active 3D imaging mode was functionally verified under indoor conditions. Timing parameters were precisely controlled via FPGA to ensure that the ramp voltage begins integration before laser pulse emission and reaches the 1V linear operating region precisely at the time of laser pulse emission. This ensures that the laser can be emitted, propagated, reflected, and echoed within the optimal operating range of the ramp voltage linearity, thereby minimizing nonlinear errors and achieving high-precision time-of-flight measurement. Figure 9 (a) is a grayscale image of the scene. This image can only reflect the target's own thermal radiation information. Although it can clearly show the target's temperature difference and outline, it lacks the target's distance dimension information and cannot provide the target's spatial position relationship. Figure 9 (b) shows the distance grayscale image generated in active 3D imaging mode. It can be seen that the closer the object is, the lower its grayscale value and the darker its color, while the farther the object is, the higher its grayscale value and the lighter its color. Figure 9 (c) This diagram shows the pseudo-color conversion effect of the grayscale distance map in the active 3D imaging mode. It is clearly observed that objects closer to the detector appear in red tones, while objects farther away appear in blue tones. The generation mechanism of this image is completely unrelated to the thermal radiation characteristics of the target; its pixel color values ​​are determined solely by the distance of each point on the target surface relative to the detector, thus intuitively displaying the three-dimensional spatial structure of the scene. The test results of this embodiment effectively verify the active 3D imaging function of the system. Through comparison... Figure 9 Figures (a), (b), and (c) clearly demonstrate that the system has successfully expanded its single two-dimensional thermal radiation sensing capability into a dual capability encompassing both thermal characteristic sensing and three-dimensional spatial sensing. The generation of distance images proves the feasibility and effectiveness of the time-of-flight measurement scheme based on linear ramp voltage in a practical system.

[0184] Example 4

[0185] This embodiment verifies the system's active and passive image fusion capabilities. The test was conducted in a standard indoor environment, with the system simultaneously operating in both passive and active 3D imaging modes. It collected thermal radiation intensity and three-dimensional spatial distance information for the same scene, respectively, and then effectively integrated the information using an image fusion algorithm.

[0186] First, obtain the feature-guided adaptive weight map of the scene, such as... Figure 10 As shown in (a), the image is divided into a low-frequency structure weight map and a high-frequency detail weight map. It can be seen that the low-frequency structure weight map focuses more on the structure and contour of the object, while the high-frequency detail map is more sensitive to the texture details of the object. Subsequently, the system layers the passive infrared image and the active 3D distance image, as shown in (a). Figure 10As shown in (b) and (c), the low-frequency layer focuses more on the structural information of the object, while the high-frequency layer displays more detailed textures. The system then fuses the low-frequency and high-frequency layers based on their respective weight maps, generating a low-frequency fused image and a high-frequency fused image. The two images are then added together to generate the final fused image, as shown below. Figure 10 As shown in (d). Finally, the fused image and the pseudo-color conversion image are fused using a color space to synthesize the final color fused image, as shown. Figure 10 As shown in (e).

[0187] The fusion effect is thus clearly visible. In the final fused image 10(e), the thermal texture details of the object are preserved more completely, and the contour edges are effectively retained, making the target clearer and more complete. Moreover, the final image retains both the thermal characteristic information of the original grayscale image and the spatial information of the distance image. Observers can judge the temperature of the target by the brightness of the image, and judge the distance of the target by the color difference of the image. For example, bright red areas in the image represent warm, nearby targets, while dark blue areas represent cold, distant targets. This embodiment successfully verifies the actual effect of the system's information fusion module, realizing a leap from separate multi-source information to unified perceptual expression.

Claims

1. An infrared multi-mode composite imaging system based on an APD detector, characterized in that, include: APD detector array is used to receive optical signals and convert them into electrical signals, which are then used to obtain digital signals. The laser emitting unit is used to emit pulsed laser light towards the target; And an FPGA control and processing unit connected to the APD detector array and the laser emitting unit, respectively; The FPGA control and processing unit is configured to switch the system to one of the following three operating modes: 1) Passive imaging mode: Control the APD detector array to receive the thermal radiation signal of the target itself and process it to generate a thermal radiation intensity image; 2) Active Gated Imaging Mode: Controls the laser emitting unit to emit laser light, and adjusts the exposure delay time of the APD detector array. and points time Establish the distance between the foreground and the depth of field in the imaging process. and depth range The linear control relationship enables the detector window to be precisely matched with the laser echo within the set depth of field in time and space, thereby achieving gating imaging of the above-mentioned set depth of field slice. 3) Active 3D Imaging Mode: Control the laser emitting unit to emit laser, and based on the Time-of-Flight (TOF) method, convert the laser flight time into a linear ramp voltage through a high linearity integral circuit for quantization, thereby achieving high-precision target distance measurement and three-dimensional image generation to obtain a distance image.

2. The infrared multi-mode composite imaging system based on an APD detector according to claim 1, characterized in that, When the system is switched to passive imaging mode, the APD detector array is controlled to receive the thermal radiation signal of the target itself and process it to generate a thermal radiation intensity image, as follows: The thermal radiation emitted by the target itself is received by the lens of the APD detector array and the signal is focused onto the surface of the APD detector. Based on the photoelectric conversion characteristics of mercury cadmium telluride material, the detector converts the received light signal into an electrical signal, and the electrical signal is amplified and amplified to obtain an analog electrical signal. The analog signal is then converted into a digital signal, which enters the signal processing module for processing to obtain a thermal radiation intensity image, which is finally displayed on the screen in real time.

3. The infrared multi-mode composite imaging system based on an APD detector according to claim 1, characterized in that: When the system is switched to active gating imaging mode, the laser emitting unit is controlled to emit a laser, and the detector exposure delay time is adjusted. and points time Establish the distance between the foreground and the depth of field in the imaging process. and depth range The linear control relationship enables precise spatiotemporal matching between the detector window and the laser echo within the set depth of field, thereby achieving gated imaging of the aforementioned set depth of field slice, as detailed below: The FPGA control and processing unit sends a trigger signal to the APD detector array, which indicates that the APD detector array will begin exposure integration after a fixed internal delay in order to receive the echo signal of the pulsed laser. Controlling the exposure delay time of the laser before the APD detector array begins exposure Emit pulsed laser to target; control detector integration time To determine the depth range of the target being detected; Through precise adjustment and These two timing parameters are used to select and control the imaging depth of field.

4. The infrared multi-mode composite imaging system based on an APD detector according to claim 3, characterized in that: Adjusting the exposure delay time Foreground distance of the selected target depth of field Adjusting the integration time To determine the depth range of the target depth of field. It satisfies the formula: ， ， in, This is the speed of light propagation.

5. The infrared multi-mode composite imaging system based on an APD detector according to claim 1, characterized in that, When the system is switched to active 3D imaging mode, the laser emitting unit is controlled to emit a laser. Based on the Time-of-Flight (TOF) method, a high-linearity integral circuit converts the laser flight time into a linear ramp voltage for quantization, achieving high-precision target distance measurement and 3D image generation, resulting in a distance image, as detailed below: 3-1) Control the laser to emit pulsed laser light towards the target; 3-2) A high-linearity integrator circuit is started simultaneously. This circuit charges or discharges the capacitor through a constant current source, thereby generating a definite slew rate that varies linearly with time. ramp voltage This converts time quantities into voltage quantities for high-precision measurement. 3-3) When the APD detector array receives the reflected echo of the pulsed laser, sample the current voltage and record it as the sampled voltage. To ensure the accuracy of flight time measurement, the linear ramp voltage generated by the integrated voltage is used. The effective operating range needs to have good linearity, therefore the ramp voltage... The sampling range is limited to a high linearity range of 1V to 3V to minimize nonlinearity error at the hardware level; 3-4) Based on the sampling voltage With ramp voltage The linear proportional relationship is used to calculate the flight time of the laser pulse. And based on this, the distance to the target is obtained. Flight time The formula is as follows: ， in, The initial value for sampling the ramp voltage. Indicates the slewing rate; The formula for TOF distance measurement is: ， in, It is the speed of light.

6. The infrared multi-mode composite imaging system based on an APD detector according to claim 5, characterized in that, The high linearity integrator circuit is as follows: The high linearity integrator circuit includes an N-channel enhancement-mode MOSFET and a capacitor. ,resistance First capacitor Reference voltage source, operational amplifier, second capacitor Switching diode and first resistor ; A constant voltage passes through the first resistor The switching diode is connected in series with the input terminal of the reference voltage source, and the output terminal of the reference voltage source is connected to the resistor. ,capacitance Series grounding, capacitor Simultaneously, an N-channel enhancement-mode MOSFET and the non-inverting input of an operational amplifier are connected in parallel, and a first capacitor is connected between the output and input pins of the reference voltage source and its own ground terminal. Second capacitor These two capacitors are connected in parallel with the output of the operational amplifier and are connected to ground together; The operational amplifier is configured as a voltage follower, with a capacitor connected to its non-inverting input. The output terminal provides a low-impedance voltage to ensure the capacitor... The voltage across the capacitor is the same as the output voltage of the operational amplifier; One end is connected to ground, and the other end is connected to a resistor. Connected in series to the output of the reference voltage source; a constant current flows through the resistor. Flow to capacitor This makes the capacitor The voltage gradually increases, i.e., the ramp voltage; the charging current flows through the resistor. The magnitude of the charging current is determined by the resistance. The operational amplifier's output voltage is determined by its reference voltage; the operational amplifier's output voltage follows the capacitor. The operational amplifier controls the output voltage to ensure the capacitor remains constant, and drives the ground terminal of the reference voltage source. The voltage across the terminals is stable relative to the reference voltage; the slope of the ramp voltage is calculated using the following formula, and the values ​​of the resistor and capacitor will be determined based on the integration time: Because the current is constant, the capacitor Voltage over time Linear increase: ， In the above formula, Indicates capacitance The voltage at both ends, Indicates resistance The resistance value, Indicates capacitance The capacitance value; Principle of slope slewing rate calculation: ， in, Indicates the reference voltage; time constant for: ， Current flowing through the circuit for: ， Then the slew rate for: 。 7. The infrared multi-mode composite imaging system based on an APD detector according to claim 1, characterized in that: The FPGA control and processing unit includes an image processing module and an information fusion module. The image processing module performs non-uniformity correction, blind pixel detection and replacement, contrast enhancement, and pseudo-color processing on the digital signals acquired by the APD detector array to improve image quality and obtain an infrared image. The information fusion module designs a two-layer fusion framework based on adaptive weights, calls the infrared image and the distance image for layered fusion to obtain a fused image, and finally synthesizes the above fused image in the YCbCr color space to output a multi-information fused image that can simultaneously display target spatial dimension information and thermal information, realizing the deep fusion of two-dimensional thermal information and three-dimensional spatial information.

8. The infrared multi-mode composite imaging system based on an APD detector according to claim 7, characterized in that, The two-layer fusion framework based on adaptive weights is as follows: 1) The feature-guided adaptive weight generation module extracts physically meaningful features from the range image and infrared image, namely the salience of the structure and the importance of the details, and calculates the spatial weight map to guide the fusion accordingly. 2) Multi-scale decomposition module: performs multi-scale decomposition and layered fusion on images in two modes to obtain their respective low-frequency and high-frequency components. It also uses weight maps to implement adaptive fusion rules in the low-frequency and high-frequency layers. The low-frequency layer contains more than 80% of the basic structure of the image, and the high-frequency layer contains more than 80% of the detailed texture of the image, thus obtaining the structure weight map and detail weight map. 3) In the layered fusion module, in the low-frequency component fusion, a weighted average fusion strategy is adopted in combination with the structure weight map to generate a fused low-frequency image; in the high-frequency component fusion, a feature selection strategy is adopted in combination with the detail weight map to retain significant thermal boundary and detail information in the infrared image to generate a fused high-frequency image; the fused low-frequency image and the high-frequency image are reconstructed to obtain the fused image; 4) Color space synthesis module: Performs channel synthesis based on YCbCr color space; uses the fused image as the luminance component (Y), and extracts the chromaticity component (Cb, Cr) from the pseudo-color 3D distance image after conversion to YCbCr space as color information; generates a new YCbCr image by directly replacing channels: its luminance is entirely determined by the fused image, ensuring that thermal targets and outlines are clearly visible; its chromaticity is entirely determined by the distance information, forming a continuous color code from red to purple, thus obtaining a multi-information fused image; the multi-information fused image is converted back to RGB space for output, so as to be displayed on the host computer.

9. The infrared multi-mode composite imaging system based on an APD detector according to claim 7, characterized in that, The FPGA control and processing unit is configured to perform the following steps to achieve pixel-level fusion, as follows: Step 1: Before fusion processing, the image processing module first registers the thermal radiation intensity image generated by the passive imaging mode with the distance image generated by the active 3D imaging mode; by switching optical components, the two imaging modes share the same optical field of view to ensure spatial reference consistency; in terms of timing control, the two imaging modes are triggered sequentially to complete continuous acquisition in a very short time, forming spatially aligned and temporally correlated image pairs for fusion; at the same time, the passive thermal radiation intensity image is sequentially subjected to non-uniformity correction, blind pixel detection and replacement, contrast enhancement, and pseudo-color processing to obtain an infrared image; Step 2: The information fusion module performs feature analysis and weight generation, extracting physically meaningful features from the distance image and infrared image respectively, namely the saliency of the structure and the importance of details, and calculating the spatial weight map to guide the fusion. The images in the two modes are decomposed into multiple scales and fused in layers. The images are decomposed into different frequency bands by guided filtering, and adaptive fusion rules are implemented in the low-frequency layer and high-frequency layer using the weight map. The low-frequency layer contains more than 80% of the basic structure of the image, and the high-frequency layer contains more than 80% of the detailed texture of the image, resulting in a fused image. Step 3: Perform color mapping on the fused image, combining the structural and texture information in the fused image with the distance information of the pseudo-color encoding in the YCbCr color space to achieve image reconstruction and generate a multi-information fused image.

10. The infrared multi-mode composite imaging system based on an APD detector according to claim 9, characterized in that, In step 1, non-uniformity correction includes single-point correction and two-point correction; blind pixel detection and replacement includes detection based on neighborhood difference method and compensation based on gradient judgment; contrast enhancement adopts contrast-limited adaptive histogram equalization algorithm.

Images