Thermal imaging and visible light fused intelligent security and protection monitoring method

Abstract

The invention relates to the technical field of computer vision and intelligent security and protection, and discloses a thermal imaging and visible light fused intelligent security and protection monitoring method, which comprises the following steps of: synchronously acquiring a visible light image sequence and a thermal imaging image sequence; carrying out defogging enhancement and time-space domain combined noise reduction processing based on a physical model; multi-scale features are extracted through a double-flow encoder, and pixel-level space matching is realized by using a cross-modal alignment module; the bimodal confidence coefficient weight is dynamically distributed in combination with environmental perception information, and robust features are generated through weighted fusion; and finally, outputting a high-precision target detection result, and supporting abnormal behavior discrimination. By constructing a bimodal image enhancement process guided by a physical model, the problem that the performance of a single sensor is sharply reduced under extreme weather and low illumination conditions is effectively solved; a cross-modal feature alignment and dynamic confidence weighted fusion mechanism is adopted, and redundancy or conflicts caused by traditional simple splicing or fixed weight fusion are avoided.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent security technology, specifically relating to an intelligent security monitoring method that integrates thermal imaging and visible light. Background Technology

[0002] With the increasing deployment of intelligent security systems in urban governance, traffic monitoring, border protection, and critical infrastructure protection, the demand for reliable target detection capabilities around the clock is becoming increasingly urgent. Traditional visible light surveillance relies on ambient lighting conditions. In low-light or severe weather conditions such as nighttime, rain, snow, fog, or smoke, image contrast drops sharply, details become blurred, or the system may even fail completely, leading to a significant increase in the rate of missed or false target detection.

[0003] Although thermal imaging technology can operate in dark environments due to its ability to sense infrared radiation, its images generally suffer from low resolution, lack of texture, blurred edges, and noise enhancement caused by environmental temperature differences. The signal-to-noise ratio deteriorates further, especially in high humidity or low temperature scenarios, making it difficult to support high-precision target recognition and behavior analysis. Single-modal perception mechanisms also face inherent limitations in complex dynamic environments, failing to balance semantic richness and robustness.

[0004] Multimodal fusion monitoring has become an important technical approach to improve system reliability, aiming to combine the high-resolution texture information of visible light images with the temperature distribution characteristics of thermal imaging images to achieve complementary advantages. However, existing fusion methods mostly rely on strictly spatiotemporally aligned paired training data, requiring the simultaneous acquisition of high-quality visible light and thermal imaging images in the same scene, which is costly and difficult to scale in practical deployments. More importantly, when both modalities are subjected to extreme environmental degradation (such as severe visible light fogging or strong noise interference in thermal imaging), traditional fusion strategies lack the ability to model the degradation mechanisms of each modality, and cannot perform synergistic enhancement of the two modalities under unsupervised or weakly supervised conditions. This results in the fusion results still retaining traces of the original degradation, or even introducing artifacts or information distortion.

[0005] Generative adversarial networks (GANs) are used for image enhancement, but they often process single modalities in isolation and lack cross-modal self-supervised constraint mechanisms. Other methods, while introducing physical degradation models, rely on precise paired samples for end-to-end training, limiting their generalization ability. In real-world security scenarios, due to heterogeneous equipment, installation deviations, and dynamic environmental changes, obtaining a large number of precisely registered bimodal degradation-sharp image pairs is almost impossible. Summary of the Invention

[0006] This invention provides an intelligent security monitoring method that fuses thermal imaging and visible light, aiming to solve the technical problem that in extreme weather (rain, snow, heavy fog) or low-light environments, the quality of visible light images is severely degraded and the noise of thermal imaging images is significantly increased, leading to the inability to reliably detect targets using a single modality. This method constructs a multimodal perception fusion architecture, introducing a physical model-guided image enhancement mechanism, a cross-modal feature alignment strategy, and a dynamic confidence-weighted decision mechanism to achieve all-weather, highly robust target detection and behavior recognition capabilities.

[0007] As one embodiment of the present invention, the intelligent security monitoring method for thermal imaging and visible light fusion includes the following steps: synchronously acquiring visible light image sequences and thermal imaging image sequences within the same field of view; performing dehazing enhancement processing based on an atmospheric scattering model on the visible light image sequences, and reconstructing brightness and contrast by combining a low-light image adaptive gain compensation algorithm; performing noise reduction processing based on non-uniformity correction and spatiotemporal joint filtering on the thermal imaging image sequences; inputting the enhanced visible light image and the denoised thermal imaging image into a dual-stream encoder network to extract multi-level spatial semantic features of their respective modes; performing pixel-level alignment of the two feature maps in the spatial coordinate system and scale level through a cross-modal feature alignment module; using a dynamic weight allocation unit, calculating the confidence weights of the visible light channel and the thermal imaging channel in real time according to the current ambient light intensity, meteorological visibility, and thermal imaging signal-to-noise ratio; weighting and fusing the aligned dual-modal features according to the confidence weights to generate a fused feature map; and outputting the target's position, category, and confidence information through a target detection head based on the fused feature map.

[0008] Furthermore, the simultaneous acquisition of visible light image sequences and thermal imaging image sequences within the same field of view is specifically achieved through a common aperture optical system. This system includes a beam splitter that divides the incident light into visible light and long-wave infrared bands, which are then projected onto the visible light image sensor and the thermal imaging focal plane array, respectively. The visible light image sensor is a global shutter type complementary metal-oxide-semiconductor image sensor with a spectral range covering 400 nm to 700 nm and a frame rate of not less than 25 Hz. The thermal imaging focal plane array is an uncooled microbolometer array with a response band of 8 μm to 14 μm, a pixel size of not more than 12 μm, and a frame rate of not less than 25 Hz. The two sensors achieve microsecond-level time synchronization through hardware trigger signals and establish a pixel-level spatial mapping relationship through a pre-calibrated intrinsic and extrinsic parameter matrix.

[0009] Furthermore, the dehazing enhancement processing based on the atmospheric scattering model employs a modified atmospheric light value estimation method. First, the initial atmospheric light value is determined by jointly using the dark channel prior and the bright channel prior. Then, the atmospheric light value is locally optimized by combining the relative depth map output by the scene depth estimation module. The transmittance map is solved using a variational optimization model under guided filtering constraints. Its energy function includes a data fidelity term, a gradient sparsity regularization term, and an edge-preserving smoothing term. A clearly structured transmittance distribution is obtained through iterative solution. Finally, the restored image is calculated by a single-scale atmospheric scattering inverse model.

[0010] Furthermore, the low-light image adaptive gain compensation algorithm first performs multi-scale Retinex decomposition on the input image to separate the reflectance component and the illuminance component; then applies dynamic range expansion based on histogram equalization to the illuminance component and introduces illuminance smoothing constraints to suppress overexposed areas; the reflectance component is enhanced for detail through an adaptive contrast stretching function; finally, the image is reconstructed by multiplying the enhanced illuminance component and the reflectance component, and the natural color temperature is restored through a color constancy correction module.

[0011] Furthermore, the denoising process based on non-uniformity correction and spatiotemporal joint filtering includes the following sub-steps: First, a two-point temperature calibration method is used to perform non-uniformity correction on the raw thermal imaging data to eliminate fixed-pattern noise; then, a three-dimensional spatiotemporal cube is constructed with a sliding window length of 5 frames along the time dimension; within this cube, wavelet threshold denoising is performed on the temporal signal at each spatial location to retain the main thermal radiation variation trend; simultaneously, anisotropic diffusion filtering is applied in the spatial domain, with its diffusion coefficient determined by the local gradient magnitude and the thermal imaging signal-to-noise ratio, so as to retain the target edge thermal features while smoothing noise.

[0012] Furthermore, the dual-stream encoder network consists of two identical convolutional neural networks, each containing five downsampling stages. Each stage comprises three convolutional layers, one batch normalization layer, and one modified linear unit activation function. The feature map output by the fifth stage has a resolution of 1 / 32 of the input image. The two encoders share some high-level semantic parameters but retain the modality specificity of the low-level feature extraction. The feature pyramid output by the encoder contains five scale levels, corresponding to feature maps downsampled by 2x, 4x, 8x, 16x, and 32x, respectively.

[0013] Furthermore, the cross-modal feature alignment module adopts a strategy combining deformable convolution and optical flow estimation. First, based on the initial registration results of the visible light and thermal imaging images, a coarse displacement field is calculated. Then, at each level of the feature pyramid, the deformable convolution kernel is used to perform geometric deformation correction on the thermal imaging feature map, so that it is precisely aligned with the visible light feature map in spatial position. The alignment process is optimized end-to-end by minimizing the cross-correlation loss function between the two feature maps.

[0014] Furthermore, the dynamic weight allocation unit includes three environmental perception sub-modules: the light intensity perception sub-module outputs a light quality score by analyzing the average brightness value and standard deviation of the visible light image; the meteorological visibility perception sub-module inverts the current visibility level based on the mean of the transmittance map estimated in the defogging process; and the thermal imaging signal-to-noise ratio perception sub-module evaluates the reliability of the thermal signal by calculating the ratio of the local variance to the global mean of the thermal image. The three scores are normalized and input into a three-layer fully connected network, outputting the visible light channel weights and the thermal imaging channel weights, the sum of which is 1.

[0015] Furthermore, the weighted fusion employs element-wise multiplication and addition operations, performed independently at each level of the feature pyramid. The fused feature map is input to a unified feature aggregation module, which uses an attention mechanism to contextually weight the fused features at different levels to generate the final fused representation. The target detection head is a single-stage detector, which includes a classification branch and a regression branch to predict the target category probability and bounding box coordinates, respectively. The detection results are processed by non-maximum suppression and then output as a final target list.

[0016] As another embodiment of the present invention, the intelligent security monitoring method that integrates thermal imaging and visible light also includes a behavior anomaly discrimination step: constructing a spatiotemporal motion map based on target trajectory data of multiple consecutive frames; modeling the motion map using a graph convolutional network to extract behavioral semantic features; jointly reasoning the behavioral semantic features with target identity attributes and environmental context information to determine whether there are preset abnormal behaviors such as intrusion, loitering, or leftovers; and outputting the abnormal behavior discrimination result along with the target detection information to the security alarm platform.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By constructing a dual-modal image enhancement workflow guided by a physical model, the problem of sharp performance degradation of a single sensor under extreme weather and low-light conditions is effectively overcome; 2. By adopting a cross-modal feature alignment and dynamic confidence weighted fusion mechanism, the optimal combination of perceptual information can be obtained under any environmental conditions, avoiding redundancy or conflict caused by traditional simple splicing or fixed weight fusion. 3. The introduced environment-aware weight allocation strategy enables the system to adapt to changes in the environment, significantly improving the accuracy and robustness of target detection. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall technical solution architecture of the intelligent security monitoring method that integrates thermal imaging and visible light proposed in this invention. Figure 2This is a schematic diagram illustrating the core principle framework of the dynamic confidence-weighted decision-making mechanism in this invention; Figure 3 This is a flowchart of the logic process of dual-modal image enhancement processing (including visible light defogging and low-light compensation, and thermal imaging noise reduction) in this invention. Figure 4 This is a logical framework diagram of cross-modal feature alignment and feature extraction and alignment of a two-stream encoder network in this invention; Figure 5 This is a logical flowchart of the multi-factor weight allocation and feature fusion driven by environmental perception in this invention. Figure 6 This is a diagram of the multi-stage joint reasoning logic framework for target detection and abnormal behavior discrimination in this invention. Detailed Implementation

[0019] Please refer to the attached document. Figures 1 to 6 This invention provides an intelligent security monitoring method that fuses thermal imaging and visible light, aiming to solve the technical problem that in extreme weather (rain, snow, fog) or low-light environments, the quality of visible light images is severely degraded and the noise of thermal imaging images is significantly increased, resulting in the inability to reliably detect targets using a single modality. This method constructs a multimodal perception fusion architecture, introducing a physical model-guided image enhancement mechanism, a cross-modal feature alignment strategy, and a dynamic confidence-weighted decision mechanism to achieve all-weather, highly robust target detection and behavior recognition capabilities.

[0020] The intelligent security monitoring method that fuses thermal imaging and visible light includes the following steps: Simultaneously acquire visible light image sequences and thermal imaging image sequences within the same field of view; The visible light image sequence is subjected to dehazing enhancement processing based on the atmospheric scattering model, and brightness and contrast are reconstructed by combining the low-light image adaptive gain compensation algorithm. Denoising processing based on non-uniformity correction and spatiotemporal joint filtering is performed on thermal imaging image sequences; The enhanced visible light image and the denoised thermal image are input into a dual-stream encoder network to extract multi-level spatial semantic features of their respective modalities. The cross-modal feature alignment module performs pixel-level alignment of the two feature maps in both spatial coordinate system and scale level. Using a dynamic weight allocation unit, the confidence weights of the visible light channel and the thermal imaging channel are calculated in real time based on the current ambient light intensity, meteorological visibility, and thermal imaging signal-to-noise ratio. The aligned dual-modal features are weighted and fused according to the confidence weights to generate a fused feature map. Based on the fused feature map, the target detection head outputs the target's location, category, and confidence information.

[0021] Simultaneous acquisition of visible light and thermal imaging image sequences within the same field of view is achieved through a common-aperture optical system. This system includes a beam-splitting prism that separates incident light into visible light and long-wave infrared bands, which are then projected onto the visible light image sensor and the thermal imaging focal plane array, respectively. The visible light image sensor is a global shutter type complementary metal-oxide-semiconductor (CMOS) image sensor with a spectral range covering 400 nm to 700 nm and a frame rate of at least 25 Hz. The thermal imaging focal plane array is an uncooled microbolometer array with a response range of 8 μm to 14 μm, a pixel size no larger than 12 μm, and a frame rate of at least 25 Hz. The two sensors achieve microsecond-level time synchronization via hardware trigger signals and establish a pixel-level spatial mapping relationship through pre-calibrated intrinsic and extrinsic parameter matrices. The intrinsic and extrinsic parameter matrices are calibrated using a high-precision calibration platform before the equipment leaves the factory. The intrinsic parameters include focal length, principal point offset, and distortion coefficient, while the extrinsic parameters include rotation matrix and translation vector, ensuring that the geometric consistency error of the two modal images in physical space is less than 0.5 pixels.

[0022] A dehazing enhancement process based on an atmospheric scattering model is performed on visible light image sequences, employing a modified atmospheric light value estimation method. First, initial atmospheric light values ​​are determined jointly using dark channel and bright channel priors. The dark channel prior assumes that in a hazy natural image, at least some local regions have pixel values ​​in at least one color channel close to 0; the bright channel prior provides an upper limit constraint on atmospheric light for bright areas such as the sky or strongly reflective surfaces. After combining these two priors, the initial atmospheric light value is taken as the average of the brightest 1 / 1000 pixels in the image, but does not exceed the threshold defined by the bright channel prior. Subsequently, the atmospheric light values ​​are locally optimized using the relative depth map output by the scene depth estimation module. The depth estimation module, based on monocular visual geometric constraints, predicts the relative depth value of each pixel using a convolutional neural network; a larger depth value indicates a greater distance. In distant regions, atmospheric light contributes more, therefore, local atmospheric light values ​​are weighted according to depth proportions. The transmittance map is solved using a variational optimization model under guided filtering constraints, with its energy function defined as: ; in, The original foggy image, This represents the locally optimized atmospheric light value. For the transmittance to be determined, To restore the image, and The regularization coefficient is . The edge weighting function is determined by the image gradient magnitude, and its value is smaller in the edge regions to preserve structure. This energy function is solved iteratively using the alternating direction multiplier method to obtain a transmittance distribution with clear structure and well-preserved edges. The final restored image is calculated using a single-scale atmospheric scattering inverse model, as shown in the formula: ; in The lower limit of transmittance is set to 0.1 to prevent division by zero errors and to retain some atmospheric scattering effect to maintain a natural appearance.

[0023] Under low-light conditions, visible light images also require an adaptive gain compensation algorithm. This algorithm first performs multi-scale Retinex decomposition on the input image, transforming the image... Represented as reflection component With illuminance component The product of, i.e. Multi-scale Retinex separates the illuminance component by taking the logarithm of the image at multiple scales using a Gaussian kernel and then averaging the results. A dynamic range extension based on histogram equalization is applied to the illuminance component. Specifically, the cumulative distribution function of the illuminance component is calculated and mapped to the full dynamic range, but an illuminance smoothing constraint is introduced, i.e., bilateral filtering is applied to the mapped illuminance to suppress overexposure areas caused by excessive stretching. The reflectance component is then enhanced for detail using an adaptive contrast stretching function, defined as: ; in and These represent the minimum and maximum values ​​of the reflection component within the local window, respectively. For local standard deviation, The enhancement factor is set to 1.5. This function amplifies differences in low-contrast regions and remains stable in high-contrast regions, avoiding noise amplification. The final image is reconstructed by multiplying the enhanced illuminance component and the reflectance component, and then the natural color temperature is restored through the color constancy correction module. Color constancy correction uses a combination of the gray-world assumption and white point correction to calculate the average ratio of the three channels and adjust the white balance of the image to ensure that the colors in nighttime images are not distorted.

[0024] Noise reduction processing based on non-uniformity correction and spatiotemporal joint filtering is performed on the thermal imaging image sequence. First, a two-point temperature calibration method is used to correct the non-uniformity of the raw thermal imaging data. This method is executed at device startup or periodically, placing the sensor under two blackbody radiation sources of known temperatures (e.g., 25°C and 50°C), recording the response value of each pixel, and establishing linear correction parameters: gain. With bias For any raw reading Corrected value for: ; This correction eliminates fixed-pattern noise, ensuring consistent output across all pixels at the same temperature. Subsequently, a three-dimensional spatiotemporal cube is constructed, with a sliding window of 5 frames along the time dimension, forming... The data block. Within this cube, for each spatial location Timing signals Wavelet thresholding denoising is performed. A three-level decomposition is conducted using the Daubechies fourth-order wavelet basis. A soft thresholding function is applied to the high-frequency coefficients, with the threshold determined by the noise standard deviation estimate, preserving the main trends in thermal radiation variation. Simultaneously, anisotropic diffusion filtering is applied in the spatial domain, with the diffusion equation as follows: ; in For thermal images, The diffusion coefficient is defined as: ; It is a constant, with a value of 10. The signal-to-noise ratio (SNR) of the current frame thermal imaging is estimated by the ratio of local variance to global mean. This design reduces diffusion in high-gradient regions (target edges) and under high SNR conditions, thereby preserving the thermal features of the target edge while smoothing out noise.

[0025] The enhanced visible light image and the denoised thermal image are input into a two-stream encoder network. This network consists of two identical convolutional neural networks, each containing five downsampling stages. Each stage comprises three convolutional layers (3×3 kernel size, stride 1), a batch normalization layer, and a modified linear unit activation function. The first stage has an input resolution of 640×480 and an output of 320×240; the second stage outputs 160×120; the third stage outputs 80×60; the fourth stage outputs 40×30; and the fifth stage outputs 20×15. The two encoders share the kernel weights of the fifth stage, but the first through fourth stages remain independent to preserve the modality specificity of the low-level features. The feature pyramid output by the encoder contains five scale levels, corresponding to feature maps downsampled by 2x, 4x, 8x, 16x, and 32x, respectively, with 64, 128, 256, 512, and 1024 feature channels per level.

[0026] A cross-modal feature alignment module performs pixel-level alignment of two feature maps in both spatial coordinates and scale levels. This module employs a strategy combining deformable convolution and optical flow estimation. First, based on the initial registration results of the visible light and thermal imaging images (provided by pre-calibrated intrinsic and extrinsic parameter matrices), a coarse displacement field is calculated. Then, at each level of the feature pyramid, deformable convolution kernels are used to perform geometric deformation correction on the thermal imaging feature map. The offset of the deformable convolution is predicted by a light quantum network, which receives the concatenation of the two modal features as input and outputs a two-dimensional offset vector for each sampling point. The alignment process is optimized end-to-end by minimizing the cross-correlation loss function between the two feature maps, defined as: ; in and These are visible light and thermal imaging feature maps, respectively. The offset field is learned by deformable convolution. This loss function maximizes the cross-correlation between two feature maps, ensuring that semantically corresponding points are aligned in the feature space.

[0027] Using a dynamic weight allocation unit, the confidence weights of the visible light channel and the thermal imaging channel are calculated in real time based on the current ambient light intensity, meteorological visibility, and thermal imaging signal-to-noise ratio. This unit comprises three environmental perception submodules. The light intensity perception submodule analyzes the average brightness value and standard deviation of the visible light image to output a light quality score. Specifically, the mean value is calculated after converting the image to grayscale. with standard deviation ,score Defined as: ; The meteorological visibility perception submodule is based on the mean of the transmittance map estimated during the defogging process. The current visibility level is retrieved. Lower transmittance indicates poorer visibility, and a lower score is assigned. Defined as The thermal imaging signal-to-noise ratio sensing submodule assesses the reliability of the thermal signal by calculating the ratio of the local variance to the global mean of the thermal image. The image is divided into 16×16 blocks, and the variance of each block is calculated. Take the maximum value and the global mean. The ratio as Estimate, rating The three scores, after being normalized, are input into a three-layer fully connected network (with 64 and 32 hidden layer nodes respectively), and the output is the visible light channel weights. With thermal imaging channel weights ,satisfy .

[0028] The aligned bimodal features are weighted and fused according to the aforementioned confidence weights to generate a fused feature map. The fusion operation is performed independently at each level of the feature pyramid, using element-wise multiplication and addition. The fused feature maps are input to a unified feature aggregation module, which uses channel attention and spatial attention mechanisms to contextually weight the fused features at different levels. Channel attention generates channel weights through global average pooling, and spatial attention generates spatial weights through convolutional layers. The two are multiplied together and then multiplied with the input features to generate the final fused representation.

[0029] Based on the fused feature map, the target detection head outputs the target's location, category, and confidence information. The target detection head is a single-stage detector, containing a classification branch and a regression branch. The classification branch predicts the probability that each anchor box belongs to the background or a preset category (such as person, vehicle, or animal), while the regression branch predicts the offset of the bounding box relative to the anchor box. The detection results undergo non-maximum suppression processing, with an intersection-over-union (IoU) threshold of 0.5 set to remove overlapping boxes, and the final target list is output.

[0030] The intelligent security monitoring method that fuses thermal imaging and visible light also includes a behavioral anomaly detection step. A spatiotemporal motion graph is constructed based on 30 consecutive frames of target trajectory data. Nodes in the graph represent the target's position in each frame, and edges represent displacement vectors between adjacent frames. A graph convolutional network is used to model the motion graph, with each layer aggregating neighbor node information and updating node features. After three layers of graph convolution, global pooling yields a behavioral semantic feature vector. This vector is concatenated with target identity attributes (such as size and thermal radiation intensity) and environmental context information (such as time and regional sensitivity level), and input into a fully connected classifier to determine whether preset abnormal behaviors such as intrusion, loitering, or leaving objects exist. The abnormal behavior detection result, along with the target detection information, is output to the security alarm platform, triggering the corresponding warning level.

[0031] The entire methodology is deployed on an embedded graphics processor, employing fixed-point quantization and layer fusion optimization to ensure real-time performance. The system was tested in real-world scenarios including rain, snow, dense fog, and nighttime operation without supplemental lighting. The target detection rate was improved by 42.6% compared to a single visible light solution, and by 28.3% compared to a single thermal imaging solution, while the false alarm rate was reduced by 61%, validating the effectiveness and robustness of the technical solution.

Claims

1. A smart security monitoring method that fuses thermal imaging and visible light, characterized in that, include: Simultaneously acquire visible light image sequences and thermal imaging image sequences within the same field of view; The visible light image sequence is subjected to dehazing enhancement processing based on an atmospheric scattering model, and brightness and contrast are reconstructed by combining a low-light image adaptive gain compensation algorithm. The thermal imaging image sequence is subjected to noise reduction processing based on non-uniformity correction and spatiotemporal joint filtering; The enhanced visible light image and the denoised thermal image are input into a dual-stream encoder network to extract multi-level spatial semantic features of their respective modalities. The cross-modal feature alignment module performs pixel-level alignment of the two feature maps in both spatial coordinate system and scale level. Using a dynamic weight allocation unit, the confidence weights of the visible light channel and the thermal imaging channel are calculated in real time based on the current ambient light intensity, meteorological visibility, and thermal imaging signal-to-noise ratio. The aligned dual-modal features are weighted and fused according to the confidence weights to generate a fused feature map. Based on the fused feature map, the target detection head outputs the target's location, category, and confidence information.

2. The intelligent security monitoring method based on the fusion of thermal imaging and visible light according to claim 1, characterized in that, The simultaneous acquisition of visible light image sequences and thermal imaging image sequences within the same field of view includes: The incident light is split into visible light and long-wave infrared bands by a beam splitter in a common aperture optical system, and then projected onto a visible light image sensor and a thermal imaging focal plane array, respectively. The visible light image sensor is a global shutter type complementary metal-oxide-semiconductor image sensor with a spectral range covering 400 nm to 700 nm and a frame rate of not less than 25 Hz. The thermal imaging focal plane array is an uncooled microbolometer array with a response band of 8 micrometers to 14 micrometers, a pixel size of no more than 12 micrometers, and a frame rate of no less than 25 Hz. The two sensors achieve microsecond-level time synchronization through hardware trigger signals and establish pixel-level spatial mapping relationships through pre-calibrated intrinsic and extrinsic parameter matrices.

3. The intelligent security monitoring method based on the fusion of thermal imaging and visible light according to claim 1, characterized in that, The dehazing enhancement process based on the atmospheric scattering model includes: The initial atmospheric light value is determined by combining the dark channel prior and the bright channel prior, and the atmospheric light value is locally optimized by combining the relative depth map output by the scene depth estimation module. A variational optimization model under guided filtering constraints is used to solve the transmittance map. Its energy function includes a data fidelity term, a gradient sparsity regularization term, and an edge-preserving smoothing term. The restored image was calculated using a single-scale atmospheric scattering inverse model.

4. The intelligent security monitoring method based on the fusion of thermal imaging and visible light according to claim 1, characterized in that, The low-light image adaptive gain compensation algorithm includes: Multi-scale Retinex decomposition is performed on the input image to separate the reflectance and illuminance components; The illuminance component is subjected to dynamic range expansion based on histogram equalization and illuminance smoothing constraints are introduced. The reflection component is enhanced with detail using an adaptive contrast stretching function; The enhanced illuminance component is multiplied by the reflection component to reconstruct the image, and the natural color temperature is restored through the color constancy correction module.

5. The intelligent security monitoring method based on the fusion of thermal imaging and visible light according to claim 1, characterized in that, The noise reduction process based on non-uniformity correction and spatiotemporal joint filtering includes: A two-point temperature calibration method was used to correct the non-uniformity of the raw thermal imaging data in order to eliminate fixed-pattern noise. Construct a three-dimensional spatiotemporal cube, and slide a window of 5 frames along the time dimension; Within the cube, wavelet thresholding denoising is performed on the time-series signal at each spatial location; When anisotropic diffusion filtering is applied in the spatial domain, its diffusion coefficient is determined by the local gradient magnitude and the thermal imaging signal-to-noise ratio.

6. The intelligent security monitoring method based on the fusion of thermal imaging and visible light according to claim 1, characterized in that, The dual-stream encoder network consists of two identical convolutional neural networks. Each network contains five downsampling stages, and each stage consists of three convolutional layers, one batch normalization layer, and one modified linear unit activation function. The feature map output in the fifth stage has a resolution of 1 / 32 of the input image; the two encoders share some high-level semantic parameters, but retain the modality specificity of the low-level feature extraction. The feature pyramid output by the encoder contains five scale levels, corresponding to feature maps downsampled by 2x, 4x, 8x, 16x and 32x respectively.

7. The intelligent security monitoring method based on the fusion of thermal imaging and visible light according to claim 1, characterized in that, The cross-modal feature alignment module adopts a strategy that combines deformable convolution with optical flow estimation. First, a coarse displacement field is calculated based on the initial registration results of visible light and thermal imaging images. Then, at each level of the feature pyramid, the deformable convolution kernel is used to perform geometric deformation correction on the thermal imaging feature map. The alignment process is optimized end-to-end by minimizing the cross-correlation loss function between the two feature maps.

8. The intelligent security monitoring method based on the fusion of thermal imaging and visible light according to claim 1, characterized in that, The dynamic weight allocation unit comprises three environment-aware sub-modules: The illumination intensity sensing submodule outputs an illumination quality score by analyzing the average brightness value and standard deviation of a visible light image. The meteorological visibility perception submodule retrieves the visibility level based on the mean of the transmittance map estimated during the defogging process; The thermal imaging signal-to-noise ratio sensing submodule evaluates the reliability of thermal signals by calculating the ratio of the local variance to the global mean of the thermal image; The three scores are normalized and then input into a three-layer fully connected network, which outputs the visible light channel weights and the thermal imaging channel weights, the sum of which is 1.

9. The intelligent security monitoring method based on the fusion of thermal imaging and visible light according to claim 1, characterized in that, The weighted fusion employs element-wise multiplication and addition operations, performed independently at each level of the feature pyramid. The fused feature maps are input into a unified feature aggregation module, which uses an attention mechanism to perform context weighting on the fused features at different levels to generate the final fused representation. The target detection head is a single-stage detector, which includes a classification branch and a regression branch to predict the target category probability and bounding box coordinates, respectively; the detection results are processed by non-maximum suppression and then output as a final target list.

10. The intelligent security monitoring method based on the fusion of thermal imaging and visible light according to claim 1, characterized in that, It also includes steps for identifying behavioral anomalies: A spatiotemporal motion graph is constructed based on target trajectory data from multiple consecutive frames; a graph convolutional network is used to model the motion graph to extract behavioral semantic features. By combining behavioral semantic features with target identity attributes and environmental context information, we can determine whether there are pre-defined abnormal behaviors such as intrusion, loitering, or leaving behind objects. The abnormal behavior determination results, along with the target detection information, are output to the security alarm platform.

Images