A cross-scale multi-modal image fusion method and system

Abstract

The present application relates to a kind of cross-scale multimodal image fusion method and system.Therein, the method includes: obtaining multimodal image data, space-time synchronization and global perspective alignment are carried out to the multimodal image data;Image data is encapsulated using channel dimension, and the input tensor of multimodal image is generated;Original modality feature map is generated by nested bottleneck feature extraction;Aggregation multi-receptive field global context information, while accessing parallel spatial attention module, output multi-scale feature pyramid set;The multiscale features of infrared and visible light are spliced along the channel, combined with dynamic convolution to generate single-scale basic fusion features;Repair scale semantic fault, output cross-scale fusion features;Fusion features are input into decoupling detection head, and image reconstruction, target classification and frame regression task are optimized;Based on the optimization result, subpixel pixel mapping reconstruction is carried out, and cross-scale multimodal fusion image is output;Cross-scale multimodal image fusion is realized.

Description

Technical Field

[0001] This invention belongs to the field of image fusion technology, specifically relating to a cross-scale multimodal image fusion method and system. Background Technology

[0002] Current multimodal image fusion technologies still have the following areas for improvement: Although multimodal fusion technology can theoretically perfectly complement the advantages of infrared and visible light information, in practical engineering applications of security scenarios, given the characteristics of dense small targets, complex backgrounds, and high real-time requirements, existing deep learning-based image fusion and target detection joint methods still have the following significant shortcomings that urgently need improvement: First, insufficient cross-scale, multi-level contextual information interaction leads to the easy under-detection of small targets in complex scenarios. Security scenarios are typically characterized by extremely large field of view and drastic changes in target scale. Existing fusion networks are often limited to single-layer feature alignment and fusion at the same scale, neglecting the vertical transmission of cross-scale hierarchical structural information. The lack of tight interconnection between local texture features and global high-level semantic features results in insufficient receptive field, artifacts, and blurred edges in the fused feature map in areas with densely distributed small targets, leading to severe loss of fine-grained information and ultimately a precipitous drop in detection recall.

[0003] Second, there is a sharp contradiction between computational complexity and the requirements of real-time inference at the edge. Some studies have attempted to improve overall performance by jointly optimizing image fusion and object detection tasks, but these multi-stage training models (such as dual-channel Faster R-CNN or early complex dual-stream variants based on YOLOv5 / v7) not only have a large number of parameters, but also suffer from extremely high inference latency due to feature alignment and attention computation. Security edge devices (such as smart cameras and drone-borne computing platforms) are usually subject to strict power consumption and computing power limitations, making it impossible to support large-scale multimodal network structures, which hinders the deployment of these advanced algorithms.

[0004] Third, the efficiency and feature fidelity of front-end preprocessing are insufficient in extreme environments such as nighttime and low light. In nighttime security scenarios, raw visible light images often suffer from severe dark area noise and texture loss. Traditional approaches either rely entirely on backend deep learning networks for feature extraction, leading to excessive network load, or use simple global histogram equalization, which easily causes local overexposure and noise amplification. Currently, there is a lack of a front-end preprocessing mechanism that can adaptively and collaboratively enhance visible light texture, suppress infrared noise, and seamlessly and efficiently couple with deep learning detection networks.

[0005] In summary, to address technical challenges such as missed detection of multi-scale small targets, gradient conflicts between fusion and detection tasks, and computational bottlenecks at the edge, this invention proposes a cross-scale multimodal image fusion method and apparatus. This solution closely relies on the newly released high-parameter-utilization YOLO11 neural network architecture from Ultralytics, achieving high-precision, robust, and real-time-compliant cross-scale multimodal target detection and image fusion without placing excessive pressure on computational resources. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a cross-scale multimodal image fusion method and system; The objective of this invention can be achieved through the following technical solutions: S1: Acquire multimodal image data, perform spatiotemporal synchronization and global perspective alignment on the multimodal image data, and output registered image pairs; perform single-modal image quality optimization based on the registered image pairs, and normalize the multimodal image pixels based on the optimization results; encapsulate the image data using the channel dimension to generate the input tensor of the standardized multimodal image; S2: Based on the input tensor, multi-level resolution original modal feature maps are generated through nested bottleneck feature extraction; multi-receptive field global context information is aggregated, and a parallel spatial attention module is connected to output a set of multi-scale feature pyramids for visible light and infrared. S3: Multi-scale features of infrared and visible light are stitched together along the channel and dynamically gated fusion is performed by dynamic convolution to generate single-scale basic fusion features; cross-scale semantic constraints are established by calculating inter-level structural similarity and gradient difference; global semantics and detailed textures are fused to repair scale semantic discontinuities and output cross-scale fusion features. S4: Lossless semantic flow is achieved through bidirectional information flow. The fused features are input into the decoupled detection head. The image reconstruction, target classification, and bounding box regression tasks are optimized synchronously through joint gradient balancing loss. Based on the optimization results, sub-pixel pixel mapping reconstruction is performed, and a cross-scale multimodal fused image is output.

[0007] Specifically, the process of acquiring multimodal image data includes: The real-time video streams of infrared and visible light are acquired through a real-time streaming protocol. The image frames are then bilinearly interpolated based on a preset timestamp to obtain a scaled image. The scaled and time-matched infrared and visible light paired images are integrated to obtain a standardized original bimodal image sequence.

[0008] Specifically, the process of outputting the registered image pair includes: Through binocular joint calibration, the mapping homography transformation matrix between the infrared imaging plane and the visible light imaging plane is obtained, and the matrix parameters are encrypted and stored in the local configuration file of the edge device. Read the homography transformation matrix stored locally, perform pixel-level perspective remapping operation on the original infrared images in the sequence, and project the infrared image pixels onto the visible light unified imaging coordinate system; Correcting three types of geometric deviations—overall translation, rotation, and scaling—results in infrared-visible image pairs.

[0009] Specifically, the process of performing single-modal image quality optimization includes: Bilateral filtering is performed on the infrared-visible image pair, and the denoised visible image is segmented to obtain local sub-image blocks; A limited contrast adaptive histogram equalization algorithm is used to process noise in dark areas, while bilinear interpolation is used to process the blocky boundary traces caused by splicing. The average grayscale value of the entire visible light image is statistically analyzed in real time. Based on the average grayscale value, the gamma correction coefficient is dynamically adjusted to reshape the overall grayscale distribution of the image and brighten the dark areas and hidden target textures in nighttime and backlight environments. Neighborhood thresholding is performed on paired infrared images, and the two optimized images are integrated to generate an image with enhanced image quality.

[0010] Specifically, the process of encapsulating image data using the channel dimension includes: Linearly map the original pixel grayscale values ​​of enhanced visible light and infrared images to a floating-point normalized range; Visible light and infrared images are encapsulated into a three-dimensional tensor format with preset height, width, and number of channels, and stored using floating-point pixels; The visible light and infrared tensors are combined and encapsulated to obtain a dual-modal tensor data packet.

[0011] Specifically, the process of extracting bottleneck features through nesting includes: The visible light and infrared tensors are input into the initial convolutional layer. Standard convolution is used to perform channel dimensionality upsampling and initial image downsampling, and shallow basic features are output. A lightweight, cross-stage local bottleneck structure is adopted to equally split the input features; Based on the splitting results, the branch output features are spliced ​​and fused along the channel dimension to generate a downsampled original feature map. The hierarchical original feature map generated by the branch network is output, and the sampled hierarchical features are input into the spatial pyramid pooling module.

[0012] Specifically, the process of aggregating multi-receptive field global context information and simultaneously integrating it into a parallel spatial attention module includes: Multi-size max pooling operations are performed using the spatial pyramid pooling module to obtain global scene features under the corresponding receptive field. The global scene features are split across stages and spatial location features are extracted through parallel multi-group convolutions. The spatial response weights are calculated using vector dot products, and the spatial weight matrix is ​​generated using a normalized exponential function. The spatial weight matrix and the original pooling features are used to perform pixel calculations to obtain the feature responses of the infrared heat source target and the dense visible light texture region in the region of interest. By integrating weighted features, a set of multi-scale feature pyramids for visible light and infrared light is generated.

[0013] Specifically, the process of generating single-scale basic fusion features includes: Select a single visible light feature map and a single infrared feature map, and directly stitch them together based on the channel dimension to generate a fused hybrid feature tensor; Global average pooling is performed on the mixed feature tensor, and weights are assigned to the corresponding channels. By employing a joint operation of global max pooling and average pooling, the corresponding spatial features are extracted to generate a two-dimensional spatial attention matrix; The channel weights, spatial weights, and hybrid features are multiplied element-wise, and multi-dimensional feature interaction is performed through full-dimensional dynamic convolution to output single-scale basic fusion features.

[0014] Specifically, the process of outputting cross-scale fusion features includes: The feature resolutions are aligned, and the gradient details of high-level features and current layer features are extracted using the Gaussian-Laplacian pyramid operator. The structural similarity values ​​of the two layers of features are calculated by dividing the image into blocks. We employ dilated convolution with multiple dilation coefficients in parallel to extract multi-receptive field context and address edge blurring and small target artifacts caused by scale switching. Based on the cross-scale reconstruction results at the corresponding level, cross-scale enhanced fusion features are output.

[0015] Specifically, the process of lossless semantic communication includes: By constructing a two-way information flow path, the texture of object edges and small targets can be supplemented; Lightweight cross-stage local modules are deployed for feature upsampling and channel splicing nodes to control the amount of floating-point operations caused by channel expansion. By connecting a cross-stage parallel spatial attention module, the region of interest of the security target is automatically identified, and the target features are enhanced by secondary weighting to obtain lossless semantic flow of multimodal images.

[0016] Specifically, the process of optimizing image reconstruction, object classification, and bounding box regression tasks includes: The aggregated features are split into two independent computation branches: the first branch uses depthwise separable convolution lightweight processing to perform confidence classification of the target category; the second branch uses standard convolution to calculate the center point of the target bounding box and its length and width offsets. Three loss functions are constructed: target classification focus loss, bounding box coordinate regression loss, and image structure similarity reconstruction loss. The unified gradient balancing loss function dynamically adjusts the weighting coefficients of the three types of losses during the training phase to balance the backpropagation gradient direction conflicts between the two major tasks of image visual reconstruction and object detection, and outputs high-dimensional abstract fusion features after gradient optimization.

[0017] Specifically, the process of outputting the cross-scale multimodal fused image includes: The optimized high-dimensional abstract fusion features are reversed by dimension mapping to obtain sub-pixel level micro-contour and fine texture information; Construct a mapping rule from abstract feature values ​​to the corresponding RGB pixels of the image to generate a cross-scale multimodal fusion image.

[0018] Specifically, a cross-scale multimodal image fusion system includes: The multimodal acquisition and preprocessing unit is used to acquire real-time video streams from both infrared and visible light sources, standardize the data format, perform bilateral filtering and CLAHE gamma adaptive image enhancement, normalize the pixel value range, and construct a standardized dual-modal input tensor. The multi-scale feature extraction unit extracts multi-level features by using a built-in lightweight C3k2 nested bottleneck backbone network and SPPF spatial pyramid pooling and C2PSA parallel spatial attention submodule to hierarchically downsample the dual-modal tensor, aggregate the global context of multiple receptive fields, and generate a set of independent multi-scale feature pyramids for visible light and infrared light. The cross-scale feature fusion unit integrates dynamic convolution, Gaussian-Laplacian pyramid, and multi-dilated dilated convolution sub-modules to concatenate and dynamically gated feature channels of the same scale dual modality; it uses SSIM structural similarity to construct cross-scale semantic constraints, uses bidirectional PAN pathways to repair scale semantic faults, and outputs cross-scale fused features. The multi-task reconstruction inference unit, through the gradient balancing joint loss calculation module, simultaneously optimizes three types of tasks: image reconstruction, object classification, and bounding box regression. It performs sub-pixel inverse mapping pixel reconstruction on optimized high-dimensional features and outputs cross-scale multimodal fusion images.

[0019] The beneficial effects of this invention are as follows: It achieves highly robust target detection capabilities in all weather conditions and harsh environments. Through adaptive collaborative preprocessing (CLAHE + Gamma correction) between the dual-stream decoupled independent feature extraction backbone network and the front-end, the model fully absorbs and deeply integrates the rich spatial texture of visible light and the high-penetration thermal radiation information of infrared light. Regardless of whether it is in no light, weak light, strong light, backlight, or complex environments such as rain, snow, fog, haze, or when the target is partially obscured by camouflage netting, the system can stably output high-quality multimodal features. This fundamentally overcomes the physical failure risk of a single visual sensor in complex weather and lighting conditions, significantly improving the all-weather operational and perception reliability of military and civilian security monitoring and UAV reconnaissance systems.

[0020] A generational leap in multi-scale, high-density micro-target detection capabilities: By innovatively introducing a cross-scale multimodal fusion module, this invention breaks the rigid limitations of same-level fusion, establishing a dynamic information pathway between low-level local high-resolution details and high-level global semantic features. Utilizing Gaussian-Laplacian pyramid structural similarity measurement and ODConv multidimensional dynamic convolution, the attenuation of fine-grained information is effectively mitigated. Simultaneously, combined with YOLO11's unique C2PSA parallel spatial attention mechanism, C2PSA is defined as a convolutional block with parallel spatial attention. Mathematically, it first performs CSP splitting, followed by parallel convolution. Subsequently, it uses a dot product vector algorithm to fuse spatial and channel information and predicts spatial response weights through the Softmax function, thereby applying spatial attention. This mechanism forces the model to adaptively focus on high thermal radiation salient regions in the infrared feature map and complex edge information-dense regions in the visible light feature map, greatly enhancing the recognition accuracy for extremely small targets, blurred targets, and severely occluded targets.

[0021] Validation data: Tests on authoritative multimodal open-source datasets (such as the DroneVehicle drone-view security dataset and the FLIR thermal imaging target detection dataset) show that the average accuracy (map@0.5) of this solution is steadily improved by 3.41% to 5.65% on the FLIR dataset, with a peak of 47.61%; and an accuracy of up to 80.53% is achieved on the DroneVehicle dataset.

[0022] Achieving an optimal balance between computational resource consumption and model representation capabilities, perfectly suited to edge device deployment requirements: Compared to traditional multi-stage solutions that simply string together a large, complex fusion reconstruction network and a detection network, this invention adopts a highly integrated architecture design. It deeply reuses and improves upon the YOLO11 core's C3k2 architecture. The C3k2 module utilizes smaller, customized convolutional kernels and flexible channel-based computation strategies to significantly reduce the model parameter size and floating-point operations while ensuring and even further enhancing feature representation capabilities.

[0023] Validation data: The improved lightweight model version not only completely surpasses its predecessor in accuracy, but also reduces the number of parameters by an astonishing 22% compared to YOLOv8m. Combined with an anchor-free decoupling head and an improved lightweight neck, it enables complex multimodal fusion systems to easily achieve high-definition real-time video stream inference at over 30 FPS on computationally limited embedded computing platforms or dedicated security NPUs, fully meeting the stringent low-latency requirements of fields such as smart security and intelligent transportation. Attached Figure Description

[0024] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart illustrating a cross-scale multimodal image fusion method and system according to the present invention. Figure 2 This is a flowchart of the cross-scale enhanced fusion feature in this invention; Figure 3 , 4 This is a partial view of the image fusion in this invention; Figure 5 This is a schematic diagram of image fusion in this invention. Detailed Implementation

[0026] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of the specific implementation methods, structures, features and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0027] Please see Figures 1-5 A cross-scale multimodal image fusion method and system, comprising: S1: Acquire multimodal image data, perform spatiotemporal synchronization and global perspective alignment on the multimodal image data, and output registered image pairs; perform single-modal image quality optimization based on the registered image pairs, and normalize the multimodal image pixels based on the optimization results; encapsulate the image data using the channel dimension to generate the input tensor of the standardized multimodal image; S2: Based on the input tensor, multi-level resolution original modal feature maps are generated through nested bottleneck feature extraction; multi-receptive field global context information is aggregated, and a parallel spatial attention module is connected to output a set of multi-scale feature pyramids for visible light and infrared. S3: Multi-scale features of infrared and visible light are stitched together along the channel and dynamically gated fusion is performed by dynamic convolution to generate single-scale basic fusion features; cross-scale semantic constraints are established by calculating inter-level structural similarity and gradient difference; global semantics and detailed textures are fused to repair scale semantic discontinuities and output cross-scale fusion features. S4: Lossless semantic flow is achieved through bidirectional information flow. The fused features are input into the decoupled detection head. The image reconstruction, target classification, and bounding box regression tasks are optimized synchronously through joint gradient balancing loss. Based on the optimization results, sub-pixel pixel mapping reconstruction is performed, and a cross-scale multimodal fused image is output.

[0028] Specifically, the process of acquiring multimodal image data includes: The real-time video streams of infrared and visible light are acquired through a real-time streaming protocol. The image frames are then bilinearly interpolated based on a preset timestamp to obtain a scaled image. The scaled and time-matched infrared and visible light paired images are integrated to obtain a standardized original bimodal image sequence.

[0029] In this embodiment, the underlying driver of the binocular PTZ camera integrates a unified RTSP real-time streaming unit. The hardware motherboard shares the same hardware clock source. The visible light channel outputs an RGB color encoded bitstream, and the infrared channel outputs a single-channel grayscale thermal imaging bitstream. Each frame of both bitstreams is written with a shared hardware timestamp, eliminating frame timing misalignment and frame offset issues at the hardware level. A continuously running bitstream parsing and caching program is deployed on the edge terminal. It subscribes to both video streams in parallel and sets up an independent circular frame buffer queue. The program uses the hardware timestamp as the matching index rule, selecting only visible light and infrared frames with completely identical timestamps to form original frame pairs. It directly discards single invalid images caused by frame drops or stuttering, avoiding target ghosting and contour splitting defects in subsequent fusion from the source. For the matched original frame pairs, a unified bilinear interpolation algorithm is used to perform size scaling. Compared to neighbor interpolation, this avoids pixel jaggedness, and compared to cubic interpolation, it significantly reduces edge computing power consumption. The scaled target size is uniformly adapted to network input specifications, solving data incompatibility issues caused by inconsistent native resolutions of different batches of cameras. After scaling, the program constructs an ordered linked list to store standardized bimodal image sequences. Each data set in the linked list is bound to four types of information: visible light image, infrared image, original timestamp, and camera channel identifier. It can be traversed in timestamp order and supplied to the preprocessing module. In all-weather, multi-scenario testing in the park, regardless of whether it is in the dark with no lighting, backlight, strong light, rain, snow, or fog, the device can stably output bimodal sequences with perfectly aligned timings, without frame matching failures due to ambient light or lens moisture. The acquisition logic is lightweight, and the embedded terminal achieves 30FPS high-definition streaming with no latency. It can be directly reused in UAV-borne binocular perception and road checkpoint dual-camera devices without modifying the underlying streaming interpolation logic.

[0030] Hardware Setup and Sensor Deployment: A binocular PTZ network camera supporting integrated camera functionality is deployed along the perimeter of key security monitoring areas. The camera hardware includes a 1080P starlight-level visible light CMOS sensor module and a 640×512 native resolution uncooled vanadium oxide infrared thermal imaging sensor module. The core control and computing unit of the system is an NVIDIA Jetson Orin edge AI computing terminal equipped with the TensorRT hardware acceleration library.

[0031] Data Acquisition and Synchronization: The system's underlying program pulls infrared and visible light video streams in real time via the RTSP protocol, setting the frame rate baseline to 30 FPS. Timestamps are synchronized using the onboard clock signal to precisely capture strictly aligned original image frames at the same moment, and the image size is uniformly interpolated and scaled to achieve the network standard input size of 640×640 pixels.

[0032] Specifically, the process of outputting the registered image pair includes: Through binocular joint calibration, the mapping homography transformation matrix between the infrared imaging plane and the visible light imaging plane is obtained, and the matrix parameters are encrypted and stored in the local configuration file of the edge device. Read the homography transformation matrix stored locally, perform pixel-level perspective remapping operation on the original infrared images in the sequence, and project the infrared image pixels onto the visible light unified imaging coordinate system; Correcting three types of geometric deviations—overall translation, rotation, and scaling—results in infrared-visible image pairs.

[0033] In this embodiment, the binocular camera equipment undergoes a unified binocular joint calibration operation before leaving the factory. The calibration tool is equipped with a checkerboard calibration board, and multiple sets of calibration images are simultaneously acquired under infrared and visible light perspectives. The homography transformation matrix between the two imaging planes is solved by jointly calculating intrinsic and extrinsic parameters. The matrix contains complete translation, rotation, and scaling mapping parameters. After calibration, the matrix is ​​written to the device's local hidden configuration file using an encrypted encoding method, and hardware-level read / write permissions are set. Only the built-in low-level preprocessing program has read permissions, preventing human tampering that could lead to registration failure. During each power-on initialization phase, the program automatically reads the encrypted matrix, eliminating the need for repeated on-site calibration and significantly reducing the labor costs of large-scale engineering deployment. The program traverses the standardized dual-modal image sequence, sequentially extracting each set of original infrared images. Using the read homography matrix as a reference, pixel-level perspective remapping is performed, performing matrix multiplication on the coordinates of each pixel in the infrared image, and projecting all infrared pixels onto the visible light imaging coordinate system, eliminating the field-of-view offset caused by the binocular installation baseline and lens focal length difference. The remapping operation synchronously completes global correction of three types of geometric deviations: translation, rotation, and scaling. There are no issues of missed local corrections or edge cropping. After correction, the pixel coordinates of the same real-world target are aligned at the sub-pixel level in both infrared and visible light images, generating high-precision registered image pairs. Each set of registered images synchronously records the corresponding calibration matrix version, facilitating subsequent maintenance and verification of registration accuracy. In scenarios involving large-scale rotation of the park's gimbal and lens zoom, the pre-stored homography matrix can continuously perform accurate perspective correction without secondary calibration. This solves the problems of modal feature misalignment and target tearing caused by insufficient registration accuracy in traditional fusion solutions, providing a spatially aligned image foundation for bilateral filtering and CLAHE adaptive enhancement.

[0034] Homography perspective transformation: The system control flow automatically calls the pre-installed homography transformation matrix (this parameter is generated by the factory's internal and external parameter hardware calibration or the previous algorithm calibration). The captured infrared image pixel matrix is ​​remapped through perspective to achieve sub-pixel-level precise spatial alignment between its field of view and physical feature points and the visible light image.

[0035] Low-light adaptive evaluation and enhancement: The detection algorithm continuously monitors the average pixel grayscale value of the current visible light image channel. If the average grayscale value is lower than the system's preset ambient low light threshold (determining that the system is at night, at dusk, or under low light conditions), the image enhancement branch is automatically activated, invoking bilateral filtering, the CLAHE algorithm, and gamma correction for processing. During this process, the contour textures of intruding vehicles and pedestrians hidden in dark areas are forcibly brightened and restored by the algorithm, while basic noise reduction and artifact removal operations are performed on the infrared image.

[0036] Specifically, the process of performing single-modal image quality optimization includes: Bilateral filtering is performed on the infrared-visible image pair, and the denoised visible image is segmented to obtain local sub-image blocks; A limited contrast adaptive histogram equalization algorithm is used to process noise in dark areas, while bilinear interpolation is used to process the blocky boundary traces caused by splicing. The average gray value of the entire visible light image is statistically analyzed in real time. Based on the average gray value, the gamma correction coefficient is dynamically adjusted to reshape the overall gray value distribution of the image and brighten the dark areas and hidden target textures in nighttime and backlight environments. Neighborhood thresholding is performed on paired infrared images, and the two optimized images are integrated to generate an image with enhanced image quality.

[0037] In this embodiment, bilateral filtering noise reduction is preferentially performed on each registered image pair. Unlike traditional Gaussian filtering, which relies solely on spatial distance to smooth pixels, bilateral filtering combines pixel spatial distance and grayscale difference as dual weights. While filtering out sensor circuit noise and lens grain noise, it fully preserves the edges of pedestrians, walls, and vehicles, preventing edge blurring and loss of detail. The denoised visible light image is uniformly divided into non-overlapping local sub-image blocks. A limited contrast adaptive histogram equalization algorithm is applied to each block. The algorithm has a built-in histogram amplification limitation mechanism to prevent simultaneous amplification of noise in dark areas. After the sub-block equalization is completed, bilinear interpolation is used to smooth the block stitching boundaries, eliminating the blocky marks of light and dark segmentation in the image and ensuring a smooth and natural grayscale transition throughout the entire image. The program accumulates the visible light grayscale values ​​of the entire image pixel by pixel, calculating the global average grayscale in real time to determine the ambient light intensity: a low average grayscale indicates nighttime, backlighting, or low light in the shade, automatically increasing the gamma correction coefficient to reshape the grayscale distribution and fully brightening the outlines of pedestrians, non-motorized vehicles, and small intrusion devices in dark areas; a high average grayscale indicates strong daylight, automatically reducing the correction coefficient to prevent overexposure and loss of detail in bright areas. Simultaneously, neighborhood thresholding is performed on paired infrared images, traversing the grayscale of each pixel's surrounding neighborhood, thresholding to filter ground and vegetation thermal reflection noise, and fully preserving the heat source features of people and vehicles. After independent optimization of the visible light and infrared paths, they are integrated to generate an enhanced image pair. The optimized image noise is significantly reduced, while visible light texture and infrared heat source information are fully preserved, greatly reducing the computational load of the backend YOLO11 dual-stream network for noise reduction. In low-light, foggy, and backlight monitoring scenarios, this preprocessing workflow can stably increase the proportion of effective image information and avoid the overexposure defects of traditional global histogram equalization.

[0038] Specifically, the process of encapsulating image data using the channel dimension includes: Linearly map the original pixel grayscale values ​​of enhanced visible light and infrared images to a floating-point normalized range; Visible light and infrared images are encapsulated into a three-dimensional tensor format with preset height, width, and number of channels, and stored using floating-point pixels; The visible light and infrared tensors are combined and encapsulated to obtain a dual-modal tensor data packet.

[0039] In this embodiment, after the image enhancement image pair is generated, the program reads all the original pixel grayscale values ​​of visible light and infrared respectively, performs linear mapping operations, and uniformly converts the original integer pixel range to a standard floating-point normalized range. This reduces the pixel value distribution shift caused by differences in exposure and imaging gain between different cameras, and avoids modal feature weight imbalance in the dual-stream backbone network. After the linear mapping is completed, a three-dimensional floating-point tensor is constructed according to the YOLO11 network's preset fixed height, width, and number of channels: a multi-channel RGB floating-point tensor is encapsulated for visible light, and a single-channel grayscale floating-point tensor is encapsulated for infrared. All pixels are uniformly stored in floating-point format, fully preserving subtle textures and weak heat source differences in dark areas, and avoiding the loss of fine-grained information due to integer quantization truncation. After the two independent three-dimensional floating-point tensors are encapsulated, modal identifiers, temporal indexes, and resolution label headers are uniformly superimposed and combined into a standardized dual-modal tensor data packet. The data packet structure is regular and can be directly sent into the dual-stream backbone network without the need for additional backend size conversion and numerical normalization preprocessing, significantly saving the inference time of the embedded terminal. JetsonOrin edge terminals have limited computing power. This encapsulation process only includes basic operations such as linear mapping and dimension reorganization, without complex matrix calculations, and will not cause computing power overload. Tensor data generated in various environments such as daytime, rain, snow and nighttime are stable in distribution, without numerical overflow or dimension disorder faults, and can be seamlessly connected to the C3k2 nested bottleneck backbone network.

[0040] Encapsulation Tensorization: Converts the preprocessed image data into a standard PyTorch tensor format, generating a pair of input tensors (Shape:) and (Shape:) for channel matching.

[0041] Specifically, the process of extracting bottleneck features through nesting includes: The visible light and infrared tensors are input into the initial convolutional layer. Standard convolution is used to perform channel dimensionality upsampling and initial image downsampling, and shallow basic features are output. A lightweight, cross-stage local bottleneck structure is adopted to equally split the input features; Based on the splitting results, the branch output features are spliced ​​and fused along the channel dimension to generate a downsampled original feature map. The hierarchical original feature map generated by the branch network is output, and the sampled hierarchical features are input into the spatial pyramid pooling module.

[0042] In this embodiment, the standardized bimodal tensor data packet is split into two independent input streams: visible light and infrared. These are fed into the independent initial convolutional layers of the dual-stream decoupled backbone network, respectively. The structural parameters of the two convolutional layers are completely identical. Channel upscaling and initial image downsampling are performed synchronously through standard convolution: convolutional upscaling expands feature channels to extract shallow visual features such as edges and light spots; initial downsampling compresses spatial resolution and removes redundant pixels, reducing the computational load of subsequent multi-layer bottlenecks. The two shallow features are completely decoupled and do not interfere with each other, preventing premature mixing of modal information and disruption of their respective data distributions. The shallow features enter a lightweight C3k2 cross-stage local bottleneck structure. Each bottleneck layer equally splits the input feature channels, and two parallel branches perform operations: one branch retains the original shallow features without complex transformations, while the other branch performs deep abstract feature extraction through small nested bottleneck convolutions, balancing feature representation capability and parameter quantity, and avoiding the computational cost of large convolutional kernels. The output features of the two branches are spliced ​​and fused along the channels, and then the redundant channels are compressed by convolution. The original feature maps with decreasing resolution are generated by downsampling layer by layer, and the layered feature maps P1-P5 are output layer by layer, which correspond to shallow texture, middle detail and global semantic information respectively.

[0043] Application of C3k2 feature extraction module: In each downsampling and feature extraction stage of the backbone network (layers P1 to P5), this network fully upgrades the C2f or CSP modules in the early YOLO model to the C3k2 module.

[0044] Algorithm Principle: The C3k2 module is a more flexible and computationally efficient cross-stage local neural network implementation. It abandons the single large convolutional kernel, instead employing multiple smaller convolutional kernels in nested bottleneck layers. The module first processes the input features through a standard Conv block, then splits it into two channels. One half retains the original information, while the other half undergoes deep feature extraction through a series of n optimized bottleneck layers. Finally, the outputs of the two parts are concatenated along the channel dimension and fused through a final convolutional block.

[0045] Technical advantages: This design significantly reduces feature redundancy and computational cost while expanding the spatial receptive field and improving the model's ability to capture spatial information from small targets. This allows YOLO11 to surpass the mAP of its predecessors despite reducing the number of parameters by 22%.

[0046] After five convolutional downsampling steps with a stride of 2, the infrared and visible light branches respectively generate multi-scale feature pyramid sets with gradually decreasing resolution but increasing channel dimension: Visible light feature set: , Infrared feature set: , in, These correspond to feature maps at scales of 1 / 2 and 1 / 4, respectively. Corresponding to The deep feature maps at each scale are the core data for subsequent cross-scale fusion and object detection. Each level of downsampled hierarchical features are simultaneously fed into the SPPF spatial pyramid pooling module: fast max pooling is performed at multiple scales, and the features are stacked and concatenated across channels. This allows the network to quickly aggregate global contextual information from different receptive fields, effectively addressing the problem of drastic changes in target size in security scenarios. This provides the data for subsequent multi-receptive field context aggregation and C2PSA parallel spatial attention computation. Feature extraction is based on an improved YOLO11C3k2 architecture, reducing the number of parameters by 22% compared to YOLOv8m, and adapting to 30FPS real-time inference at edges. Binocular features are extracted independently from two paths, preventing the loss of visible light textures and infrared heat sources due to overlap. In scenarios with dense small targets, such as pedestrian walkways and vehicle entrances / exits, the hierarchical features completely preserve the contours and deep semantics of small targets without premature feature decay.

[0047] Specifically, the process of aggregating multi-receptive field global context information and simultaneously integrating it into a parallel spatial attention module includes: Multi-size max pooling operations are performed using the spatial pyramid pooling module to obtain global scene features under the corresponding receptive field. The global scene features are split across stages and spatial location features are extracted through parallel multi-group convolutions. The spatial response weights are calculated using vector dot products, and the spatial weight matrix is ​​generated using a normalized exponential function. The spatial weight matrix and the original pooling features are used to perform pixel calculations to obtain the feature responses of the infrared heat source target and the dense visible light texture region in the region of interest. By integrating weighted features, a multi-scale feature pyramid set of visible light and infrared light is generated.

[0048] In this embodiment, the original features layered from P1 to P5 are fed into the SPPF spatial pyramid pooling module. The module performs multi-size max pooling in parallel, with different pooling sizes corresponding to different receptive fields: small pooling captures local small target details, while large pooling captures the global context of the entire scene. The pooling results from multiple receptive fields are stacked and stitched together to fully integrate local textures and the global environment, solving the pain point that security scenarios have a large target scale span and a single receptive field cannot cover both large and small targets. The stitched global features are split across stages, and multiple sets of parallel convolutions independently extract the spatial location features of each pixel, accurately distinguishing foreground target pixels from invalid background pixels such as sky and empty ground. The spatial features output by each set of convolutions are subjected to vector dot product operations to calculate the spatial response weight of each pixel. The weight value represents the proportion of effective target information at that position. All weights are fed into the Softmax normalized exponential function to generate a standardized spatial weight matrix. The weight matrix is ​​multiplied pixel-by-pixel with the original multi-receptive field pooling features. High-weight amplification feature responses are applied to infrared human body, vehicle heat source pixels, and visible light object texture pixels, while low-weight suppression of redundant information is applied to large areas of vegetation and wall clutter, automatically filtering out worthless background interference. After weighted hierarchical feature integration, two sets of multi-scale feature pyramids with one-to-one correspondence between visible light and infrared levels are constructed respectively, providing a scale-matched dual-modal feature source for subsequent cross-scale ODConv dynamic gating fusion. In open, long-distance monitoring scenarios in parks, this mechanism automatically focuses on distant, small pedestrians and non-motorized vehicles, significantly suppressing large-area background noise and remarkably improving the recognition of small target features.

[0049] Specifically, the process of generating single-scale basic fusion features includes: Select a single visible light feature map and a single infrared feature map, and directly stitch them together based on the channel dimension to generate a fused hybrid feature tensor; Global average pooling is performed on the mixed feature tensor, and weights are assigned to the corresponding channels. By employing a joint operation of global max pooling and average pooling, the corresponding spatial features are extracted to generate a two-dimensional spatial attention matrix; The channel weights, spatial weights, and hybrid features are multiplied element-wise, and multi-dimensional feature interaction is performed through full-dimensional dynamic convolution to output single-scale basic fusion features.

[0050] In this embodiment, each pair of matching levels of the visible light and infrared feature pyramids is traversed, and visible light and infrared feature maps of the same resolution are directly spliced ​​along the channels to generate a hybrid feature tensor. The tensor simultaneously carries information from the visible light texture channel and the infrared heat source channel, preserving the independent distribution patterns of the two modalities and preventing the premature loss of any modal key features. The hybrid tensor performs global average pooling to calculate adaptive channel weights. In nighttime and backlight scenes, the weight of the infrared heat source channel is automatically increased, while in strong daylight, the weight of the visible light texture channel is automatically increased, adaptively balancing the proportion of dual-modal information. Simultaneously, global max and average pooling operations are performed to combine the advantages of the two pooling methods to extract two-dimensional spatial distribution features and generate a two-dimensional spatial attention matrix to distinguish foreground and background regions. The channel weights and spatial attention matrix are simultaneously multiplied element-wise with the hybrid tensor to complete dual adaptive weighting, filtering out conflict noise caused by modal differences. The weighted hybrid features are fed into ODConv full-dimensional dynamic convolution. The dynamic convolution adaptively adjusts the convolution kernel parameters of each channel and pixel to deeply mine the correlation features between visible light texture and infrared heat source, eliminate the defects of modal heterogeneity and fragmentation, and output the single-scale basic fusion features corresponding to the level. In park vehicle entrances and exits and densely populated scenarios, single-scale fusion features can simultaneously preserve vehicle outlines and human body heat sources, without one modality suppressing another. ODConv dynamic interaction is lightweight and has no excessive floating-point operations, ensuring real-time edge processing. The splicing, dual-pooling weighting, and dynamic convolution processes are based on the same-scale dynamic gating fusion logic, without custom weight adjustment rules.

[0051] Specifically, the process of outputting cross-scale fusion features includes: The feature resolutions are aligned, and the gradient details of high-level features and current layer features are extracted using the Gaussian-Laplacian pyramid operator. The structural similarity values ​​of the two layers of features are calculated by dividing the image into blocks. We employ dilated convolution with multiple dilation coefficients in parallel to extract multi-receptive field context and address edge blurring and small target artifacts caused by scale switching. Based on the cross-scale reconstruction results at the corresponding level, cross-scale enhanced fusion features are output.

[0052] In this embodiment, after each layer of single-scale basic fusion features is generated, the higher-resolution high-level fusion features of the previous layer are retrieved, and the system extracts the higher-level single-scale fusion features. Upsampling is performed using bilinear interpolation to make its resolution match the current layer features. Consistency, denoted as Bilinear interpolation upsampling completes resolution alignment, ensuring pixel-by-pixel comparison between high-level global semantic features and current-layer detail features. After resolution alignment, the Gaussian-Laplacian pyramid operator is used to extract gradient details from the high-level and current-layer layers respectively. The gradients fully represent the target edges and texture changes. The aligned two-layer features are evenly divided into blocks, and the Structural Similarity (SSIM) value is calculated for each block. , This represents the Laplacian gradient operator. This is an adjustable coefficient; Characterizing the similarity between two image patches in terms of brightness, contrast, and structure, among which... This represents the mean of x. The variance of x is represented by x. Denotes the covariance of x and y. This represents the stability constant.

[0053] Numerical values ​​represent the semantic consistency between two layers, establishing cross-scale semantic constraints. These constraints guide the reconstruction of lower-level details using high-level global semantics, preventing target contour breaks and semantic confusion during scale switching. Multiple sets of dilated convolutions with different dilation coefficients are deployed in parallel, expanding the receptive field without increasing the kernel size. The context of multiple receptive fields is extracted in parallel to specifically repair edge blurring, artifacts on small targets, and incomplete contours caused by scale switching, filling in the gaps in fine-grained information lost during cross-layer transmission. Under the combined effect of cross-scale semantic constraints and multi-dilated convolution context, hierarchical feature reconstruction is completed, fusing high-level global category semantics with the current layer's lower-level texture details to repair semantic gaps between scales and output cross-scale enhanced fusion features. In scenarios involving long-distance monitoring of small pedestrians and non-motorized vehicles in parks, this mechanism simultaneously considers high-level target classification semantics and lower-level small contours, addressing the pain points of traditional same-scale fusion methods such as loss of small target details and edge blurring.

[0054] Specifically, the process of lossless semantic communication includes: By constructing a two-way information flow path, the texture of object edges and small targets can be supplemented; Lightweight cross-stage local modules are deployed for feature upsampling and channel splicing nodes to control the amount of floating-point operations caused by channel expansion. By connecting a cross-stage parallel spatial attention module, the region of interest of the security target is automatically identified, and the target features are enhanced by secondary weighting to obtain lossless semantic flow of multimodal images.

[0055] In this embodiment, after all levels of cross-scale enhanced fusion features are completed, a PANet bidirectional information flow path is established: the top-down path transmits high-level global category semantics, guiding the lower level to distinguish between the target and the background; the bottom-up path transmits low-level high-resolution edges and minute target textures, compensating for the fine-grained attenuation of high-level downsampling. The bidirectional communication complements each other's complete object outlines and minor missing textures, enabling multi-level information interaction and preventing the loss of details in unidirectional information flow. All feature upsampling and channel splicing nodes in the network are uniformly deployed with C3k2 lightweight cross-stage local modules. When channel splicing leads to channel expansion, the module automatically compresses redundant channels and simplifies invalid features, strictly controlling the floating-point operation increment caused by channel expansion, avoiding insufficient computing power of embedded terminals and soaring inference latency. The lightweight architecture maintains high-definition 30FPS real-time processing throughout the entire process. All bidirectional flow paths are connected in series with C2PSA cross-stage parallel spatial attention modules at their ends. This automatically traverses all fused feature maps, identifies regions of interest for security targets such as pedestrians, vehicles, and surveillance equipment, distinguishes between blank backgrounds and foreground targets, and performs secondary weighted enhancement on target region features to further amplify target feature responses and suppress background clutter. After secondary weighting, the multi-layer fused features completely retain all semantic, texture, and heat source information, achieving lossless semantic flow with no semantic attenuation or loss of detail during multi-layer transmission. In scenarios with multiple occlusions such as trees, walls, and water mist in a park, the bidirectional paths complement each other to compensate for missing textures caused by occlusion, and the attention modules enhance residual target features under occlusion, significantly reducing the false negative rate for occluded targets.

[0056] Specifically, the process of optimizing image reconstruction, object classification, and bounding box regression tasks includes: The aggregated features are split into two independent computation branches: the first branch uses depthwise separable convolution lightweight processing to perform confidence classification of the target category; the second branch uses standard convolution to calculate the center point of the target bounding box and its length and width offsets. Three loss functions are constructed: target classification focus loss, bounding box coordinate regression loss, and image structure similarity reconstruction loss. The unified gradient balancing loss function dynamically adjusts the weighting coefficients of the three types of losses during the training phase to balance the backpropagation gradient direction conflicts between the two major tasks of image visual reconstruction and object detection, and outputs high-dimensional abstract fusion features after gradient optimization.

[0057] In this embodiment, the aggregated and fused features after lossless semantic flow are fed into the anchorless decoupled detection head. The detection head is split into two completely independent computation branches, with classification and regression operations not interfering with each other, avoiding mutual interference between features across tasks. The first classification branch uses depthwise separable convolution for lightweight dimensionality reduction throughout, splitting spatial and channel convolutions to significantly reduce floating-point computations, adapting to low computing power at the edge; after convolution processing, it outputs the confidence scores of various targets such as pedestrians, cars, trucks, and micro-drones, completing multi-class prediction. The second regression branch uses standard convolution to fully preserve the feature space coordinate association, accurately calculating the target center point and the length and width offset of the bounding box within the grid, ensuring positioning accuracy and preventing predicted box drift and size distortion. During the model training phase, three types of basic losses are constructed simultaneously: target classification focus loss to alleviate the imbalance between positive and negative samples in the scene and reduce the confidence shift caused by a large number of negative background samples; bounding box regression loss to constrain the coordinate deviation between the predicted box and the ground truth labeled box, improving positioning accuracy; and fused image SSIM reconstruction loss to constrain the fused features while inheriting visible light texture and infrared heat source distribution, ensuring the image quality of the fused image. The UnityGrad unified gradient balancing loss function inputs three types of losses. This function dynamically adjusts the weighting coefficients of the three types of losses in real-time based on the current training stage: in the early stages of training, it prioritizes increasing the weights of the reconstruction loss to ensure the basic image quality of the fusion; in the mid-stage, it balances the weights of the three types to simultaneously optimize reconstruction and detection; and in the later stages, it boosts the classification and regression weights to enhance detection performance. This perfectly solves the industry pain points of gradient cancellation during backpropagation and model convergence difficulties in both fusion reconstruction and object detection tasks. After iterative optimization of the gradient balancing loss, it outputs high-dimensional abstract fusion features, simultaneously meeting the dual requirements of high-definition image reconstruction and high-precision object detection, without sacrificing the performance of any single task.

[0058] Specifically, the process of outputting the cross-scale multimodal fused image includes: The optimized high-dimensional abstract fusion features are reversed by dimension mapping to obtain sub-pixel level micro-contour and fine texture information; Construct a mapping rule from abstract feature values ​​to the corresponding RGB pixels of the image to generate a cross-scale multimodal fusion image.

[0059] In this embodiment, the high-dimensional abstract fusion feature after gradient optimization is multi-layer convolutional downsampling compressed abstract data. The program initiates a dimension inverse mapping restoration operation, performing upsampling and channel inverse transformation layer by layer to restore the compressed feature to the original image input resolution. The inverse mapping process completely extracts sub-pixel micro-contours and fine textures, including the edges of small distant targets, subtle textures of objects, and weak heat source boundaries, without permanent loss of downsampling details or overall blurring of the fused image. After the dimension inverse mapping is completed, a fixed feature value-RGB pixel mapping rule is constructed based on the visible light texture and infrared heat source joint mechanism. High-texture areas are mapped to native visible light RGB colors, and high-heat source areas are mapped to infrared high-brightness grayscale. The rule is uniformly adapted to all monitoring scenarios, without needing to modify the mapping logic according to illumination or ambient temperature. The abstract feature values ​​are converted to standard RGB pixels pixel by pixel according to the mapping rule, generating a complete two-dimensional pixel matrix row by row, and outputting the final cross-scale multimodal fusion image. The fusion image has the advantages of rich color levels of visible light and high brightness of infrared heat sources in all weather conditions. The colors are clear during the day and the heat sources are prominent in foggy nighttime conditions, with no large-area noise, artifacts, or edge blurring. In the context of a large-screen visualization system for park security, fused images can be directly output for manual inspection without the need for post-processing image enhancement; subpixel detail restoration facilitates operators' long-distance identification of small, obscured intrusion targets.

[0060] Specifically, a cross-scale multimodal image fusion system includes: The multimodal acquisition and preprocessing unit is used to acquire real-time video streams from both infrared and visible light sources, standardize the data format, perform bilateral filtering and CLAHE gamma adaptive image enhancement, normalize the pixel value range, and construct a standardized dual-modal input tensor. The multi-scale feature extraction unit extracts multi-level features by using a built-in lightweight C3k2 nested bottleneck backbone network and SPPF spatial pyramid pooling and C2PSA parallel spatial attention submodule to hierarchically downsample the dual-modal tensor, aggregate the global context of multiple receptive fields, and generate a set of independent multi-scale feature pyramids for visible light and infrared light. The cross-scale feature fusion unit integrates dynamic convolution, Gaussian-Laplacian pyramid, and multi-dilated dilated convolution sub-modules to concatenate and dynamically gated feature channels of the same scale dual modality; it uses SSIM structural similarity to construct cross-scale semantic constraints, uses bidirectional PAN pathways to repair scale semantic faults, and outputs cross-scale fused features. The multi-task reconstruction inference unit, through the gradient balancing joint loss calculation module, simultaneously optimizes three types of tasks: image reconstruction, object classification, and bounding box regression. It performs sub-pixel inverse mapping pixel reconstruction on optimized high-dimensional features and outputs cross-scale multimodal fusion images.

[0061] In this embodiment, the multimodal acquisition and preprocessing unit is the data input front-end module of the entire system, fully implementing the preprocessing logic of binocular synchronous registration, multimodal collaborative enhancement, and tensor normalization. On the hardware side, it synchronously pulls visible light RGB and infrared grayscale bitstreams via the RTSP protocol, completes frame timing alignment using the shared hardware clock of the cameras, and discards invalid frames with timing misalignment. It reads the binocular monography transformation matrix encrypted and stored at the device's factory, performs pixel-level perspective remapping on the infrared image, completes sub-pixel spatial registration of the two types of imaging images, eliminates field-of-view offset caused by binocular baseline and lens focal length, and solves the problem of target misalignment and fusion tearing in modal images. At the algorithm level, a collaborative enhancement scheme is employed, setting up illumination determination logic: real-time statistical analysis of the global average grayscale of the visible light image; automatic activation of a combined process of bilateral filtering + CLAHE restricted histogram equalization + AGQE gamma adaptive enhancement for low-light and nighttime scenes; bilateral filtering removes sensor noise while fully preserving the edges of pedestrians and vehicles; CLAHE suppresses local exposure blocky artifacts; and AGQE dynamically adjusts the gamma coefficient to brighten dark areas and conceal targets; in daylight scenes with strong light, only basic noise reduction is performed to avoid overexposure and distortion. Finally, the enhanced visible light and infrared pixels are linearly normalized, encapsulated into a fixed-size dual-channel 3D tensor, and the standardized bimodal tensor is output and transmitted to the downstream feature extraction unit. Through pre-processed hardware-level image enhancement, the computational overhead of noise reduction in the backend backbone network is significantly reduced, addressing the pain points of insufficient feature fidelity and excessive backend network load under low illumination from the source.

[0062] Multi-scale Feature Extraction Unit: This unit is the core of the system's feature extraction. It implements dual-stream decoupled extraction using an improved C3k2 lightweight backbone architecture. Unlike traditional single-channel image stitching for feature extraction, it assigns two completely independent C3k2 feature extraction branches to visible light and infrared images, avoiding interference between the two modalities. Each branch performs layered downsampling (P1-P5) through multiple rounds of nested bottleneck convolution, extracting shallow texture, mid-level details, and global semantic features layer by layer. The C3k module employs a channel-split nested convolution design, reducing the number of parameters by 22% compared to a YOLOv8 model of the same scale. This perfectly adapts to the computing power constraints of edge devices, solving the problems of large model parameters and high edge inference latency. The SPPF spatial pyramid pooling module is connected in series at the deepest level of the backbone network. It aggregates global scene information from multiple receptive fields through multi-size max pooling. Then, it is connected to the C2PSA parallel spatial attention submodule. The pixel spatial response weights are calculated using vector dot product. The spatial weight matrix is ​​generated by Softmax normalization. Feature weighting is automatically applied to infrared heat source regions and visible light texture-dense regions to suppress ineffective background noise. Two independent multi-scale feature pyramid sets are generated for visible light and infrared, respectively, providing hierarchical and scale-matched dual-modal feature materials for cross-scale fusion.

[0063] Cross-scale feature fusion unit: This unit carries the core innovative cross-scale fusion algorithm of this invention, fully implementing the complete logic of ODConv dynamic gating fusion, Gaussian-Laplacian SSIM semantic constraints, multi-dilated dilated convolution, and PAN bidirectional information flow. First, visible light and infrared features of the same resolution are stitched together along the channels. Channel and spatial dual attention weights are generated through global averaging / max pooling. Combined with full-dimensional dynamic convolution ODConv, adaptive gating fusion is completed to balance the proportion of information from the two modalities and eliminate feature conflicts caused by modal heterogeneity. Then, upsampled features from higher layers are retrieved and aligned with the resolution of the current layer features. Gradient information is extracted through the Gaussian-Laplacian operator, and SSIM structural similarity is calculated to establish cross-scale semantic constraints, distinguishing effective target textures from scale switching artifacts. Multiple sets of dilated convolutions with different dilation coefficients are used to expand the receptive field and repair the problems of blurred edges and incomplete contours of small targets. Finally, relying on the PANet bidirectional information pathway, global semantics are transmitted from top to bottom and fine textures of the underlying layer are supplemented from bottom to top, opening up the information interaction channel between high and low scales, repairing the semantic gaps caused by traditional single-scale fusion, and outputting cross-scale fusion features with complete global semantics and details of small targets, thus solving the core defects of insufficient cross-level information interaction and missed detection of small targets in existing fusion networks.

[0064] Multi-task reconstruction inference unit: This unit completes multi-task synchronous optimization and final image reconstruction, matching the anchorless decoupled detection head, Unity gradient balancing loss, and sub-pixel reconstruction design. Feature fusion is split into two independent computational branches: a depthwise separable convolutional lightweight classification branch outputs the confidence scores of targets such as pedestrians and vehicles, while a standard convolutional regression branch accurately calculates the center and width / length offset of the target bounding boxes. The two tasks are decoupled to avoid feature interference. During training, a self-developed UnityGrad unified gradient balancing loss is used, dynamically adjusting the weighting coefficients of the classification focus loss, bounding box regression loss, and image SSIM reconstruction loss to balance the gradient conflicts during backpropagation of image reconstruction and object detection, solving the problems of gradient cancellation and convergence difficulties in traditional multi-task models.

[0065] UnityGrad joint gradient balance loss function It detects target loss. Similarity loss for multimodal reconstruction Dynamic weighted combination.

[0066] , Target detection loss : By L BCE Confidence classification loss, L CIoU Bounding box coordinate regression loss and L DFL The distributed focal loss composition ensures that the position and size of the detection box accurately approximate the true value. This is a hyperparameter used to adjust the proportion of each loss in the total loss.

[0067] , Multimodal similarity reconstruction loss In order to force the fusion of features Simultaneously inheriting the high-frequency texture details of visible light images and the low-frequency heat source saliency of infrared images, this invention calculates structural similarity loss, pixel-level mean square error, and gradient loss along the feature dimension. The specific definitions are as follows: , Among them, α1, α2, and α3 are adjustable weight parameters.

[0068] Loss: Calculates the structural similarity between the fused features and the visible light features. This involves the pixel mean. Standard deviation Calculation of block covariance This is an adjustable weighting parameter. This ensures that the fused features are aligned with the high-resolution visible light image within the structural framework.

[0069] Loss: Calculates the pixel-level mean square error between the fused features and the infrared features. This term ensures that the activation intensity of the heat source target in the fused image remains consistent with that in the infrared image, highlighting the heat source characteristics.

[0070] Gradient loss: Further enhances the learning of fine edge gradients in visible light images, making the texture of the target equally clear at night.

[0071] in, Optimize binary and multi-class prediction to improve classification confidence and reduce false alarms; , This represents the total number of positive and negative sample anchor points involved in the calculation. This represents the number of target categories detected. For the first Each sample belongs to the ground truth label of category c.

[0072] , which is the predicted probability value output by the network classification branch after Sigmoid activation.

[0073] : Optimizes the center point, width and height, and aspect ratio of the bounding box to improve the absolute accuracy of target spatial positioning, suitable for dense targets; Structural similarity constraints (visible light dominant) ensure that the feature distribution framework, local texture, and visible light source are highly consistent. / Pixel energy constraint (infrared dominant); Edge gradient constraints enhance the transmission of high-frequency information and prevent artifacts and blurring during the fusion process.

[0074] Among them, the complete intersection-union ratio loss of bounding boxes is: L CIoU Used for bounding box position regression, it penalizes both the Euclidean distance and aspect ratio difference of the center point on top of the traditional IoU, improving the localization accuracy of dense small targets: , , and These represent the predicted bounding box and the ground truth bounding box, respectively. b=(x,y) and b gt =(x gt ,y gt ) are the center coordinates of the predicted bounding box and the ground truth bounding box, respectively. represents the Euclidean distance. c is the diagonal length of the smallest bounding rectangle that can simultaneously contain both the predicted and ground truth boxes. v is used to measure the consistency of the aspect ratio. For dynamic balancing weight parameters, and These represent the width-to-height ratios of the predicted bounding box and the ground truth bounding box, respectively.

[0075] Distribution focus loss: L DFL The YOLO architecture models continuous bounding box offsets as discrete probability distributions. DFL forces the network to quickly focus the probability quality on discrete intervals near the true target value, improving the regression performance for blurry edges and small targets. , y represents the position label of the continuous real border. Satisfy (y) i With y i+1 For adjacent discrete grid division integers, typically y i+1 =y i+1 ).

[0076] With S i+1 Let be the discrete probability value predicted by the network regression branch after Softmax activation, and satisfy: , S i The essence of this formula is (y i+1 -y) and (yy) i) represents the cross-entropy loss between adjacent sides of the weight.

[0077] (Exponential mapping): using the base of the natural constant Non-linear amplification of the score: Ensure that all values ​​are strictly positive. ); Amplify the gap between the highest-scoring item and the lowest-scoring item.

[0078] (Normalized denominator): Discrete division of all possible partitions on this channel.

[0079] By dividing the numerator by the denominator, the predicted values ​​for the discrete interval are strictly constrained to the range (0, 1), and the sum of the probabilities of all discrete intervals is guaranteed to be 1. (Right now This forms a valid discrete probability distribution.

[0080] Infrared-dominated pixel energy loss: MSE ir / L pixel Calculate the pixel-level mean square error (MSE) between the multimodal fusion features and the input infrared features, and constrain the fusion features to retain the significant thermal radiation energy intensity of the infrared modes: , H, W, and C represent the height, width, and number of channels of the feature map, respectively. F fused (c,h,c,w) is the fused feature tensor generated after processing by the Cross-Scale Multimodal Fusion Module (CSMF). F ir (c,h,w) represents the original infrared feature tensor output by the backbone network's infrared single branch at the corresponding scale.

[0081] By combining the above innovative mechanisms, the network can automatically balance gradient conflicts, thereby avoiding a situation where improved fusion quality leads to a decrease in detection accuracy.

[0082] During the inference phase, subpixel inverse mapping is performed on the optimized high-dimensional fusion features to restore the minute contours and fine texture information lost in downsampling. The final all-weather fusion image is generated according to the pixel mapping rule of visible light color + infrared heat source. At the same time, NMS non-maximum suppression filters overlapping detection boxes and outputs visualized results with target annotations. The system is also equipped with an offline incremental fine-tuning mechanism, which iteratively updates the model based on difficult samples in the field during idle time to continuously adapt to environmental domain shifts caused by seasons and scenes.

[0083] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A cross-scale multimodal image fusion method, characterized in that, Includes the following steps: S1: Acquire multimodal image data, perform spatiotemporal synchronization and global perspective alignment on the multimodal image data, and output registered image pairs; perform single-modal image quality optimization based on the registered image pairs, and normalize the multimodal image pixels based on the optimization results; Image data is encapsulated using the channel dimension to generate a standardized multimodal image input tensor; S2: Based on the input tensor, multi-level resolution original modal feature maps are generated through nested bottleneck feature extraction; It aggregates global context information from multiple receptive fields and simultaneously integrates a parallel spatial attention module to output a multi-scale feature pyramid set for visible light and infrared. S3: Multi-scale features of infrared and visible light are stitched together along the channel and dynamically gated fusion is performed by dynamic convolution to generate single-scale basic fusion features; By calculating the structural similarity and gradient difference between hierarchical levels, cross-scale semantic constraints are established; global semantics and detailed textures are fused to repair scale semantic discontinuities and output cross-scale fused features. S4: Lossless semantic flow is achieved through bidirectional information flow. The fused features are input into the decoupled detection head. The image reconstruction, target classification, and bounding box regression tasks are optimized synchronously through joint gradient balancing loss. Based on the optimization results, sub-pixel pixel mapping reconstruction is performed, and a cross-scale multimodal fused image is output.

2. The method according to claim 1, characterized in that, The specific process of outputting the registered image pair includes: The real-time video streams of infrared and visible light are acquired through a real-time streaming protocol. The image frames are then bilinearly interpolated based on a preset timestamp to obtain a scaled image. The scaled and time-matched infrared and visible light paired images are integrated to obtain a standardized original bimodal image sequence; Through binocular joint calibration, the mapping homography transformation matrix between the infrared imaging plane and the visible light imaging plane is obtained, and the matrix parameters are encrypted and stored in the local configuration file of the edge device. Read the homography transformation matrix stored locally, perform pixel-level perspective remapping operation on the original infrared images in the sequence, and project the infrared image pixels onto the visible light unified imaging coordinate system; Correcting three types of geometric deviations—overall translation, rotation, and scaling—results in infrared-visible image pairs.

3. The method according to claim 1, characterized in that, The specific process of performing single-modal image quality optimization includes: Bilateral filtering is performed on the infrared-visible image pair, and the denoised visible image is segmented to obtain local sub-image blocks; A limited contrast adaptive histogram equalization algorithm is used to process noise in dark areas, while bilinear interpolation is used to process the blocky boundary traces caused by splicing. The average gray value of the entire visible light image is statistically analyzed in real time. Based on the average gray value, the gamma correction coefficient is dynamically adjusted to reshape the overall gray value distribution of the image and brighten the dark areas and hidden target textures in nighttime and backlight environments. Neighborhood thresholding is performed on paired infrared images, and the two optimized images are integrated to generate an image with enhanced image quality.

4. The method according to claim 1, characterized in that, The specific process of encapsulating image data using the channel dimension includes: Linearly map the original pixel grayscale values ​​of enhanced visible light and infrared images to a floating-point normalized range; Visible light and infrared images are encapsulated into a three-dimensional tensor format with preset height, width, and number of channels, and stored using floating-point pixels; The visible light and infrared tensors are combined and encapsulated to obtain a dual-modal tensor data packet.

5. The method according to claim 1, characterized in that, The specific process of extracting bottleneck features through nesting includes: The visible light and infrared tensors are input into the initial convolutional layer. Standard convolution is used to perform channel dimensionality upsampling and initial image downsampling, and shallow basic features are output. A lightweight, cross-stage local bottleneck structure is adopted to equally split the input features; Based on the splitting results, the branch output features are spliced ​​and fused along the channel dimension to generate a downsampled original feature map. The hierarchical original feature map generated by the branch network is output, and the sampled hierarchical features are input into the spatial pyramid pooling module.

6. The method according to claim 1, characterized in that, The specific process of aggregating multi-receptive field global context information and simultaneously integrating it into the parallel spatial attention module includes: Multi-size max pooling operations are performed using the spatial pyramid pooling module to obtain global scene features under the corresponding receptive field. The global scene features are split across stages and spatial location features are extracted through parallel multi-group convolutions. The spatial response weights are calculated using vector dot products, and the spatial weight matrix is ​​generated using a normalized exponential function. The spatial weight matrix and the original pooling features are used to perform pixel calculations to obtain the feature responses of the infrared heat source target and the dense visible light texture region in the region of interest. By integrating weighted features, a set of multi-scale feature pyramids for visible light and infrared light is generated.

7. The method according to claim 1, characterized in that, The specific process for generating single-scale basic fusion features includes: Select a single visible light feature map and a single infrared feature map, and directly stitch them together based on the channel dimension to generate a fused hybrid feature tensor; Global average pooling is performed on the mixed feature tensor, and weights are assigned to the corresponding channels. By employing a joint operation of global max pooling and average pooling, the corresponding spatial features are extracted to generate a two-dimensional spatial attention matrix; The channel weights, spatial weights, and hybrid features are multiplied element-wise, and multi-dimensional feature interaction is performed through full-dimensional dynamic convolution to output single-scale basic fusion features.

8. The method according to claim 1, characterized in that, The specific process of outputting cross-scale fusion features includes: The feature resolutions are aligned, and the gradient details of high-level features and current layer features are extracted using the Gaussian-Laplacian pyramid operator. The structural similarity values ​​of the two layers of features are calculated by dividing the image into blocks. We employ dilated convolution with multiple dilation coefficients in parallel to extract multi-receptive field context and address edge blurring and small target artifacts caused by scale switching. Based on the cross-scale reconstruction results at the corresponding level, cross-scale enhanced fusion features are output.

9. The method according to claim 1, characterized in that, The specific process for lossless semantic communication includes: By constructing a two-way information flow path, the texture of object edges and small targets can be supplemented; Lightweight cross-stage local modules are deployed for feature upsampling and channel splicing nodes to control the amount of floating-point operations caused by channel expansion. By connecting a cross-stage parallel spatial attention module, the region of interest of the security target is automatically identified, and the target features are enhanced by secondary weighting to obtain lossless semantic flow of multimodal images.

10. The method according to claim 1, characterized in that, The specific process of optimizing image reconstruction, object classification, and bounding box regression tasks includes: The aggregated features are split into two independent computation branches: the first branch uses depthwise separable convolution lightweight processing to perform confidence classification of the target category; the second branch uses standard convolution to calculate the center point of the target bounding box and its length and width offsets. Three loss functions are constructed: target classification focus loss, bounding box coordinate regression loss, and image structure similarity reconstruction loss. The unified gradient balancing loss function dynamically adjusts the weighting coefficients of the three types of losses during the training phase to balance the backpropagation gradient direction conflicts between the two major tasks of image visual reconstruction and object detection, and outputs high-dimensional abstract fusion features after gradient optimization.

11. The method according to claim 1, characterized in that, The specific process of outputting the cross-scale multimodal fused image includes: The optimized high-dimensional abstract fusion features are reversed by dimension mapping to obtain sub-pixel level micro-contour and fine texture information; Construct a mapping rule from abstract feature values ​​to the corresponding RGB pixels of the image to generate a cross-scale multimodal fusion image.

12. A cross-scale multimodal image fusion system for implementing the method as described in any one of claims 1-11, characterized in that, include: The multimodal acquisition and preprocessing unit is used to acquire real-time video streams from both infrared and visible light sources, standardize the data format, perform bilateral filtering and CLAHE gamma adaptive image enhancement, normalize the pixel value range, and construct a standardized dual-modal input tensor. The multi-scale feature extraction unit extracts multi-level features by using a built-in lightweight C3k2 nested bottleneck backbone network and SPPF spatial pyramid pooling and C2PSA parallel spatial attention submodule to hierarchically downsample the dual-modal tensor, aggregate the global context of multiple receptive fields, and generate a set of independent multi-scale feature pyramids for visible light and infrared light. The cross-scale feature fusion unit integrates dynamic convolution, Gaussian-Laplacian pyramid, and multi-dilated dilated convolution sub-modules to concatenate and dynamically gated feature channels of the same scale dual modality; it uses SSIM structural similarity to construct cross-scale semantic constraints, uses bidirectional PAN pathways to repair scale semantic faults, and outputs cross-scale fused features. The multi-task reconstruction inference unit, through the gradient balancing joint loss calculation module, simultaneously optimizes three types of tasks: image reconstruction, object classification, and bounding box regression. It performs sub-pixel inverse mapping pixel reconstruction on optimized high-dimensional features and outputs cross-scale multimodal fusion images.

Images