Lighting adaptive multi-modal non-registered image unsupervised fusion method, system, device and medium

Abstract

The present application relates to a light adaptive multi-modal non-registered image unsupervised fusion method, system, device and medium. The present application generates modality adaptive weights through light distribution estimation for feature modulation and loss weighting; multi-scale common feature regression affine transformation is used to realize coarse registration, and then local deformation field is predicted by a fine registration network to complete pixel-level alignment; combined with multi-scale attention, edge enhancement and BiFPN enhancement of small target features, and a saliency map is generated. Six types of unsupervised collaborative optimization losses (light adaptive fusion, texture preservation, small target saliency consistency, cross-modal structure consistency, deformation field smoothing, and cycle consistency) are constructed to jointly train the registration and fusion network. The present application does not require artificial registration labels, realizes light adaptive fusion, effectively preserves small targets, and is suitable for low-altitude safety, automatic driving and other scenes.

Description

Technical Field

[0001] This invention belongs to the field of multimodal image processing and intelligent sensing technology, and particularly relates to an unsupervised fusion method, system, device, and medium for multimodal unregistered images with adaptive illumination. Specifically, this invention mainly solves the problem of high-quality fusion of multimodal images under complex and extreme illumination interference conditions caused by the lack of strict physical registration between heterogeneous sensors. It can be widely applied in computer vision systems that require multi-source sensing, such as low-altitude safety monitoring, autonomous driving, drone detection, and video surveillance. Background Technology

[0002] In modern intelligent sensing systems such as safety monitoring of the low-altitude economy, autonomous driving, and drone detection, multi-source sensing combining visible light and thermal infrared imagery is typically required. Visible light imaging devices, similar to human vision, can capture rich texture details and color information, but their image quality deteriorates sharply under low light, backlight, or inclement weather conditions. Infrared sensors, based on thermal radiation imaging, can penetrate darkness and camouflage to capture thermal targets in all weather conditions, but they generally lack texture details and have lower image resolution and contrast. Therefore, effectively fusing these two modalities to generate high-quality images that retain both thermal targets and clear textures is crucial.

[0003] Despite the progress made in deep learning fusion methods based on convolutional neural networks, generative adversarial networks, or transformers, the following shortcomings still exist:

[0004] First, registration is highly dependent on physical models, making it difficult to adapt to real-world physical scenarios. Existing deep learning fusion methods typically assume that the input images have already undergone perfect physical spatial registration. However, in real-world low-altitude or outdoor environments, different sensors often exhibit differences in parallax, field of view, and resolution, making it difficult to achieve strict alignment at the physical level. Traditional pre-registration followed by fusion methods operate independently, heavily relying on manually calibrated registration datasets. Once registration errors occur, the fusion process can lead to blurred edges or ghosting. Most existing image alignment methods only operate within a single modal setting, heavily relying on the similarity of distribution and appearance, making it difficult to directly bridge the cross-modal domain gap between infrared and visible light.

[0005] Second, they exhibit poor adaptability to strong interference and lack dynamic perception of lighting and semantics. Existing deep learning methods essentially rely primarily on statistical regularities learned from pixels or low-level features, lacking explicit modeling of high-level semantics and ambient lighting information. Due to their reliance on pixel-level constraints, they struggle to dynamically and rationally allocate the contributions of infrared and visible light based on the lighting distribution of complex scenes (such as strong backlighting or low light at night). Furthermore, the commonly used simple linear fusion strategies such as addition and stitching have poor adaptive capabilities, making it difficult to intelligently filter complementary information and easily resulting in the loss of target features under strong light interference.

[0006] Third, weak features of small targets are easily lost. In tasks such as low-altitude reconnaissance, small, slow-moving drones at long distances occupy only a few pixels in the image. In the current multimodal fusion downsampling or feature mapping process, these weak high-frequency features are easily filtered out as background clutter or noise, causing the features of small targets to be submerged and greatly affecting the accuracy of subsequent target detection.

[0007] To address the aforementioned issues, researchers have attempted to transform cross-modal registration into single-modal registration by generating pseudo-infrared images, or to guide the training of fusion networks through illumination-aware weights. However, these methods still fail to simultaneously solve the three core problems of pixel-level alignment under non-registration conditions, illumination-adaptive weight generation, and small target feature enhancement. Therefore, there is an urgent need for a multimodal image fusion method that can achieve illumination adaptation, pixel-level alignment, and high-fidelity preservation of weak target features without relying on manual registration labels and fusion ground truth. Summary of the Invention

[0008] The purpose of this invention is to provide an illumination-adaptive multimodal unregistered image unsupervised fusion method, system, device, and medium to solve the problems of existing visible light-infrared image fusion methods in real outdoor low-altitude scenes, such as strong dependence on manual registration, fixed fusion weights under complex illumination, easy weakening of high-frequency features of small targets, and easy ghosting of fusion results.

[0009] This invention, through the overall process of "scene illumination estimation - adaptive weight generation - coarse-to-fine registration - small target saliency enhancement - unsupervised fusion optimization", enables visible light images and infrared images to obtain fused images with thermal target saliency, texture clarity and structural consistency even if there are parallax, scale differences and local non-rigid deviations.

[0010] Unlike directly applying the publicly available gradient sparsity constraint, this invention modifies the fine registration optimization objective into an unsupervised collaborative optimization objective comprised of illumination adaptive fusion loss, texture preservation loss, small target significant consistency loss, cross-modal structural consistency loss, deformation field smoothing loss, and cyclic consistency loss. Specifically, the cross-modal structural consistency loss, deformation field smoothing loss, and cyclic consistency loss directly constrain the fine registration network, while the illumination adaptive fusion loss, texture preservation loss, and small target significant consistency loss indirectly constrain the fine registration network through the fusion result. This forms an illumination-target-registration collaborative fusion mechanism suitable for low-altitude small target scenarios.

[0011] The first technical solution of the present invention is the illumination-adaptive multimodal unregistered image unsupervised fusion method, which is characterized by including the following steps: Step S1: Image Acquisition and Preprocessing Acquire visible light images of the same scene and thermal infrared images and perform preprocessing; Step S2: Scene-aware adaptive illumination distribution estimation: Visible light images Input the illumination distribution estimation network to obtain the probability that the current scene belongs to a daytime environment. and the probability of belonging to a nighttime environment ,based on and Generate modal adaptive weights and ; , ; In the formula To prevent constants with a denominator of zero; Used to characterize the contribution of infrared modes in low-light or nighttime scenes. Used to characterize the contribution of visible light modes in daylight or scenes with clear textures; Step S3: Multi-scale common feature extraction and coarse image registration: Will and A multi-scale shared feature extraction network is input to obtain cross-modal shared features. Based on the shared features, affine transformation parameters are regressed to construct an affine matrix. ; using thermal infrared images For reference, the visible light image Perform an affine transformation to obtain a coarsely registered image. ; Step S4: Small target feature enhancement and modality-specific feature extraction: Coarsely registered images With thermal infrared images The input modality-specific feature extraction component performs feature extraction and enhancement. This component includes a multi-scale attention module, an edge enhancement branch, and a bidirectional feature pyramid network (BiFPN). The multi-scale attention module, based on... and Dynamically adjust the channel weights and spatial weights of different modalities to construct a saliency map of the small target. The BiFPN performs bidirectional fusion of shallow high-resolution features and deep semantic features. Step S5: Fine-matching and joint training based on unsupervised collaborative optimization loss: Coarsely registered visible light images With infrared images Input a fine-registration network to predict the local deformation field from the visible light mode to the infrared mode. And an aligned visible light image is obtained through a spatial transformation layer. ;Will and A common input stitching and fusion network is used, and the stitching and fusion network is based on illumination weights. and Small target salience map In addition to modal-specific features, output fused images ; The fine registration network and the stitching and fusion network are jointly trained using unsupervised collaborative optimization loss, which includes illumination adaptive fusion loss. Texture preservation loss Significant Consistency Loss of Small Objectives Cross-modal structural consistency loss Deformation field smoothing loss and cycle consistency loss Output fused image .

[0012] As a preferred embodiment: the illumination adaptive fusion loss Represented as: Used to constrain the fused image to adaptively retain infrared thermal information and visible light texture information under different lighting conditions; In the formula, To merge images, For thermal infrared images, Visible light image; The texture preservation loss Represented as: Used to constrain the fused image to retain more prominent edges and high-frequency textures in both modalities; In the formula, This represents the gradient operator, and `max` indicates taking the maximum value element by element. To fuse the gradient maps of the images, This is the gradient map of the finely registered visible light image. The gradient of the thermal infrared image.

[0013] As a preferred option: the small objective has a significant consistency loss. Represented as: ; In the formula, This represents element-wise multiplication. This represents a mask for a salient area of ​​a small target. These are the weighting coefficients; Used to simultaneously preserve infrared thermal salience and visible light edge detail within a small target area, where This is used to assign higher weights to the areas where weak targets such as drones, small aircraft, or small heat sources are located; the small target salient area mask is generated by infrared thermal anomaly response, visible light local contrast, edge response, candidate target scale prior or unsupervised saliency detection network;

[0014] Preferably: the cross-modal structural consistency loss Represented as: This is used to ensure that the aligned visible light image and infrared image maintain consistency in modality-invariant structural features. In the formula, Φ represents the structural feature extraction function, Φ(·) is the operator used to extract modality-invariant structural features; SSIM represents the structural similarity index; The deformation field smoothing loss Represented as: Used to limit unreasonable abrupt changes in the deformation field; In the formula, and These represent the difference operators along the horizontal and vertical directions, respectively; Represents the horizontal displacement field. Represents the displacement field in the vertical direction; The cycle consistency loss Represented as: Used to constrain forward and reverse deformations as approximately inverse transformations of each other; In the formula, Visible light image, , This represents the inverse deformation field, where I is the unit mapping.

[0015] As a preferred option, the overall training loss is expressed as: ; In the formula, to The weights are adjustable; the total loss is optimized simultaneously through backpropagation of the fine registration network and the stitching and fusion network; cross-modal structural consistency loss. Deformation field smoothing loss and cycle consistency loss This imposes a direct constraint on the fine registration network, with an adaptive fusion loss for illumination. Texture preservation loss Significant consistency loss with small objectives By fusing images The quality feedback indirectly constrains the fine registration network.

[0016] Preferably, the illumination distribution estimation network employs a lightweight convolutional network, a visual transformer, or a convolutional-attention hybrid network, and its input is one or more of the following: a global brightness histogram, a local contrast map, and deep semantic features of the image. and Under conditions of strong light, backlight, or low illumination, it is not directly used as a simple pixel weighting coefficient, but as an adjustment factor for subsequent feature enhancement, attention allocation, and loss function weights, thereby avoiding the loss of target features caused by fixed fusion rules; The multi-scale shared feature extraction network obtains cross-modal shared features through shared convolution, correlation matching, or cross attention; the affine transformation is used to correct global translation, rotation, scaling, and shearing differences. The Bidirectional Feature Pyramid Network (BiFPN) performs bidirectional fusion of shallow high-resolution features and deep semantic features, which enhances the edges, corners and high-frequency textures of small targets during multi-scale propagation. The preprocessing includes size normalization, grayscale or luminance channel extraction, strong noise suppression, and dynamic range normalization. For input images with different resolutions, the visible light image is scaled to the same spatial size as the thermal infrared image, using the thermal infrared image as a reference.

[0017] The second technical solution of the present invention is the illumination-adaptive multimodal unregistered image unsupervised fusion system, which is characterized by including: The image acquisition and preprocessing module is used to acquire visible light images and thermal infrared images of the same scene and perform preprocessing. The illumination distribution estimation module is used to input the visible light image into the illumination distribution estimation network to obtain the probability that the scene belongs to a daytime environment. and the probability of belonging to a nighttime environment and based on and Generate modal adaptive weights and ; The multi-scale common feature extraction and coarse registration module is used to extract cross-modal common features and regress affine transformation parameters based on the common features. Using the thermal infrared image as a reference, the visible light image is subjected to affine transformation to obtain a coarse registration image. The small target feature enhancement and modality-specific feature extraction module is used to extract and enhance features from coarsely registered images and thermal infrared images through a multi-scale attention module, an edge enhancement branch, and a bidirectional feature pyramid network, and generate a small target saliency map. The fine registration-fusion joint training module is used to input coarse registration visible light images and infrared images into the fine registration network to predict the local deformation field, obtain the aligned visible light image, and then input the aligned visible light image and infrared image into the stitching and fusion network to output the fused image. The module is then jointly trained by unsupervised collaborative optimization of the loss. Image output module, used to output fused image.

[0018] Preferably, the unsupervised collaborative optimization loss includes an illumination adaptive fusion loss. Texture preservation loss Significant Consistency Loss of Small Objectives Cross-modal structural consistency loss Deformation field smoothing loss and cycle consistency loss .

[0019] The third technical solution of the present invention is the electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The special feature is that when the processor executes the program, it implements the steps of the illumination-adaptive multimodal unregistered image unsupervised fusion method of any of the preceding claims.

[0020] The fourth technical solution of the present invention is the computer-readable storage medium storing a computer program thereon, wherein the program, when executed by a processor, implements the steps of the illumination-adaptive multimodal unregistered image unsupervised fusion method of any of the preceding claims.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) End-to-end unsupervised fusion without manual registration labels. This invention achieves pixel-level alignment and feature fusion of unregistered multimodal images without any manually labeled registration labels or ground truth images. It overcomes the bottleneck of cascading errors caused by traditional registration-then-fusion techniques.

[0022] (2) Dynamic adjustment of illumination-adaptive weights to improve fusion quality under complex lighting conditions. This invention explicitly models scene lighting conditions through an illumination distribution estimation network, generates modality-adaptive weights, and uses these weights as adjustment factors for feature enhancement, attention allocation, and loss function weights. This enables the fusion network to dynamically and rationally allocate the contributions of infrared and visible light according to the illumination distribution of complex scenes, overcoming the defect of traditional fixed fusion rules losing target features under strong light interference.

[0023] (3) Significantly enhanced features of small targets improve the fidelity of weak targets. This invention enhances the edges, corners, and high-frequency textures of small targets during multi-scale propagation through a combination of multi-scale attention modules, edge enhancement branches, and bidirectional feature pyramid networks. At the same time, through significant consistency loss for small targets, both infrared thermal saliency and visible light edge details are maintained within the small target region, effectively solving the problem that high-frequency features of small targets are easily filtered out as background clutter in existing multimodal fusion methods.

[0024] (4) Collaborative optimization mechanism to achieve joint optimization of illumination-target-registration. This invention extends the fine registration optimization objective to an unsupervised collaborative optimization objective composed of six types of losses. Among them, structural alignment, deformation smoothing, and cycle consistency directly constrain the fine registration network, while fusion quality indirectly constrains the fine registration network. This forms an illumination-target-registration collaborative fusion mechanism suitable for low-altitude small target scenarios, enabling each module to promote each other and optimize together during training. Attached Figure Description

[0025] Figure 1 This is the overall flowchart of the illumination-adaptive multimodal unregistered image unsupervised fusion method of the present invention;

[0026] Figure 2 This is a schematic diagram of the unsupervised collaborative optimization loss calculation module of the present invention. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] Example 1: An unsupervised fusion method for multimodal unregistered images with illumination-adaptive characteristics.

[0029] An illumination-adaptive unsupervised fusion method for multimodal unregistered images includes the following steps: Step S1: Image Acquisition and Preprocessing Acquire visible light images of the same scene and thermal infrared images In this embodiment, the visible light image is acquired by a visible light camera, and the thermal infrared image is acquired by a thermal infrared sensor. Both are installed on the same platform but have certain parallax and field of view differences. Preprocessing is performed on the two acquired images: First, the visible light image and the thermal infrared image are normalized in size. Using the spatial size of the thermal infrared image as a reference, the visible light image is scaled to the same width and height as the thermal infrared image. Second, the luminance channel of the visible light image is extracted (if it is a color image, it is converted to YCbCr or Lab color space and the luminance component is extracted), and aligned with the grayscale channel of the thermal infrared image. Then, median filtering or bilateral filtering is used for strong noise suppression. Finally, dynamic range normalization is performed on the two images, mapping the pixel values ​​to the [0,1] interval. Step S2: Scene-aware adaptive illumination distribution estimation: Preprocessed visible light image and thermal infrared images The input is an illumination distribution estimation network. In this embodiment, the illumination distribution estimation network adopts a lightweight convolutional neural network structure with shared weights in two paths, including: a first convolutional layer (3×3 kernel size, stride 1, 32 output channels) for extracting shallow features; a first pooling layer (max pooling, 2×2 kernel size, stride 2) for downsampling; a second convolutional layer (3×3 kernel size, stride 1, 64 output channels); a second pooling layer (max pooling, 2×2 kernel size, stride 2); a third convolutional layer (3×3 kernel size, stride 1, 128 output channels); a global average pooling layer for compressing the feature map into a feature vector; a first fully connected layer (64 output dimensions); a second fully connected layer (2 output dimensions), followed by a Softmax activation function, outputting... and : Based on the probability P of daytime environment N The probability P of a nighttime environment D ,based on and Generate modal adaptive weights and : ; ; In the formula, To prevent constants with a denominator of zero, =10 -6 ; Used to characterize the contribution of infrared modes in low-light or nighttime scenes. Used to characterize the contribution of visible light modes in daylight or scenes with clear textures; When the scene is during the day Approaching 1, Approaching 0, the fusion process focuses more on the texture information of the visible light image; when the scene is at night, Approaching 1, Approaching 0, the fusion process focuses more on the thermal target information of the infrared image; when the scene is under transitional lighting conditions such as dusk, dawn, or backlighting, and Both are at the intermediate value, and the contribution of the two modes is dynamically balanced according to the illumination conditions.

[0030] Under conditions of strong light, backlighting, or low illumination and Instead of being directly used as a simple pixel weighting coefficient, it serves as a modulating factor for subsequent feature enhancement, attention allocation, and loss function weights. Specifically, in the modality-specific feature extraction stage, and Used to adjust the channel weights and spatial weights of different modalities in the multi-scale attention module; during the loss function calculation stage, and Weighted calculation of the illumination-adaptive fusion loss L_illum.

[0031] Step S3: Multi-scale common feature extraction and coarse image registration; Preprocessed visible light image and thermal infrared images The input is a multi-scale shared feature extraction network. In this embodiment, the multi-scale shared feature extraction network adopts a two-branch Siamese network structure, with the two branches sharing weights. Each branch includes: a first convolutional module (7×7 kernel size, stride 2, 64 output channels, followed by batch normalization and ReLU activation) for extracting shallow edge features; a first residual module (containing two 3×3 convolutions, 64 output channels) for further refining shallow features; a second residual module (containing two 3×3 convolutions, 128 output channels, stride 2) for extracting mid-level texture features; and a third residual module (containing two 3×3 convolutions, 256 output channels, stride 2) for extracting high-level structural features.

[0032] The features at corresponding levels of the two branches are relevance matched and fused with cross-attention to obtain cross-modal shared features; specifically, let the first... The visible light branch output characteristics of the layer are The thermal infrared branch output characteristics are For each layer of features through respectively Convolutional mapping yields query features Key features Sum value characteristics And calculate the cross-modal correlation matrix. ,in For the first The number of feature channels in each layer is then used to obtain the attention weights through Softmax normalization. Finally, attention weights are applied to the feature values. Enhanced features obtained by infrared guidance The enhanced shared features are obtained. ,in This indicates concatenation along the channel dimension. This indicates the convolution fusion operation.

[0033] Shared features across multiple scales After fusion, affine transformation parameters are predicted using a regression network; these parameters include: rotation angle θ, horizontal translation tx, vertical translation ty, and horizontal scaling factor. Vertical scaling factor and shearing factor The affine matrix H is represented as: ; thermal infrared images For reference, the visible light image Applying the above affine transformation yields a coarsely registered image. In the formula This indicates a coarse registration based on the affine matrix H transformation operation; this coarse registration is used to correct global translation, rotation, scaling, and shearing differences, providing a good initialization for subsequent local fine registration. Step S4: Small target feature enhancement and modality-specific feature extraction; Coarsely registered images With thermal infrared images Input modality-specific feature extraction component; this component includes a multi-scale attention module, an edge enhancement branch, and a bidirectional feature pyramid network (BiFPN); The multi-scale attention module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule performs global average pooling and global max pooling on the input features, generates channel attention weights through a shared multilayer perceptron, and then... and Dynamic adjustment of channel weights: The spatial attention submodule performs average pooling and max pooling on the input features along the channel dimension, concatenates them, and generates spatial attention weights through a convolutional layer, which are also dynamically adjusted according to the illumination weights; The edge enhancement branch uses the Sobel or Laplacian operator to extract the edge response map of the image, and then enhances the features of the edge region through a trainable edge enhancement convolutional layer to make the contour information of small targets more prominent.

[0034] BiFPN performs bidirectional fusion of shallow high-resolution features and deep semantic features. Specifically, BiFPN includes top-down and bottom-up paths, introducing learnable weights at each fusion layer to perform weighted fusion of features at different scales. Through multi-scale propagation of BiFPN, the edge, corner, and high-frequency texture features of small targets such as long-range drones, small aircraft, and small heat sources are enhanced.

[0035] Constructing a saliency map of small goals In this embodiment, The saliency map is generated jointly by infrared local thermal anomaly response and visible light local contrast. Specifically, the infrared local thermal anomaly response is obtained by calculating the difference between each pixel in the infrared image and the mean of its neighborhood; the visible light local contrast is obtained by calculating the standard deviation within the neighborhood of each pixel in the visible light image. After normalization, both are weighted and fused to obtain the saliency map of the small target. , The high-value regions in the data correspond to regions that may contain small targets, and are used to increase the weight of these small target regions in subsequent registration and fusion loss. Step S5: Fine registration-fusion joint training based on unsupervised collaborative optimization loss to coarsely register visible light images With infrared images The input is a fine registration network; in this embodiment, the fine registration network adopts the U-Net architecture, including an encoder-decoder structure. The encoder extracts multi-scale features through progressive downsampling, and the decoder restores spatial resolution through upsampling and skip connections, finally outputting a two-dimensional deformation field of the same size as the input image. (Including displacement in both horizontal and vertical directions).

[0036] Through the spatial transformation layer, based on the deformation field For coarsely registered visible light images Pixel-level resampling is performed to obtain the aligned visible light image. In the formula This indicates a grid sampling operation (i.e., a differentiable image distortion operation).

[0037] Aligned visible light image With infrared images The image is stitched along the channel dimension and input into the stitching and fusion network. In this embodiment, the stitching and fusion network adopts an encoder-decoder architecture. The encoder extracts multi-scale features through convolution and pooling operations, and the decoder restores the resolution through upsampling and skip connections, finally outputting a single-channel fused image. Illumination weights are introduced in each layer of the encoder and decoder. and and small target saliency map As a modulation factor, it guides the fusion and reconstruction of features; like Figure 2 As shown, the unsupervised collaborative optimization loss calculation module simultaneously receives the fused image. Aligned visible light image Thermal infrared images and deformation field Calculate six types of loss: (1) Illumination adaptive fusion loss : ; In the formula, I f To fuse images, I i For thermal infrared images, I v Visible light image; The loss function constrains the fused image to adaptively retain infrared thermal information and visible light texture information under different lighting conditions; when the scene is at night, Larger images tend to retain infrared information when fused; when the scene is daytime, Larger images tend to retain visible light textures when merged. (2) Texture preservation loss : ; In the formula, This represents the gradient operator (including the gradient in the horizontal and vertical directions), and `max` represents taking the maximum value element by element. To fuse the gradient maps of the images, This is the gradient map of the finely registered visible light image. The gradient of the thermal infrared image is used; this loss function constrains the fused image to retain more significant edge and high-frequency texture information in both modalities by maximizing the gradient preservation strategy. (3) Significant Consistency Loss of Small Objectives : ; In the formula, This represents element-wise multiplication. This represents a mask for a salient area of ​​a small target. Indicates the weighting coefficient; Let be the value of the mask for the salient region of a small target at pixel (i,j); used to simultaneously preserve infrared thermal saliency and visible light edge details within the small target region, where This method assigns higher weights to areas containing small targets such as drones, small aircraft, or tiny heat sources. The small target salient region mask is generated by infrared thermal anomaly response, visible light local contrast, edge response, candidate target scale prior, or an unsupervised saliency detection network. This loss assigns higher weights to areas containing small targets while constraining the fused image to maintain infrared thermal saliency and visible light edge details within these areas, thus prioritizing the preservation of small target features during the fusion process. (4) Cross-modal structural consistency loss : ; In the formula, Φ represents the structural feature extraction function, Φ(·) is the operator used to extract modality-invariant structural features; SSIM represents the structural similarity index; it is used to extract relatively stable cross-modality edge, contour, gradient direction, local phase consistency or shallow convolution structural features from the input image; the loss constraint ensures that the aligned visible light image and infrared image are consistent in modality-invariant structural features, thereby reducing ghosting, edge breakage and structural blur caused by local misalignment; (5) Deformation field smoothing loss : The deformation field smoothing loss Represented as: Used to limit unreasonable abrupt changes in the deformation field; In the formula, and These are the difference operators along the horizontal and vertical directions, respectively; Represents the horizontal displacement field. This represents the vertical displacement field; by constraining the spatial gradient of the deformation field, this loss limits unreasonable abrupt changes in the deformation field, ensuring the smoothness and physical rationality of the deformation field. (6) Cyclic consistency loss : ; In the formula, Visible light image, , Represents the inverse deformation field, where I is the unit mapping; this loss constraint ensures that the forward and inverse deformations are approximately inverse transformations of each other, guaranteeing the reversibility and consistency of the deformation field and preventing folding or degradation phenomena.

[0038] The overall training loss is expressed as: ; In the formula, to The weights are adjustable; the total loss is optimized simultaneously through backpropagation of the fine registration network and the stitching and fusion network; cross-modal structural consistency loss. Deformation field smoothing loss and cycle consistency loss This imposes a direct constraint on the fine registration network, with an adaptive fusion loss for illumination. Texture preservation loss Significant consistency loss with small objectives By fusing images The quality feedback indirectly constrains the fine registration network; In this embodiment, the values ​​of each weight coefficient are: λ1=1.0, λ2=0.8, λ3=0.5, λ4=0.3, λ5=0.1, λ6=0.1; the above values ​​can be adjusted according to specific application scenarios and training data. The total loss is optimized by backpropagation simultaneously using the parameters of the fine registration network and the stitching and fusion network. , and The output (deformation field) directly affects the fine registration network. (and aligned images), directly constraining the fine registration network; illumination adaptive fusion loss Texture preservation loss and cross-modal structural consistency loss Deformation field smoothing loss By fusing images The quality feedback indirectly affects the learning of the fine registration network; therefore, the fine registration network is not driven solely by the single gradient sparsity of the fusion result, but is jointly guided by structural alignment, deformation rationality, cycle consistency and fusion quality, forming a lighting-target-registration collaborative fusion mechanism. During training, the Adam optimizer was used, with an initial learning rate set to 1×10⁻⁶. -4 The batch size is set to 4, and the number of training epochs is set to 200. The learning rate decays to 0.5 times the current value every 50 epochs. Step S6: Output the fused image; The fused image obtained after the above steps As the final output, the fused image retains the saliency of thermal targets in the infrared image and contains clear texture details in the visible light image. It also maintains good structural consistency across the entire image and does not exhibit ghosting or edge blurring. The fused image can be directly used for subsequent visual tasks such as target detection, recognition, tracking, or semantic segmentation.

[0039] Example 2: Illumination-adaptive multimodal unregistered image unsupervised fusion system.

[0040] This embodiment provides an illumination-adaptive multimodal unregistered image unsupervised fusion system, including the following modules: The image acquisition and preprocessing module is used to acquire visible light and thermal infrared images of the same scene, and perform size normalization, grayscale or brightness channel extraction, strong noise suppression, and dynamic range normalization on the two images. The input of this module is the original visible light and thermal infrared images, and the output is a preprocessed normalized image pair; The illumination distribution estimation module is used to input the preprocessed visible light image into the illumination distribution estimation network to obtain the probability that the current scene belongs to a daytime environment. and the probability of belonging to a nighttime environment and based on and Modal adaptive weights are generated according to the following formula: ; ; In the formula, To prevent constants with a denominator of zero; Used to characterize the contribution of infrared modes in low-light or nighttime scenes. Used to characterize the contribution of visible light modes in daylight or scenes with clear textures; The output of this module is modal adaptive weights. and ; The multi-scale shared feature extraction and coarse registration module is used to input preprocessed visible light and thermal infrared images into the multi-scale shared feature extraction network to extract shallow edge features, mid-level texture features, and high-level structural features, respectively. Cross-modal shared features are obtained through shared convolution, correlation matching, or cross-attention. Based on these shared features, affine transformation parameters are regressed to construct an affine matrix. Using the thermal infrared image as a reference, an affine transformation is performed on the visible light image to obtain a coarsely registered image. The output of this module is a coarsely registered visible light image.

[0041] The small target feature enhancement and modality-specific feature extraction module is used to input coarsely registered images and thermal infrared images into the modality-specific feature extraction component, and then, through the multi-scale attention module, based on... and The channel and spatial weights of different modalities are dynamically adjusted, edge responses are enhanced through edge enhancement branches, and shallow high-resolution features and deep semantic features are fused bidirectionally through a Bidirectional Feature Pyramid Network (BiFPN); simultaneously, a small target saliency map is constructed. The output of this module is enhanced modality-specific features and small target saliency maps; The fine registration-fusion joint training module is used to input coarse registration visible light images and infrared images into the fine registration network to predict the local deformation field from the visible light mode to the infrared mode. And an aligned visible light image is obtained through a spatial transformation layer. ;Will and A common input stitching and fusion network is used, and the stitching and fusion network is based on illumination weights. , Small target salience map In addition to modal-specific features, output fused images Joint training is performed using unsupervised collaborative optimization loss, which includes illumination adaptive fusion loss. Texture preservation loss Significant Consistency Loss of Small Objectives Cross-modal structural consistency loss Deformation field smoothing loss and cycle consistency loss The output of this module is the trained fine-registration network and stitching fusion network, as well as the fused image; Image output module, used to output fused image. .

[0042] Example 3: Electronic device.

[0043] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the program, it implements the steps of the illumination-adaptive multimodal unregistered image unsupervised fusion method as described in Embodiment 1.

[0044] The electronic device can be a general-purpose computer, a dedicated image processing workstation, an embedded system, an onboard processing unit for unmanned aerial vehicles (UAVs), or an onboard computing platform for autonomous vehicles. The memory can be random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or solid-state drive, etc. The processor can be a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), or field-programmable gate array (FPGA), etc.

[0045] Example 4: Computer-readable storage medium.

[0046] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the illumination-adaptive multimodal unregistered image unsupervised fusion method as described in Embodiment 1.

[0047] The computer-readable storage medium can be any tangible medium capable of storing program code, including but not limited to: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk, solid-state drive, etc.

[0048] Example 5: Fusion effect under different lighting conditions: This embodiment demonstrates the fusion effect of the present invention under different lighting conditions: In daylight scenes with strong light: visible light images have clear textures and rich colors, but some areas are overexposed; infrared images show prominent thermal targets but lack texture details. This invention uses an illumination distribution estimation network to output p day ≈0.95, p night ≈0.05, generated ≈0.95 ≈0.05, the fused image retains visible light texture details while supplementing the thermal target information of overexposed areas with infrared information.

[0049] Backlit scenes: In visible light images, target areas are insufficiently bright and details are lost due to backlighting; in infrared images, thermal targets are clear. This invention identifies the uneven illumination characteristics of backlit scenes through an illumination distribution estimation network and generates appropriately balanced illumination. and (Approximately 0.5 each), the fused image uses more infrared information in the target area and more visible light texture in the background area, achieving illumination-adaptive fusion.

[0050] In low-light nighttime scenes: visible light images are severely degraded, making targets almost undetectable; however, thermal targets are clearly visible in infrared images. This invention utilizes an illumination distribution estimation network to output p... day ≈0.02, p night ≈0.98, generated ≈0.98 With a resolution of ≈0.02, the fused image mainly retains infrared thermal target information while making the most of the weak texture information in the visible light image to achieve high-quality fusion in nighttime scenes.

[0051] Example 6: Fusion effect under non-registration conditions: This embodiment demonstrates the fusion effect of the present invention under non-registration input conditions: When there are translations, rotations, scale differences, or local non-rigid deformations between visible light and infrared images, traditional methods of registration followed by fusion produce obvious ghosting and edge blurring. This invention corrects global differences through coarse registration (affine transformation) and then uses a fine registration network to predict local deformation fields for pixel-level alignment. Combined with constraints from cross-modal structural consistency loss, deformation field smoothing loss, and cyclic consistency loss, it can achieve accurate pixel-level alignment without the need for manual registration labels.

[0052] Experimental results show that even when the input image has a large parallax (horizontal offset of up to 20 pixels and rotation angle of up to 5°), the present invention can still output a fused image with no ghosting and clear edges. The structural consistency index (such as SSIM) of the fused image is improved by more than 15% compared with the traditional method, and the recall rate of small target detection is improved by more than 12%.

[0053] Creative discourse: 1. The syllogistic argument of technical problem—technical means—technical effect: Technical issues: Existing visible-infrared image fusion methods suffer from three major drawbacks—high registration dependence (requiring manual labeling of registration points), poor illumination adaptability (fixed fusion weights), and easy loss of small target features. These three issues are interconnected: registration errors cause ghosting, fixed weights lead to target loss under illumination interference, and downsampling causes small target features to be obscured.

[0054] Technical means: This invention adopts a complete process of "illuminance estimation - weight generation - coarse registration - fine registration - small target enhancement - collaborative optimization" as the core of the unsupervised collaborative optimization objective composed of six types of losses, and incorporates registration, fusion, illumination adaptation and small target enhancement into a unified framework for joint optimization.

[0055] Technical effects: Pixel-level alignment can be achieved without manual registration labels; adaptive lighting weights enable the fusion strategy to be dynamically adjusted according to the scene; small target enhancement mechanism ensures high-fidelity preservation of features of weak and small targets; collaborative optimization of six types of losses enables each module to promote each other, and the three work together to achieve high-quality fusion under complex lighting and non-registration conditions.

[0056] 2. Demonstration of synergistic effects and systemic effects among features: There is a significant synergistic effect among the various technical features of this invention, rather than a simple additive effect: Synergy between illumination weights and registration: Illumination weights and It is used not only for weighting in the fusion stage, but also for attention modulation and weighted calculation of the loss function in the feature extraction stage. This means that when learning the deformation field, the registration network can perceive which modal features are more reliable under different lighting conditions, and thus adjust the alignment strategy—relying more on infrared structural information for alignment at night and more on visible light texture information for alignment during the day.

[0057] Synergy between Small Target Saliency Map and Loss: Small Target Saliency Map It also participated in the significant consistency loss of small objectives. Cross-modal structural consistency loss Deformation field smoothing loss The computation and splicing fusion network feature modulation. Significant consistency loss in small targets. middle, Directly used as weights for salient regions, enabling the network to preferentially preserve infrared thermal saliency and visible light edge details in areas with weak target saliency; in the fusion network, The network is guided to allocate more attention to small target regions. This dual effect strengthens and protects the features of small targets during both training and inference phases.

[0058] Synergy between fine registration and fusion: Fine registration networks and stitching fusion networks achieve significant consistency loss through small objectives. Perform joint optimization instead of independent training. Use cross-modal structural consistency loss. Deformation field smoothing loss and cycle consistency loss Directly constrain the quality of the deformation field, and use illumination-adaptive fusion loss. Texture preservation loss and cross-modal structural consistency loss Deformation field smoothing loss The deformation field is indirectly constrained by the quality of the fused images. This "direct + indirect" dual constraint mechanism enables the fine registration network to not only pursue the accuracy of geometric alignment, but also the optimal quality of the fused image, thus realizing a virtuous cycle in which registration serves fusion and fusion provides feedback to registration.

[0059] 3. Creativity of the overall technical solution Compared with the prior art, the overall technical solution of the present invention has the following inventive contributions: (1) This invention breaks through the paradigm of "registration first, then fusion". Existing methods treat registration and fusion as two independent stages, and registration errors are amplified in the fusion result through cascading. This invention enables the registration network to sense fusion quality feedback through joint optimization, fundamentally breaking the chain of cascading errors.

[0060] (2) This invention achieves a leap from "post-processing" to "pre-fusion" in illumination perception. Existing illumination adaptive methods mostly perform illumination compensation on the results after fusion, which is a passive adaptation. This invention puts illumination estimation in the pre-process, and the generated weights actively participate in the entire process of feature extraction, attention allocation and loss calculation, realizing the active guidance of illumination information on the fusion process.

[0061] (3) A three-in-one collaborative fusion framework of "illumination-target-registration" is constructed. Existing methods usually deal with the three problems of illumination adaptation, target enhancement and registration alignment in isolation. This invention incorporates the three into the same optimization framework through a unified loss function system, so that each module promotes each other during training and produces a system effect of "1+1+1>3".

[0062] In summary, this invention has significant inventiveness in terms of technical problems, technical means, and technical effects. The proposed illumination-adaptive multimodal unregistered image unsupervised fusion method, system, device, and medium provide a brand-new technical solution for the field of multi-source intelligent sensing.

[0063] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be covered by the claims of the present invention.

Claims

1. An illumination-adaptive multimodal unregistered image unsupervised fusion method, characterized in that, Includes the following steps: Step S1: Image Acquisition and Preprocessing Acquire visible light images of the same scene and thermal infrared images and perform preprocessing; Step S2: Scene-aware adaptive illumination distribution estimation: Visible light images Input the illumination distribution estimation network to obtain the probability that the current scene belongs to a daytime environment. and the probability of belonging to a nighttime environment ,based on and Generate modal adaptive weights and ; ； In the formula To prevent constants with a denominator of zero; Used to characterize the contribution of infrared modes in low-light or nighttime scenes. Used to characterize the contribution of visible light modes in daylight or scenes with clear textures; Step S3: Multi-scale common feature extraction and coarse image registration: Will and A multi-scale shared feature extraction network is input to obtain cross-modal shared features. Based on the shared features, affine transformation parameters are regressed to construct an affine matrix. ; using thermal infrared images For reference, the visible light image Perform an affine transformation to obtain a coarsely registered image. ; Step S4: Small target feature enhancement and modality-specific feature extraction: Coarsely registered images With thermal infrared images The input modality-specific feature extraction component performs feature extraction and enhancement. This component includes a multi-scale attention module, an edge enhancement branch, and a bidirectional feature pyramid network (BiFPN). The multi-scale attention module, based on... and The channel weights and spatial weights of different modalities are dynamically adjusted; the Bidirectional Feature Pyramid Network (BiFPN) performs bidirectional fusion of shallow high-resolution features and deep semantic features, enhancing the edges, corners, and high-frequency textures of small targets during multi-scale propagation; and a saliency map of the small target is constructed. , The saliency map of a small target is generated jointly by the infrared local thermal anomaly response and the visible light local contrast. Specifically, the infrared local thermal anomaly response is obtained by calculating the difference between each pixel in the infrared image and the mean of its neighborhood; the visible light local contrast is obtained by calculating the standard deviation of each pixel's neighborhood in the visible light image. After normalization, the two are weighted and fused to obtain the saliency map of the small target. ; Step S5: Fine-matching and joint training based on unsupervised collaborative optimization loss: Coarsely registered visible light images With infrared images Input a fine-registration network to predict the local deformation field from the visible light mode to the infrared mode. And an aligned visible light image is obtained through a spatial transformation layer. ;Will and Common input splicing and fusion network, based on illumination weights and Small target salience map In addition to modal-specific features, output fused images ; The fine registration network and the stitching and fusion network are jointly trained using unsupervised collaborative optimization loss, which includes illumination adaptive fusion loss. Texture preservation loss Significant Consistency Loss of Small Objectives Cross-modal structural consistency loss Deformation field smoothing loss and cycle consistency loss .

2. The illumination-adaptive multimodal unregistered image unsupervised fusion method according to claim 1, characterized in that, The illumination adaptive fusion loss Represented as: ； This is used to constrain the fused image to adaptively retain infrared thermal information and visible light texture information under different lighting conditions; In the formula, I f To fuse images, I i For thermal infrared images, I v Visible light image; The texture preservation loss Represented as: Used to constrain the fused image to retain more prominent edges and high-frequency textures in both modalities; In the formula, This represents the gradient operator, and `max` indicates taking the maximum value element by element. To fuse the gradient maps of the images, This is the gradient map of the finely registered visible light image. The gradient of the thermal infrared image.

3. The illumination-adaptive multimodal unregistered image unsupervised fusion method according to claim 1, characterized in that, The significant consistency loss of the small objective Represented as: ； In the formula, This represents element-wise multiplication. This represents a mask for a salient area of ​​a small target. These are the weighting coefficients; Used to simultaneously preserve infrared thermal salience and visible light edge detail within a small target area, where This is used to assign higher weights to the regions where small targets are located; the small target salient region mask is generated by infrared thermal anomaly response, visible light local contrast, edge response, candidate target scale prior or unsupervised saliency detection network.

4. The illumination-adaptive multimodal unregistered image unsupervised fusion method according to claim 1, characterized in that, The cross-modal structural consistency loss Represented as: ； This is used to ensure that the aligned visible light image and infrared image maintain consistency in modality-invariant structural features. The aligned visible light image; In the formula, This represents the structural feature extraction function. This is an operator used to extract mode-invariant structural features; SSIM stands for Structural Similarity Index; The deformation field smoothing loss Represented as: ； Used to limit unreasonable abrupt changes in the deformation field; In the formula, and These represent the difference operators along the horizontal and vertical directions, respectively; Represents the horizontal displacement field. Represents the displacement field in the vertical direction; The cycle consistency loss Represented as: ； This is used to constrain forward deformation and reverse deformation, which are approximately inverse transformations of each other; In the formula, This is a visible light image.

5. The illumination-adaptive multimodal unregistered image unsupervised fusion method according to claim 1, characterized in that, The overall training loss is expressed as: ； In the formula, to The weights are adjustable; the total loss is optimized simultaneously through backpropagation of the fine registration network and the stitching and fusion network; cross-modal structural consistency loss. Deformation field smoothing loss and cycle consistency loss This imposes a direct constraint on the fine registration network, with an adaptive fusion loss for illumination. Texture preservation loss Significant consistency loss with small objectives By fusing images The quality feedback indirectly constrains the fine registration network.

6. The illumination-adaptive multimodal unregistered image unsupervised fusion method according to claim 1, characterized in that, The illumination distribution estimation network employs a lightweight convolutional network, a visual transformer, or a convolutional-attention hybrid network, and its input consists of one or more of the following: a global brightness histogram, a local contrast map, and deep semantic features of the image. and Under conditions of strong light, backlight, or low illumination, it is not directly used as a simple pixel weighting coefficient, but as an adjustment factor for subsequent feature enhancement, attention allocation, and loss function weights, thereby avoiding the loss of target features caused by fixed fusion rules; The multi-scale shared feature extraction network obtains cross-modal shared features through shared convolution, correlation matching, or cross attention; the affine transformation is used to correct global translation, rotation, scaling, and shearing differences. The preprocessing includes size normalization, grayscale or luminance channel extraction, strong noise suppression, and dynamic range normalization. For input images with different resolutions, the visible light image is scaled to the same spatial size as the thermal infrared image, using the thermal infrared image as a reference.

7. An illumination-adaptive multimodal unregistered image unsupervised fusion system according to the illumination-adaptive multimodal unregistered image unsupervised fusion method of claim 1, characterized in that, include: The image acquisition and preprocessing module is used to acquire visible light images and thermal infrared images of the same scene and perform preprocessing. The illumination distribution estimation module is used to input the visible light image into the illumination distribution estimation network to obtain the probability that the scene belongs to a daytime environment. and the probability of belonging to a nighttime environment and based on and Generate modal adaptive weights and ; The multi-scale common feature extraction and coarse registration module is used to extract cross-modal common features and regress affine transformation parameters based on the common features. Using the thermal infrared image as a reference, the visible light image is subjected to affine transformation to obtain a coarse registration image. The small target feature enhancement and modality-specific feature extraction module is used to extract and enhance features from coarsely registered images and thermal infrared images through a multi-scale attention module, an edge enhancement branch, and a bidirectional feature pyramid network, and generate a small target saliency map. The fine registration-fusion joint training module is used to input coarse registration visible light images and infrared images into the fine registration network to predict the local deformation field, obtain the aligned visible light image, and then input the aligned visible light image and infrared image into the stitching and fusion network to output the fused image. The module is then jointly trained by unsupervised collaborative optimization of the loss. Image output module, used to output fused image.

8. The illumination-adaptive multimodal unregistered image unsupervised fusion system according to claim 7, characterized in that, The unsupervised collaborative optimization loss includes illumination adaptive fusion loss. Texture preservation loss Significant Consistency Loss of Small Objectives Cross-modal structural consistency loss Deformation field smoothing loss and cycle consistency loss .

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the illumination-adaptive multimodal unregistered image unsupervised fusion method as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the illumination-adaptive multimodal unregistered image unsupervised fusion method as described in any one of claims 1 to 6.

Images