A rattlesnake visual mechanism and illumination self-adaptive image fusion method and system

Abstract

This invention discloses an image fusion method and system based on rattlesnake visual mechanisms and illumination adaptation. The method includes: inputting a visible light image into an adaptive illumination perception network; the network extracts spatial distribution features through a coordinate attention mechanism and outputs a daytime or nighttime determination result; selecting a corresponding fusion model based on the determination result: the daytime model uses visible light as the primary signal and infrared as an auxiliary enhancement signal, while the nighttime model uses infrared as the primary signal and visible light as an auxiliary background signal, generating a grayscale fused image; converting the grayscale fused image and the visible light image to a color space decoupled from brightness and chromaticity; correcting the brightness channel of the grayscale fused image based on the brightness of the visible light image; compensating for high-frequency components in the corrected brightness channel and recombining it with the chromaticity channel of the visible light image; and outputting a color fused image through inverse transformation. This invention improves the fusion accuracy under complex illumination conditions, avoids color distortion and detail loss, and achieves a balance between the target and the background.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and image processing technology, and more specifically to an image fusion method and system based on rattlesnake vision mechanism and illumination adaptation. Background Technology

[0002] Infrared and visible light image fusion technology, by integrating the saliency of thermal radiation in infrared images with the textural details in visible light images, has demonstrated significant application value in fields such as autonomous driving, video surveillance, and all-weather visual perception. Infrared images can capture the thermal radiation of objects themselves, are all-weather, and have strong anti-interference capabilities, highlighting high-value thermal targets; visible light images provide high spatial resolution and rich textural details, highly compatible with human visual habits. Fusing the two can highlight key thermal radiation targets while preserving rich environmental textures and structural details.

[0003] Biological research has shown that rattlesnakes possess a highly developed bimodal perception mechanism: they receive visible light signals through both eyes, while simultaneously using their pit organs to precisely sense infrared thermal radiation, and achieving deep coordination of heterogeneous signals in the optic tectum region. Biomimetic fusion algorithms that simulate this mechanism are gradually becoming an important research direction due to their independence from massive training data and unique multimodal processing advantages.

[0004] In 1997, Waxman's team, based on a biological visual receptive field model, simulated the neural signal processing mechanism of viper bimodal cells and constructed the classic Waxman fusion network architecture, achieving feature-level fusion of visible light and infrared images. Subsequently, researchers made several improvements: Jing et al. broke through the single-band limitation by incorporating mid-wave infrared and long-wave infrared into the fusion framework; Li et al. proposed an improved algorithm based on biological visual enhancement mechanisms in 2016; Zhang et al. introduced "enhancing cells" and "inhibiting cells" models, effectively improving image edge sharpness; Wang et al. proposed a pseudo-color fusion model based on a visual imaging system in 2022, improving image naturalness and contrast by extracting common and unique information from the source images and simplifying the receptive field model; subsequently, a three-order neural network structure based on the visual receptive field was designed; and in 2024, Wang et al. further introduced a day / night dual-level adaptive mechanism, attempting to dynamically adjust the network to adapt to different lighting conditions.

[0005] However, existing technologies still have the following shortcomings when applied to complex real-world scenarios:

[0006] First, the method for determining illumination is rigid.

[0007] Existing biomimetic fusion algorithms (including the day-night dual-level adaptive mechanism proposed by Wang et al. in 2024) still use fixed pixel statistical thresholds to distinguish between day and night. For example, if the proportion of pixels with a value higher than 110 in a visible light grayscale image is less than 15%, it is determined to be nighttime; otherwise, it is determined to be daytime. This hard threshold determination method lacks the ability to adaptively perceive local lighting changes and complex environments (such as dawn / dusk transitions and interference from local strong light sources), resulting in low scene judgment accuracy and consequently, incorrect selection of fusion strategies.

[0008] Secondly, false color mapping leads to color distortion and loss of detail.

[0009] Existing methods (including the Waxman series and its improved algorithms) typically map the output of the fusion model directly to the RGB three channels or employ a simple global color transfer strategy when generating color fusion images. This approach results in fusion images with strange colors (such as cyan-purple artifacts), blurred textures, and crucial high-frequency details that are smoothed out or lost during the mapping process.

[0010] Third, the global fusion strategy lacks regional adaptability.

[0011] Existing biomimetic fusion methods generally employ globally uniform fusion rules, which cannot dynamically adjust fusion weights based on image content (target region and background region). In highly heterogeneous scenes, the significant difference between infrared targets and visible light backgrounds makes it difficult for the fusion results to take both into account; either the infrared target is not prominent, or the details of the visible light background are excessively masked. Summary of the Invention

[0012] In view of the above problems, the present invention is proposed to provide an image fusion method based on rattlesnake vision mechanism and illumination adaptation that overcomes or at least partially solves the above problems.

[0013] To achieve the above objectives, the present invention adopts the following technical solution: S1. Acquire visible light and infrared images of the same scene; S2. Input the visible light image into an adaptive illumination sensing network, extract and classify the illumination distribution features of the visible light image, and output the illumination environment determination result, which is either a daytime scene or a nighttime scene. Based on the lighting environment determination results, a corresponding fusion model is selected between the daytime fusion model and the nighttime fusion model. The visible light image and the infrared image are then input into the selected fusion model for fusion. If the daytime fusion model is selected, the visible light image is enhanced by an ON-central receptive field to obtain an enhanced visible light signal. The infrared image is processed by a dual-mode cell operator and then fed into an ON-central receptive field to obtain an infrared feature signal. The enhanced visible light signal and the infrared feature signal are input into the cell model together to output a daytime grayscale fusion image. If the nighttime fusion model is selected, the infrared image is enhanced by an ON-central receptive field to obtain an enhanced infrared signal. The visible light image is processed by a dual-mode cell operator and then fed into the ON-central receptive field to obtain a visible light background signal. The enhanced infrared signal and the visible light background signal are input into the cell model together to output a nighttime grayscale fusion image. S3. Convert the grayscale fusion image output in step S2 and the visible light image to a color space where luminance and chrominance are decoupled, respectively. Correct the luminance channel of the grayscale fusion image based on the luminance distribution of the visible light image, and reconstitute the high-frequency components in the corrected luminance channel with the chrominance channel of the visible light image. Output a color fusion image through inverse color space transformation.

[0014] Preferably, the dual-mode cellular operator includes an infrared-enhanced visible light operator, an infrared-suppressed visible light operator, a visible light-enhanced infrared operator, and a visible light-suppressed infrared operator. The infrared-enhanced visible light operator is used to simulate the enhancement effect of infrared signals on visible light signals, the infrared-suppressed visible light operator is used to simulate the suppression effect of infrared signals on visible light signals, the visible light-enhanced infrared operator is used to simulate the enhancement effect of visible light signals on infrared signals, and the visible light-suppressed infrared operator is used to simulate the suppression effect of visible light signals on infrared signals.

[0015] Preferably, the adaptive illumination sensing network includes a basic feature extraction layer, an attention enhancement layer, and a classification decision head connected in sequence; The basic feature extraction layer is used to perform convolution, batch normalization and nonlinear activation processing on the visible light image to output a preliminary feature map; The attention enhancement layer includes at least one level of coordinate attention module. Each level of coordinate attention module extracts global features in the horizontal direction and global features in the vertical direction from the input preliminary feature map, and generates horizontal attention weights and vertical attention weights respectively. The preliminary feature map is reweighted using the horizontal attention weights and the vertical attention weights, and a weighted feature map is output. The attention enhancement layer includes multiple levels of cascaded coordinate attention modules, and the input of the subsequent level of coordinate attention module is the weighted feature map output by the previous level of coordinate attention module. The classification decision head is used to perform global pooling and fully connected mapping on the weighted feature map finally output by the attention enhancement layer, and output the lighting environment determination result.

[0016] Preferably, each level of the coordinate attention module performs the following reweighting step on its input feature map: Step A: Perform one-dimensional global average pooling on the input feature map along the height direction to obtain the first direction aggregated feature. The dimension of the first direction aggregated feature is H×1×C, where H is the height dimension and C is the number of channels. Step B: Perform one-dimensional global average pooling on the input feature map along the width direction to obtain the second-direction aggregated feature. The dimension of the second-direction aggregated feature is 1×W×C, where W is the width dimension. Step C: The first direction aggregation feature and the second direction aggregation feature are concatenated in the spatial dimension. After the first convolution transformation, batch normalization and nonlinear activation, they are separated into the first branch feature and the second branch feature along the spatial dimension. Step D: The first branch features are transformed by a second convolution and activated by a sigmoid function to generate the horizontal attention weights, and the second branch features are transformed by a third convolution and activated by a sigmoid function to generate the vertical attention weights. Step E: Apply the horizontal attention weights and the vertical attention weights to the corresponding directions of the input feature map to complete the reweighting of the input feature map.

[0017] Preferably, the daytime fusion model includes a multi-channel feature preprocessing stage, a first-level ON-central receptive field antagonism stage, a second-level ON-central receptive field information refinement stage, and / or a cell decision fusion stage; The multi-channel feature preprocessing stage includes: processing the visible light image and the infrared image using infrared-enhanced visible light operator, infrared-suppressed visible light operator, visible light-enhanced infrared operator, and visible light-suppressed infrared operator to generate V∪IR, V∩IR, +IR←V, and ... IR←V; The visible light image is processed using an ON-center receptive field to generate an enhanced visible light signal ON_VIS; The first-level ON-central receptive field antagonism phase includes: feeding V∪IR into the central excitation region of the ON-central receptive field, feeding V∩IR into the surrounding inhibition region of the ON-central receptive field, generating +OR_AND; and then connecting +IR←V with... IR←V input ON-central receptive field, generating infrared enhancement map +IR; The second-level ON-central receptive field information refinement stage includes: inputting +OR_AND and +IR into the ON-central receptive field to generate the core feature map X1; The OR cell decision fusion stage includes: inputting ON_VIS and X1 into the OR cell model to generate the daytime grayscale fusion image.

[0018] Preferably, the nighttime fusion model includes a multi-channel feature preprocessing stage, a first-level ON-central receptive field antagonism stage, and a multi-level ON-central receptive field differential fusion stage; The multi-channel feature preprocessing stage includes: processing the visible light image and the infrared image using infrared-enhanced visible light operator, infrared-suppressed visible light operator, visible light-enhanced infrared operator, and visible light-suppressed infrared operator to generate V∪IR, V∩IR, +IR←V, and ... IR←V; The visible light image is processed using an ON-center receptive field to generate an enhanced visible light signal ON_VIS; The first-level ON-central receptive field antagonism phase includes: feeding V∪IR into the central excitation region of the ON-central receptive field, feeding V∩IR into the surrounding inhibition region of the ON-central receptive field, generating +OR_AND; and then connecting +IR←V with... IR←V input ON-central receptive field, generating infrared enhancement map +IR; The multi-level ON-central receptive field differential fusion stage includes: First level: +OR_AND and +IR are fed into the central excitation region and the surrounding inhibition region of the ON-central receptive field, respectively, to generate the first comprehensive feature map X1; Second level: Feed ON_VIS and +IR into the ON-central receptive field to generate the second comprehensive feature map X2; Third level: Feed X1 and X2 into the ON-central receptive field to generate the third comprehensive feature map X3; Input X1 and X3 into the cell model to generate the nighttime grayscale fusion image.

[0019] Preferably, in step S3, the color space where brightness and chromaticity are decoupled is: lαβ Color space; Correcting the brightness channel of the grayscale fused image based on the brightness distribution of the visible light image includes: Calculate the mean values ​​of the grayscale fused image and the visible light image in the luminance channel, respectively. , and standard deviation , Calculate the luminance components after migration :

[0020] in, For the grayscale fused image in lαβ The luminance channel component in a color space.

[0021] Preferably, the high-frequency components in the corrected luminance channel are compensated and then recombined with the chroma channel of the visible light image, and a color fused image is output after inverse color space transformation, including: A Gaussian low-pass filter with a standard deviation of σ is used. G (.) for the migrated luminance components Filtering is performed, and high-frequency texture residuals are extracted through differential operations. :

[0022] Introduce a preset enhancement coefficient The final brightness channel after compensation :

[0023] The final brightness channel With the visible light image in lαβ Chroma channel components in a color space Reorganization, after lαβ Inverse color space transformation outputs a color-blended image. :

[0024] in, This represents the inverse color space transformation function.

[0025] Based on the same inventive concept, this invention also discloses an image fusion system based on rattlesnake visual mechanisms and illumination adaptation, used to perform the above-described image fusion method, including: The image acquisition module is used to acquire visible light and infrared images of the same scene; An adaptive illumination sensing module is used to extract and classify the illumination distribution features of the visible light image and output the illumination environment determination result for daytime or nighttime scenes. The fusion model selection module is used to select the corresponding fusion model between the daytime fusion model and the nighttime fusion model based on the lighting environment determination result. The daytime fusion module is used to, when the daytime fusion model is selected, enhance the visible light image through an ON-central receptive field to obtain an enhanced visible light signal, process the infrared image using a dual-mode cell operator and feed it into the ON-central receptive field to obtain an infrared feature signal, input the enhanced visible light signal and the infrared feature signal into the cell model together, and output a daytime grayscale fusion image. The night fusion module is used to, when the night fusion model is selected, enhance the infrared image by passing it through an ON-central receptive field to obtain an enhanced infrared signal, process the visible light image using a dual-mode cell operator and feed it into the ON-central receptive field to obtain a visible light background signal, input the enhanced infrared signal and the visible light background signal into the cell model together, and output a night grayscale fusion image. The color reconstruction module is used to convert the grayscale fused image and the visible light image to color spaces where luminance and chrominance are decoupled, respectively. It corrects the luminance channel of the grayscale fused image based on the luminance distribution of the visible light image, and reconstructs the high-frequency components in the corrected luminance channel with the chrominance channel of the visible light image after compensation. The color fused image is then output through inverse color space transformation.

[0026] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: This invention achieves high-precision perception of global and local illumination distribution by constructing an adaptive illumination perception network incorporating a coordinate attention mechanism, overcoming the low accuracy of fixed thresholds or simple CNN classifiers in complex lighting environments. By designing a hierarchical differentiated fusion model that prioritizes visible light during the day and infrared light at night, and employing an "OR" cell model for adaptive fusion in the decision-making stage, it effectively resolves the contradiction that a globally unified fusion rule cannot adequately consider both target and background, achieving an optimal balance between infrared target enhancement and visible light background detail. Furthermore, through… lαβ Performing luminance statistical transfer and high-frequency detail compensation in the color space achieves decoupling of luminance and chrominance, avoiding color distortion and texture blurring caused by direct RGB mapping, and significantly improving the color naturalness and detail clarity of the fused image under complex lighting conditions. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0028] Figure 1This is a flowchart illustrating the image fusion method based on rattlesnake vision mechanism and illumination adaptation provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the adaptive illumination sensing network provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the network structure of the daytime fusion model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the network structure of the nighttime fusion model provided in an embodiment of the present invention; Figure 5 This is a comparison chart of the first group of source images in the RoadScene dataset provided in this embodiment of the invention and the fusion results of each method; Figure 6 This is a comparison chart of the second set of source images in the RoadScene dataset provided in this embodiment of the invention and the fusion results of each method. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] In the description of this invention, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. An ON-central receptive field refers to a concentric circle structure antagonistic receptive field model where retinal nerve cells are centrally excited and surrounded by inhibitory signals. Alternatively, a cell model refers to a fusion model simulating a rattlesnake's dual-mode cells that generates a response in the presence of either visible or infrared light signals.

[0031] In one embodiment, a method for image fusion based on rattlesnake visual mechanisms and adaptive lighting is provided. This method includes an adaptive lighting perception stage, a day / night level differential fusion stage, and a detail-preserving color reconstruction stage. Figure 1 The overall flow of the method is shown.

[0032] First, step S1 is executed to acquire visible light and infrared images of the same scene. Visible light and infrared images of the same scene are acquired using a visible light camera and an infrared camera, respectively. After registration processing, the pixel positions of the two images are ensured to correspond one-to-one, serving as input for subsequent fusion processing.

[0033] Next, step S2, the adaptive illumination perception stage, is executed. The visible light image is input into the Adaptive Illumination Perception Network (AIP-Net), which extracts and classifies the illumination distribution features of the visible light image and outputs an illumination environment determination result. The illumination environment determination result is either a daytime scene or a nighttime scene. The specific structure of AIP-Net will be described in detail below.

[0034] Based on the lighting environment determination results, the corresponding fusion model is selected between the daytime fusion model and the nighttime fusion model, and the visible light image and infrared image are input into the selected fusion model for fusion.

[0035] If the scene is determined to be a daytime scene, the daytime fusion model is invoked. This model uses a visible light image as the primary signal and an infrared image as an auxiliary enhancement signal. Specifically, the visible light image is enhanced by an ON-central receptive field to obtain an enhanced visible light signal; the infrared image is processed by dual-mode cell operations and fed into an ON-central receptive field to obtain an infrared feature signal; the enhanced visible light signal and the infrared feature signal are input together into an "OR" cell model to output a daytime grayscale fusion image.

[0036] If the scene is determined to be nighttime, a nighttime fusion model is invoked. This model uses an infrared image as the primary source and a visible light image as an auxiliary background signal. Specifically, the infrared image is enhanced using an ON-central receptive field to obtain an enhanced infrared signal; the visible light image undergoes dual-mode cellular processing and is then fed into an ON-central receptive field to obtain a visible light background signal; the enhanced infrared signal and the visible light background signal are then input into an "OR" cellular model to output a nighttime grayscale fusion image.

[0037] Finally, step S3, the detail-preserving color reconstruction stage, is performed. The grayscale fused image and the visible light image output from step S2 are converted to color spaces where luminance and chrominance are decoupled. The luminance channel of the grayscale fused image is corrected based on the luminance distribution of the visible light image. The high-frequency components in the corrected luminance channel are compensated and then reconstructed with the chrominance channel of the visible light image. The resulting color fused image is output after inverse color space transformation. The specific method of color reconstruction will be described in detail below.

[0038] In one embodiment, the structure of the Adaptive Illumination Sensing Network (AIP-Net) is as follows: Figure 2 As shown, AIP-Net consists of a basic feature extraction layer, an attention enhancement layer, and a classification decision head connected in sequence.

[0039] The basic feature extraction layer performs convolution, batch normalization, and nonlinear activation on the visible light image, outputting a preliminary feature map. Specifically, the input visible light image is an RGB three-channel image with dimensions of 256×256×3. The basic feature extraction layer uses a 3×3 convolution kernel with a stride of 2, combined with batch normalization and the LeakyReLU activation function, to project the input image into a high-dimensional feature space. The output preliminary feature map has 16 channels, and its spatial size is reduced to 128×128 due to downsampling with a stride of 2. The function of this layer is to perform preliminary denoising and shallow feature extraction, while reducing the computational complexity of subsequent operations through downsampling.

[0040] The attention enhancement layer comprises at least one level of coordinate attention module (CoordAtt). In this embodiment, the attention enhancement layer adopts a three-level cascaded design, with the number of feature channels increasing progressively (e.g., 16→32→64). Each level of coordinate attention module extracts the global features in the horizontal and vertical directions of the input preliminary feature map, generating horizontal and vertical attention weights respectively. The preliminary feature map is then reweighted using these horizontal and vertical attention weights, outputting a weighted feature map. The input to the next level of coordinate attention module is the weighted feature map output by the previous level.

[0041] Each level of the coordinate attention module performs the following reweighting step on its input feature map: Step A: Perform one-dimensional global average pooling along the height direction on the input feature map to obtain the first-direction aggregated feature. Specifically, for the input feature map X (with dimensions H×W×C, where H is the height, W is the width, and C is the number of channels), perform global average pooling along the height direction to generate a first-direction aggregated feature with dimensions H×1×C. This feature contains global information for each row in the horizontal direction.

[0042] Step B involves performing one-dimensional global average pooling along the width direction on the input feature map to obtain the second-direction aggregated feature. Specifically, global average pooling is performed along the width direction to generate a second-direction aggregated feature with dimensions 1×W×C. This feature contains global information for each column in the vertical direction.

[0043] Step C involves concatenating the first-direction aggregation feature and the second-direction aggregation feature along the spatial dimension. After a first convolutional transformation, batch normalization, and nonlinear activation, the features are separated into first-branch features and second-branch features along the spatial dimension. Specifically, the two features H×1×C and 1×W×C are concatenated along the spatial dimension, then subjected to a 1×1 convolutional transformation, batch normalization, and nonlinear activation, and finally separated into two branch features along the spatial dimension.

[0044] Step D involves transforming the first branch features through a second convolutional transformation and then applying a sigmoid activation to generate horizontal attention weights; and transforming the second branch features through a third convolutional transformation and then applying a sigmoid activation to generate vertical attention weights. The sigmoid activation maps the output values ​​to the (0,1) interval, which serve as the attention weights.

[0045] Step E involves applying the horizontal and vertical attention weights to the corresponding directions of the input feature map, respectively, to complete the reweighting. That is, the horizontal attention weights are applied to each row of the input feature map, and the vertical attention weights are applied to each column of the input feature map, resulting in the output weighted feature map.

[0046] Through the aforementioned coordinate attention mechanism, the network can simultaneously capture long-range dependencies in both the horizontal and vertical directions, generating attention weights with direction awareness and position sensitivity. This effectively suppresses interference from local strong light sources and guides the network to focus on key lighting areas.

[0047] The classification decision head is used to perform global pooling and fully connected mapping on the weighted feature map of the final output of the attention enhancement layer, and output the lighting environment determination result.

[0048] Specifically, the classification decision head uses adaptive global average pooling to compress the feature map into a one-dimensional vector, which is then mapped by a fully connected layer and regularized by Dropout (with a dropout rate of 0.5). Finally, a two-dimensional Logits vector is output through a linear classification layer, which corresponds to the confidence scores of "daytime" and "nighttime" respectively. The category with the higher confidence score is taken as the final decision result.

[0049] In one embodiment, AIP-Net is trained using a cross-entropy loss function combined with a label smoothing strategy. The label smoothing coefficient is 0.1. The Adam optimizer is used with an initial learning rate of 0.0002 and a piecewise linear decay strategy. The training dataset contains 2011 visible light images from the MSRS and M3FD datasets, preprocessed to a size of 256×256, and divided into training and validation sets in a 9:1 ratio, for a total of 100 training rounds.

[0050] In one embodiment, the daytime fusion model is as follows: Figure 3 As shown, the model sequentially includes a multi-channel feature preprocessing stage, a first-level ON-central receptive field antagonism stage, a second-level ON-central receptive field information refinement stage, and an "or" cell decision fusion stage.

[0051] In the multi-channel feature preprocessing stage, four operators from the rattlesnake dual-mode cellular mathematical model are used to process the visible light and infrared images, generating V∪IR, V∩IR, +IR←V, and ... IR←V. Where V∪IR is the output signal of the OR operation, V∩IR is the output signal of the AND operation, and +IR←V is the infrared-enhanced visible light signal. IR←V represents infrared suppression of the visible light signal. Simultaneously, the visible light image is processed using an ON-center receptive field to generate an enhanced visible light signal ON_VIS.

[0052] It should be noted that the six response modes of the rattlesnake dual-mode cell and their mathematical models are existing technologies and will not be explained redundantly here.

[0053] In the first-order ON-central receptive field antagonism stage, V∪IR is fed into the central excitation region of the ON-central receptive field, and V∩IR is fed into the surrounding inhibition region of the ON-central receptive field, generating +OR_AND to highlight complementary details between spectra; simultaneously, +IR←V and Input IR←V to ON-central receptive field, generating infrared enhancement image +IR.

[0054] In the second-level ON-central receptive field information refinement stage, +OR_AND and +IR are input into the ON-central receptive field to generate a core feature map X1 that carries high-contrast thermal source features and dual-source complementary details.

[0055] In the cell decision fusion stage, ON_VIS and X1 are input into the cell model to generate a daytime grayscale fusion image.

[0056] In one embodiment of the present invention, the nighttime fusion model is as follows: Figure 4 As shown, the nighttime fusion model includes a multi-channel feature preprocessing stage, a first-level ON-central receptive field antagonism stage, and a multi-level ON-central receptive field differential fusion stage.

[0057] In the multi-channel feature preprocessing stage, four operators from the rattlesnake dual-mode cellular mathematical model are used to process the visible light and infrared images, generating V∪IR, V∩IR, +IR←V, and ... IR←V; simultaneously, the visible light image is processed through an ON-central receptive field to generate an enhanced visible light signal ON_VIS, and the infrared image is processed through an ON-central receptive field to generate an enhanced infrared signal +IR. The six response modes of rattlesnake dual-mode cells and their mathematical models are existing technologies and will not be explained redundantly here.

[0058] In the first-order ON-central receptive field antagonism stage, V∪IR is fed into the central excitation region of the ON-central receptive field, and V∩IR is fed into the surrounding inhibition region of the ON-central receptive field, generating +OR_AND to highlight complementary details between spectra; simultaneously, +IR←V and Input IR←V to ON-central receptive field, generating infrared enhancement image +IR.

[0059] In the multi-level ON-central receptive field differential fusion stage, firstly, +OR_AND and +IR are fed into the central excitation region and the surrounding inhibition region of the ON-central receptive field, respectively, to generate the first comprehensive feature map X1. X1 fully preserves the saliency of the target and weak background details. Secondly, ON_VIS and +IR are fed into the ON-central receptive field to generate the second comprehensive feature map X2. X2 highlights the cool background details and suppresses overexposed areas. Thirdly, X1 and X2 are fed into the ON-central receptive field again to generate the third comprehensive feature map X3. By removing redundant noise, the heat source outline and key environmental edges are sharpened from the night background.

[0060] Finally, input X1 and X3 into the cell model to obtain a nighttime grayscale fusion image.

[0061] In one embodiment, to avoid color distortion and loss of detail caused by traditional pseudo-color mapping, the present invention proposes the following color reconstruction method: Specifically, the RGB format visible light image and the grayscale fused image are first converted to... lαβ A color space is used to decouple the luminance component (L) from the chrominance components (α,β). Since the infrared characteristics of the grayscale fused image are mainly manifested as luminance differences, this step focuses on performing statistical distribution alignment on the luminance channel.

[0062] The mean values ​​of the grayscale fused image and the original visible light image in the L channel were calculated. with standard deviation The brightness distribution of the fused image is initially remapped using the Reinhard statistical transfer formula:

[0063] In the formula, The brightness component of the grayscale fused image. These are the initial values ​​after mapping; and These represent the statistical moments of the grayscale fused image and the original visible light image in the luminance channel, respectively. This operation corrects the dynamic range of the fused image to the level of natural visible light. However, Reinhard transfer is a global linear transformation, which inevitably compresses local contrast while adjusting the overall luminance distribution, resulting in a smoothing reduction of texture details.

[0064] In one embodiment, to restore the weakened texture details mentioned above, a differential filtering method is further used to extract and compensate for the detail components lost during the migration process. Specifically: Using standard deviation σGaussian low-pass filter G(.) For the migrated luminance components Smoothing is performed to obtain its low-frequency approximate components; differential operations are used to extract the weakened pure high-frequency texture residuals during the migration process. :

[0065] Subsequently, an enhancement coefficient was introduced. The extracted high-frequency texture residuals Compensation In the process, the final luminance components after structural fidelity correction are obtained:

[0066] Color reconstruction: Constructing a system that includes enhanced luminance components. Compared with the original chromaticity components of the visible light image The new feature vector, and through The inverse transform formula completes color reconstruction:

[0067] In the formula, This represents the inverse color space transformation function. This is the final color-blended image.

[0068] Based on the same inventive concept, this invention also discloses an image fusion system based on rattlesnake visual mechanisms and illumination adaptation, used to perform any of the above-mentioned image fusion methods, including: The image acquisition module is used to acquire visible light and infrared images of the same scene; The adaptive illumination sensing module is used to extract and classify the illumination distribution features of visible light images and output the illumination environment determination results for daytime or nighttime scenes. The fusion model selection module is used to select the corresponding fusion model between the daytime fusion model and the nighttime fusion model based on the lighting environment determination results. The daytime fusion module is used to enhance the visible light image through the ON-central receptive field to obtain the enhanced visible light signal when the daytime fusion model is selected. The infrared image is processed by the dual-mode cell operator and then fed into the ON-central receptive field to obtain the infrared feature signal. The enhanced visible light signal and the infrared feature signal are input together into the cell model to output the daytime grayscale fusion image. The night fusion module is used to enhance the infrared image by passing it through an ON-central receptive field to obtain an enhanced infrared signal when the night fusion model is selected. The visible light image is processed by a dual-mode cell operator and then fed into the ON-central receptive field to obtain a visible light background signal. The enhanced infrared signal and the visible light background signal are input together into the cell model to output a night grayscale fusion image. The color reconstruction module is used to convert the grayscale fused image and the visible light image to a color space where luminance and chrominance are decoupled, respectively. It corrects the luminance channel of the grayscale fused image based on the luminance distribution of the visible light image, and reconstructs the high-frequency components in the corrected luminance channel with the chrominance channel of the visible light image after compensation. The resulting color fused image is then output through inverse color space transformation.

[0069] To verify the effectiveness of the technical solution of this invention, comparative experiments were conducted on three public datasets: MSRS, M3FD, and RoadScene.

[0070] The comparison methods include six existing methods: BA-RBBC, RRFM, DNRVF, CMF Tusion, DCSFuse, and MaeFuse, as well as the method of this invention. The evaluation metrics include six objective image quality evaluation metrics: Information Entropy (EN), Mutual Information (MI), Spatial Frequency (SF), Average Gradient (AG), Edge Intensity (EI), and Variance (VAR).

[0071] Light perception accuracy verification To verify the illumination classification performance of the adaptive illumination perception network AIP-Net, the method of this invention was compared with the PIA-Net baseline model and PIA-Net+Attn (which introduces an attention mechanism on top of PIA-Net). As shown in Table 1, PIA-Net achieved accuracies of 83.87% and 82.76% in daytime and nighttime scenes, respectively, with an overall accuracy of 83.33%. The overall accuracy of PIA-Net+Attn was improved to 88.33%. In contrast, the overall accuracy of the proposed AIP-Net reached 96.67%, with 96.77% accuracy in daytime scenes and 96.55% accuracy in nighttime scenes. This invention significantly improves the accuracy of day and night scene classification under complex illumination environments by introducing a coordinate attention mechanism.

[0072] Table 1

[0073] Objective evaluation of fused image quality As shown in Tables 2 to 4, the method of this invention achieved excellent performance on all six evaluation metrics on the MSRS, M3FD, and RoadScene datasets, especially significantly outperforming the comparison methods on core metrics such as EN, AG, and EI.

[0074] On the MSRS dataset (Table 2), the method of this invention achieves an EN of 7.3593, an AG of 5.8240, an EI of 63.7315, and a VAR of 55.0495, all of which are significantly better than all the comparison methods.

[0075] Table 2

[0076] On the M3FD dataset (Table 3), the proposed method achieves an EN of 7.2574 and an MI of 1.8118, both outperforming all comparable methods. In terms of AG and EI, the proposed method achieves 5.4854 and 60.0568 respectively, demonstrating excellent performance across all metrics.

[0077] Table 3

[0078] On the RoadScene dataset (Table 4), the method of this invention achieves an EN of 7.3522, an AG of 5.6959, and an EI of 62.3919, all of which are superior to all the comparative methods.

[0079] Table 4

[0080] Meanwhile, to verify the effectiveness of each core module of the invention, four ablation scenarios were set up for comparative experiments: Case 1: Without using a light-aware network, regardless of the input image, a daytime model is used for fusion and color and detail reconstruction. Case 2: Instead of using a light-sensing network, a nighttime model is used for fusion, and color and detail reconstruction are performed. Case 3: A dual-branch fusion strategy guided by AIP-Net is adopted, but only grayscale fusion results are output without subsequent color and detail reconstruction. Case 4: Baseline control group. No illumination-sensing network was used; a daytime model was employed, and no subsequent color and detail reconstruction was performed. Table 5

[0081] As shown in Table 5, the complete solution of this invention outperforms all ablation scenarios in all evaluation indicators. The ablation experiment results demonstrate that the collaborative work of the adaptive illumination sensing module, the day / night hierarchical fusion module, and the color reconstruction module makes an irreplaceable contribution to achieving the technical effects of this invention.

[0082] To further verify the visual fusion effect of the method of the present invention in real complex scenes, three representative images were selected from the RoadScene dataset for subjective visual comparison. Figure 5 , Figure 6 The fusion results of each method are shown in two sets of scenes. The results of each set are, in order, visible light images, infrared images, and fused images of BA-RBBC, RRFM, DNRVF, CMF Tusion, DCSFuse, MaeFuse, and the method of this invention.

[0083] like Figure 5 The multi-scene experimental results on the RoadScene dataset show that BA-RBBC, RRFM, and DNRVF all exhibit significant overexposure, especially in light-emitting and high-brightness areas. Detailed textures are difficult to discern due to brightness saturation, accompanied by severe spectral distortion and color cast. While CMTFusion is relatively stable in brightness, its fused image is a grayscale image, resulting in the loss of some detail information and insufficient visual semantic expression. Furthermore, MaeFuse and DCSFuse show relatively blurred boundaries between the target and background, with weak edge detail rendering capabilities. Figure 6 As can be seen, DCSFuse is highly susceptible to interference from car headlights and reflected light from the ground, resulting in large areas of white light in localized areas, severely affecting visual observation. In contrast, the method proposed in this invention can effectively suppress overexposure, preserving prominent infrared targets while achieving the most natural color reproduction and clear detail, resulting in the most balanced visual effect.

[0084] In summary, the method of the present invention can effectively suppress overexposure and maintain color naturalness under different lighting conditions and scene complexity, while clearly presenting the outline of thermal targets and environmental texture details. The subjective visual effect is superior to existing comparative methods.

[0085] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0086] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for image fusion based on rattlesnake visual mechanisms and illumination adaptation, characterized in that, Includes the following steps: S1. Acquire visible light and infrared images of the same scene; S2. Input the visible light image into an adaptive illumination sensing network, extract and classify the illumination distribution features of the visible light image, and output the illumination environment determination result, which is either a daytime scene or a nighttime scene. Based on the lighting environment determination results, a corresponding fusion model is selected between the daytime fusion model and the nighttime fusion model. The visible light image and the infrared image are then input into the selected fusion model for fusion. If the daytime fusion model is selected, the visible light image is enhanced by an ON-central receptive field to obtain an enhanced visible light signal. The infrared image is processed using a dual-mode cellular operator and then fed into the ON-central receptive field to obtain an infrared feature signal. The enhanced visible light signal and the infrared feature signal are then input into the OR cell model to output a daytime grayscale fusion image. The daytime fusion model includes a multi-channel feature preprocessing stage, a first-level ON-central receptive field antagonism stage, a second-level ON-central receptive field information refinement stage, and / or a cell decision fusion stage. The multi-channel feature preprocessing stage includes: processing the visible light image and the infrared image using infrared-enhanced visible light operator, infrared-suppressed visible light operator, visible light-enhanced infrared operator, and visible light-suppressed infrared operator to generate V∪IR, V∩IR, +IR←V, and ... IR←V; The visible light image is processed using an ON-center receptive field to generate an enhanced visible light signal ON_VIS; The first-level ON-central receptive field antagonism phase includes: feeding V∪IR into the central excitation region of the ON-central receptive field, feeding V∩IR into the surrounding inhibition region of the ON-central receptive field, generating +OR_AND; and then connecting +IR←V with... IR←V input ON-central receptive field, generating infrared enhancement map +IR; The second-level ON-central receptive field information refinement stage includes: inputting +OR_AND and +IR into the ON-central receptive field to generate the core feature map X1; The or cell decision fusion stage includes: inputting ON_VIS and X1 into the or cell model to generate the daytime grayscale fusion image; If the nighttime fusion model is selected, the infrared image is enhanced by an ON-central receptive field to obtain an enhanced infrared signal. The visible light image is processed using a dual-mode cellular operator and then fed into the ON-central receptive field to obtain a visible light background signal. The enhanced infrared signal and the visible light background signal are then input into the cellular model to output a nighttime grayscale fusion image. The nighttime fusion model includes a multi-channel feature preprocessing stage, a first-level ON-central receptive field antagonism stage, and a multi-level ON-central receptive field differential fusion stage. The multi-channel feature preprocessing stage includes: processing the visible light image and the infrared image using infrared-enhanced visible light operator, infrared-suppressed visible light operator, visible light-enhanced infrared operator, and visible light-suppressed infrared operator to generate V∪IR, V∩IR, +IR←V, and ... IR←V; The visible light image is processed using an ON-center receptive field to generate an enhanced visible light signal ON_VIS; The first-level ON-central receptive field antagonism phase includes: feeding V∪IR into the central excitation region of the ON-central receptive field, feeding V∩IR into the surrounding inhibition region of the ON-central receptive field, generating +OR_AND; and then connecting +IR←V with... IR←V input ON-central receptive field, generating infrared enhancement map +IR; The multi-level ON-central receptive field differential fusion stage includes: First level: +OR_AND and +IR are fed into the central excitation region and the surrounding inhibition region of the ON-central receptive field, respectively, to generate the first comprehensive feature map X1; Second level: Feed ON_VIS and +IR into the ON-central receptive field to generate the second comprehensive feature map X2; Third level: Feed X1 and X2 into the ON-central receptive field to generate the third comprehensive feature map X3; Input X1 and X3 into the cell model to generate the nighttime grayscale fusion image; S3. Convert the grayscale fusion image output in step S2 and the visible light image to a color space where luminance and chrominance are decoupled, respectively. Correct the luminance channel of the grayscale fusion image based on the luminance distribution of the visible light image, and reconstitute the high-frequency components in the corrected luminance channel with the chrominance channel of the visible light image. Output a color fusion image through inverse color space transformation.

2. The method according to claim 1, characterized in that, The dual-mode cellular operator includes an infrared-enhanced visible light operator, an infrared-suppressed visible light operator, a visible light-enhanced infrared operator, and a visible light-suppressed infrared operator. The infrared-enhanced visible light operator is used to simulate the enhancement effect of infrared signals on visible light signals. The infrared-suppressed visible light operator is used to simulate the suppression effect of infrared signals on visible light signals. The visible light-enhanced infrared operator is used to simulate the enhancement effect of visible light signals on infrared signals. The visible light-suppressed infrared operator is used to simulate the suppression effect of visible light signals on infrared signals.

3. The method according to claim 1, characterized in that, The adaptive illumination sensing network comprises a basic feature extraction layer, an attention enhancement layer, and a classification decision head connected in sequence. The basic feature extraction layer is used to perform convolution, batch normalization and nonlinear activation processing on the visible light image to output a preliminary feature map; The attention enhancement layer includes at least one level of coordinate attention module. Each level of coordinate attention module extracts global features in the horizontal direction and global features in the vertical direction from the input preliminary feature map, and generates horizontal attention weights and vertical attention weights respectively. The preliminary feature map is reweighted using the horizontal attention weights and the vertical attention weights, and a weighted feature map is output. The attention enhancement layer includes multiple levels of cascaded coordinate attention modules, and the input of the subsequent level of coordinate attention module is the weighted feature map output by the previous level of coordinate attention module. The classification decision head is used to perform global pooling and fully connected mapping on the weighted feature map finally output by the attention enhancement layer, and output the lighting environment determination result.

4. The method according to claim 3, characterized in that, Each level of the coordinate attention module performs the following reweighting step on its input feature map: Step A: Perform one-dimensional global average pooling on the input feature map along the height direction to obtain the first direction aggregated feature. The dimension of the first direction aggregated feature is H×1×C, where H is the height dimension and C is the number of channels. Step B: Perform one-dimensional global average pooling on the input feature map along the width direction to obtain the second-direction aggregated feature. The dimension of the second-direction aggregated feature is 1×W×C, where W is the width dimension. Step C: The first direction aggregation feature and the second direction aggregation feature are concatenated in the spatial dimension. After the first convolution transformation, batch normalization and nonlinear activation, they are separated into the first branch feature and the second branch feature along the spatial dimension. Step D: The first branch features are transformed by a second convolution and activated by a sigmoid function to generate the horizontal attention weights, and the second branch features are transformed by a third convolution and activated by a sigmoid function to generate the vertical attention weights. Step E: Apply the horizontal attention weights and the vertical attention weights to the corresponding directions of the input feature map to complete the reweighting of the input feature map.

5. The method according to claim 1, characterized in that, In step S3, the color space where brightness and chromaticity are decoupled is: lαβ Color space; Correcting the brightness channel of the grayscale fused image based on the brightness distribution of the visible light image includes: Calculate the mean values ​​of the grayscale fused image and the visible light image in the luminance channel, respectively. , and standard deviation , Calculate the luminance components after migration : in, For the grayscale fused image in lαβ The luminance channel component in a color space.

6. The method according to claim 5, characterized in that, The high-frequency components in the corrected luminance channel are compensated and then reconstructed with the chrominance channel of the visible light image. A color-fused image is then output via inverse color space transformation, including: A Gaussian low-pass filter with a standard deviation of σ is used. G (.) for the migrated luminance components Filtering is performed, and high-frequency texture residuals are extracted through differential operations. : Introduce a preset enhancement coefficient The final brightness channel after compensation : The final brightness channel With the visible light image in lαβ Chroma channel components in a color space Reorganization, after lαβ Inverse color space transformation outputs a color-blended image. : in, This represents the inverse color space transformation function.

7. An image fusion system based on rattlesnake visual mechanisms and illumination adaptation, characterized in that, For performing the method as described in any one of claims 1 to 6, comprising: The image acquisition module is used to acquire visible light and infrared images of the same scene; An adaptive illumination sensing module is used to extract and classify the illumination distribution features of the visible light image and output the illumination environment determination result for daytime or nighttime scenes. The fusion model selection module is used to select the corresponding fusion model between the daytime fusion model and the nighttime fusion model based on the lighting environment determination result. The daytime fusion module is used to, when the daytime fusion model is selected, enhance the visible light image through an ON-central receptive field to obtain an enhanced visible light signal, process the infrared image using a dual-mode cell operator and feed it into the ON-central receptive field to obtain an infrared feature signal, input the enhanced visible light signal and the infrared feature signal into the cell model together, and output a daytime grayscale fusion image. The night fusion module is used to, when the night fusion model is selected, enhance the infrared image by passing it through an ON-central receptive field to obtain an enhanced infrared signal, process the visible light image using a dual-mode cell operator and feed it into the ON-central receptive field to obtain a visible light background signal, input the enhanced infrared signal and the visible light background signal into the cell model together, and output a night grayscale fusion image. The color reconstruction module is used to convert the grayscale fused image and the visible light image to color spaces where luminance and chrominance are decoupled, respectively. It corrects the luminance channel of the grayscale fused image based on the luminance distribution of the visible light image, and reconstructs the high-frequency components in the corrected luminance channel with the chrominance channel of the visible light image after compensation. The color fused image is then output through inverse color space transformation.

Images