Visible light and infrared image fusion method, apparatus, device and storage medium

Abstract

This invention discloses a method, apparatus, device, and storage medium for fusion of visible light and infrared images. The method, by introducing a hybrid expert system with a light-sensing gating network, establishes a relationship between lighting conditions and network learning, enabling better adaptation to different lighting conditions and achieving complementary and enhanced image information under various lighting scenarios. This allows it to adapt to complex and changing lighting environments, thereby achieving efficient and accurate fusion of visible light and infrared images. It can dynamically complement and enhance image information under different lighting scenarios, improving the efficiency and accuracy of visible light and infrared image fusion.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for efficient and accurate visible light and infrared image fusion based on a light-sensing hybrid expert system. Background Technology

[0002] With the continuous development of image processing technology, visible light and infrared image fusion technology is increasingly widely used in fields such as military reconnaissance, security monitoring, medical diagnosis, and autonomous driving. Traditional fusion algorithms, such as weighted average-based fusion algorithms, are simple and efficient in the initial fusion stage, while multi-scale decomposition-based fusion algorithms enrich the fusion results by decomposing and reconstructing the image at multiple levels. Nevertheless, in the face of complex and ever-changing scenes, traditional algorithms often struggle to cope with the challenges brought by changes in illumination, leading to the loss or distortion of image information.

[0003] In recent years, the rise of deep learning methods, especially those based on Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), and Transformers, has brought new ideas to image fusion. Existing methods typically employ a "preprocessing before fusion" approach to mitigate the effects of lighting. However, this approach not only increases model complexity but may also lead to a decrease in the model's generalization ability.

[0004] To address this issue, some researchers have begun exploring simple and direct fusion frameworks, attempting to introduce illumination-aware mechanisms to improve image fusion performance. However, existing methods still have some limitations. For example, Tang et al., in their paper "PIAFusion: A progressive infrared and visible image fusion network based on illumination aware," designed an illumination-aware network to predict illumination distribution and guide the training process of the fusion network, which improved the fusion effect to some extent. However, illumination information was not considered during the model's inference phase.

[0005] In addition, Cao et al. used a hybrid expert system to dynamically fuse multimodal images in their paper “Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusion”. Although this method improved the flexibility of fusion to some extent, it failed to establish an explicit correlation between illumination information and network learning. Summary of the Invention

[0006] In view of the above problems, the present invention provides a visible light and infrared image fusion method, apparatus, device, and storage medium for overcoming or at least partially solving the above problems. This method achieves complementarity and enhancement of image information under different lighting scenarios by introducing a hybrid expert system with a light-sensing gating network.

[0007] This invention provides the following solution:

[0008] A method for fusing visible light and infrared images, comprising:

[0009] Acquire a visible light image and an infrared image to be fused;

[0010] The visible light image is input into the illumination sensing gating network to obtain the illumination factor; the illumination sensing gating network includes a multi-layer convolutional neural network composed of residual blocks, and under the constraint of the binary cross-entropy loss function, it learns multimodal image features under different illumination conditions and predicts the illumination distribution in order to obtain the illumination factor.

[0011] The visible light image is converted from the RGB color space to the YUV color space, and the corresponding luminance component, chrominance component and saturation component are separated.

[0012] The infrared image and the brightness component are input together into a fusion network based on a light-sensing hybrid expert system, and weighted by the light factor to obtain a preliminary fusion result; the fusion network based on a light-sensing hybrid expert system includes a shallow feature extraction network, a deep feature extraction network, and high and low light expert systems;

[0013] The chromaticity component, the concentration component, and the preliminary fusion result are combined to obtain a fusion result. The fusion result is then converted from the YUV color space back to the RGB color space to obtain the fusion result of the visible light and infrared image.

[0014] Preferably, the multilayer convolutional neural network composed of residual blocks includes the ResNet18 network structure.

[0015] Preferably, the shallow feature extraction network is used to perform preliminary feature extraction on the visible light image and the infrared image respectively through a 3×3 convolutional kernel and an activation function with leakage correction linear units.

[0016] Preferably, the deep feature extraction network includes a Swing-Transformer layer and a residual dense convolutional layer. The Swing-Transformer layer is used to extract deep features from the visible light image, and the residual dense convolutional layer is used to extract deep features from the infrared image.

[0017] Preferably, the high and low illumination expert systems include a chiral feature fusion module; in the high illumination expert system, the chiral feature fusion module uses the visible light image as the primary modal input and the infrared image as the auxiliary modal input; in the low illumination expert system, the chiral feature fusion module uses the visible light image as the auxiliary modal input and the infrared image as the primary modal input; so that the inputs of the high illumination expert system and the low illumination expert system are chirally symmetrical.

[0018] Preferably, the high and low illumination expert system includes an image reconstruction module, which includes three convolutional layers: two convolutional layers are composed of 3×3 convolutional kernels and Leaky ReLU activation functions, and one convolutional layer is composed of 3×3 convolutional kernels and Tanh activation functions.

[0019] Preferably, it also includes the target loss function for training the fusion network based on the illumination-aware hybrid expert system. The target loss function is used to explicitly divide scenes with different illumination intensities, and by calculating the loss between the fusion image generated by each expert sub-network and the original image, it is weighted in combination with the illumination factor so that each expert sub-network can independently generate a complete fusion image in the corresponding scene.

[0020] A visible light and infrared image fusion apparatus for performing the above-described visible light and infrared image fusion method, the apparatus comprising:

[0021] The image acquisition unit is used to acquire a visible light image and an infrared image to be fused.

[0022] The illumination factor acquisition unit is used to input the visible light image into the illumination sensing gating network to obtain the illumination factor; the illumination sensing gating network includes a multi-layer convolutional neural network composed of residual blocks, and under the constraint of the binary cross-entropy loss function, it learns multimodal image features under different illumination conditions and predicts the illumination distribution in order to obtain the illumination factor.

[0023] The component separation unit is used to convert the visible light image from the RGB color space to the YUV color space and separate the corresponding luminance component, chrominance component and saturation component.

[0024] The preliminary fusion result acquisition unit is used to input the infrared image and the brightness component together into the fusion network based on the illumination perception hybrid expert system, and perform weighted processing in combination with the illumination factor to obtain the preliminary fusion result; the fusion network based on the illumination perception hybrid expert system includes a shallow feature extraction network, a deep feature extraction network, and high and low illumination expert systems.

[0025] The fusion result acquisition unit is used to combine the chromaticity component, the concentration component and the preliminary fusion result to obtain a fusion result, and convert the fusion result from the YUV color space back to the RGB color space to obtain the fusion result of the visible light and infrared image.

[0026] A visible light and infrared image fusion device, the device comprising a processor and a memory:

[0027] The memory is used to store program code and transmit the program code to the processor;

[0028] The processor is used to execute the above-described visible light and infrared image fusion method according to the instructions in the program code.

[0029] A computer-readable storage medium for storing program code for executing the above-described visible light and infrared image fusion method.

[0030] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0031] This application provides a visible light and infrared image fusion method, apparatus, device, and storage medium. By introducing a hybrid expert system with a light-sensing gating network, it establishes a relationship between lighting conditions and network learning, enabling better adaptation to different lighting conditions and achieving complementarity and enhancement of image information under different lighting scenarios. This allows it to adapt to complex and ever-changing lighting environments, thereby achieving efficient and accurate visible light and infrared image fusion.

[0032] In addition, in the preferred implementation, a loss function is provided to better train this complex network. This loss function can explicitly divide scenes with different lighting intensities and calculate the loss between the fused image generated by each expert sub-network and the original image, weighted by the lighting factor, thereby ensuring that each expert sub-network can independently generate a complete fused image in its corresponding scene.

[0033] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0035] Figure 1This is a flowchart of the visible light and infrared image fusion method provided in the embodiments of the present invention;

[0036] Figure 2 This is a network diagram of the visible light and infrared image fusion method provided in the embodiments of the present invention;

[0037] Figure 3 This is a network diagram of the chiral feature fusion module provided in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the visible light and infrared image fusion device provided in an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram of the visible light and infrared image fusion device provided in an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0041] See Figure 1 This invention provides a visible light and infrared image fusion method, such as... Figure 1 As shown, the method may include:

[0042] S101: Acquire a visible light image and an infrared image to be fused; wherein the visible light image is a three-channel RGB visible light image and the infrared image is a single-channel infrared image.

[0043] S102: Input the visible light image into the illumination-aware gating network to obtain the illumination factor; the illumination-aware gating network includes a multi-layer convolutional neural network composed of residual blocks, and under the constraint of the binary cross-entropy loss function, learns multimodal image features under different illumination conditions and predicts the illumination distribution in order to obtain the illumination factor; in specific implementation, the multi-layer convolutional neural network composed of residual blocks includes a ResNet18 network structure.

[0044] S103: Convert the visible light image from the RGB color space to the YUV color space, and separate the corresponding luminance component, chrominance component and saturation component;

[0045] S104: The infrared image and the brightness component are input together into the fusion network based on the illumination perception hybrid expert system, and weighted processing is performed in conjunction with the illumination factor to obtain a preliminary fusion result; the fusion network based on the illumination perception hybrid expert system includes a shallow feature extraction network, a deep feature extraction network, and high and low illumination expert systems; in specific implementation, the embodiments of this application can provide the shallow feature extraction network to perform preliminary feature extraction on the visible light image and the infrared image respectively through a 3×3 convolution kernel and an activation function with leakage correction linear units.

[0046] The deep feature extraction network includes a Swin-Transformer layer and a residual dense convolutional layer. The Swin-Transformer layer is used to extract deep features from the visible light image, and the residual dense convolutional layer is used to extract deep features from the infrared image.

[0047] The high and low illumination expert systems include a chiral feature fusion module. In the high illumination expert system, the chiral feature fusion module uses the visible light image as the primary modal input and the infrared image as the auxiliary modal input. In the low illumination expert system, the chiral feature fusion module uses the visible light image as the auxiliary modal input and the infrared image as the primary modal input, so that the inputs of the high illumination expert system and the low illumination expert system are chirally symmetrical.

[0048] The high and low illumination expert system includes an image reconstruction module, which includes three convolutional layers. Two of the convolutional layers consist of 3×3 convolutional kernels and Leaky ReLU activation functions, and one convolutional layer consists of 3×3 convolutional kernels and Tanh activation functions.

[0049] S105: Combine the chromaticity component, the concentration component, and the preliminary fusion result to obtain a fusion result, and convert the fusion result from the YUV color space back to the RGB color space to obtain the fusion result of the visible light and infrared image.

[0050] To better train the network provided in this application embodiment, this application embodiment may also provide a target loss function for training the fusion network based on the illumination-aware hybrid expert system. The target loss function is used to explicitly divide scenes with different illumination intensities, and calculates the loss between the fusion image generated by each expert sub-network and the original image, and weights it in combination with the illumination factor, so that each expert sub-network can independently generate a complete fusion image in the corresponding scene.

[0051] The visible light and infrared image fusion method provided in this application introduces a hybrid expert system with a light-sensing gating network to achieve complementarity and enhancement of image information under different lighting scenarios.

[0052] The visible light and infrared image fusion method provided in this application will be described in detail below.

[0053] The methods provided in this application mainly include:

[0054] (1) Construct a light-sensing gating network based on the ResNet18 network structure and the binary cross-entropy loss function, train it on high and low light visible light image datasets, and then predict the light distribution to obtain the light factor.

[0055] (2) Establish shallow and deep feature extraction networks to progressively represent complementary information of images of different modalities.

[0056] (3) Heterogeneous high-light and low-light expert systems based on hand signal feature fusion modules.

[0057] (4) Design a loss function for the hybrid expert system fusion network based on competitive learning to guide the hybrid expert sub-network to independently generate the fused image under the corresponding lighting scene.

[0058] In practice:

[0059] S1: Obtain a three-channel RGB visible light image to be fused (using I... vi (represented) and single-channel infrared images (using I) ir express).

[0060] S2: The visible light image I acquired in S1 vi The input is a light-sensing gating network, which predicts the light distribution of the input image and calculates the light factor p. i .

[0061] The illumination-aware gating network described is designed based on the ResNet18 architecture and trained on a dataset of visible light images under both high and low illumination conditions. A binary cross-entropy loss function is used to facilitate the model's learning of image features under different illumination conditions, predict illumination distribution, and obtain illumination factors. The ResNet18 network structure consists of multiple convolutional layers, residual blocks, pooling layers, and fully connected layers, containing 18 layers of weights. This structure enables ResNet18 to effectively learn image features and performs well in various computer vision tasks.

[0062] S3: The visible light image I acquired in S1 vi Switch to the YUV color space to separate the corresponding luminance components. chromaticity components and concentration components

[0063] S4: The luminance component obtained in S3 The single-channel infrared image I obtained with S1 ir The image is input into an image fusion network based on a light-sensing hybrid expert system, and a preliminary fusion result I is obtained. f .

[0064] The image fusion network based on the illumination-aware hybrid expert system includes a shallow feature extraction network, a deep feature extraction network, and high and low illumination expert systems.

[0065] The shallow feature extraction network performs preliminary feature extraction on visible light and infrared images respectively through 3×3 convolutional kernels and Leaky ReLU activation function.

[0066] The deep feature extraction network includes a Swin-Transformer layer and a residual dense convolutional layer. The Swin-Transformer layer extracts deep features from visible light images, and the residual dense convolutional layer extracts deep features from infrared images.

[0067] The high and low illumination expert systems include a chiral fusion block (CFB) module and an image reconstruction module. In the high illumination expert system, the chiral fusion block uses visible light as the primary modal input and infrared light as the secondary modal input; in the low illumination expert system, it uses visible light as the secondary modal input and infrared light as the primary modal input.

[0068] This flexible processing method can better adapt to different lighting conditions. The image reconstruction module includes three convolutional layers: two convolutional layers consist of 3×3 convolutional kernels and Leaky ReLU activation functions, and one convolutional layer consists of 3×3 convolutional kernels and Tanh activation functions.

[0069] S5: The fusion result obtained in S4 is I f The chromaticity components obtained with S3 Concentration components The images are merged and converted back to the RGB color space to obtain the fusion result of the visible light and infrared image pairs.

[0070] like Figure 2 As shown, the image fusion network of the illumination-aware hybrid expert system can be implemented in software or hardware, specifically including an illumination-aware gating network, a feature extraction network, high and low illumination expert systems, and a loss function.

[0071] After step S2, the model parameters of the light-sensing gating network are frozen and used to output the light factor p. i .

[0072] The feature extraction network includes a shallow feature extraction module and a deep feature extraction module. The shallow feature extraction module uses a 3×3 convolutional layer H. SC (·) Extract visible light images and infrared image I ir The characteristic of this process can be represented as:

[0073]

[0074] in, This represents shallow features extracted from a visible light image. This represents the shallow features extracted from the infrared image.

[0075] In the deep feature extraction module, the Swin-Transformer module H is used. DT (·) and residual dense connection CNN module H DC (·) Extract feature information from visible light and infrared images respectively. This process can be represented as:

[0076]

[0077] in, This represents the deep features extracted from the shallow features of visible light. This represents the deep features extracted from the shallow infrared features.

[0078] The high- and low-light hybrid expert system includes a chiral fusion block (CFB) and an image reconstruction module. The CFB network structure is as follows: Figure 3 As shown, it specifically includes:

[0079] exist Figure 3 In part (a), the high-light expert system uses visible light feature vectors Z, as the principal mode feature vector maj Infrared feature vector group Z, as the auxiliary modal feature vector set aux In low-light expert systems, the primary and secondary modes exhibit opposite characteristics and a chiral symmetry relationship.

[0080] Deep features obtained from feature extraction networks The feature vector Z is obtained by performing linear mapping (LP) and layer normalization (LN) processing. embed ={Z vi Z ir}

[0081] This process can be represented as:

[0082] Z embed =LN(Transpose(Flatten(Conv2d(F input ))))

[0083]

[0084] Where Transpose(·) represents the matrix transpose operation, Flatten(·) represents the operation of unfolding the matrix into a one-dimensional vector, Conv2d(·) represents the two-dimensional convolution operation, and H, W, and C represent the width, height, and number of channels of the input feature, respectively.

[0085] The eigenvector Z embed ={Z vi Z ir}enter Figure 3 The Cross Attention module shown in section (b) performs cross attention operations to enhance feature fusion.

[0086] Principal mode eigenvector Z maior and auxiliary modal eigenvectors Z aux The fully connected layer generates three vectors with different weights: query Q, key K, and value V.

[0087] The concatenation of the K and V vectors of the primary and secondary modes along the channel dimension can be represented as follows:

[0088] K d =Concat(K maj K aux )

[0089] V d =Concat(V maj V aux )

[0090] Through this operation, K d and V d It contains information from both the primary and secondary modalities, while the query vector uses the primary modal information, i.e., Q. d =Q maior .

[0091] Then, the attention weights are calculated, as follows:

[0092]

[0093] The image reconstruction module mainly includes three convolutional layers. The first two layers consist of 3×3 convolutional kernels and Leaky ReLU activation functions as units, and the third layer consists of 1×1 convolutional kernels and Tanh activation functions, denoted as H respectively. CBL (.) and HCBT (·)

[0094]

[0095] in, This represents the output of the i-th souvenir feature fusion module. This represents the fusion result of the i-th expert system subnetwork. This indicates the preliminary fusion results.

[0096] Then and The merged colors are converted to RGB color space, and the result is output as the final fusion result.

[0097] Furthermore, to better train this complex network, embodiments of this application may also provide a loss function. This loss function can explicitly divide scenes into different lighting intensities, and by calculating the loss between the fused image generated by each expert sub-network and the original image, and weighting it with lighting factors, it ensures that each expert sub-network can independently generate a complete fused image in its corresponding scene.

[0098] The loss function mentioned above is the loss function used during training to guide the generation of fused images in a scene based on a light-aware hybrid expert system, and to promote competition among sub-networks. Specifically:

[0099]

[0100] In summary, the visible light and infrared image fusion method provided in this application establishes a relationship between lighting conditions and network learning by introducing a hybrid expert system with a light-sensing gating network. This allows for better adaptation to different lighting conditions, enabling the complementarity and enhancement of image information under various lighting scenarios, and adapting to complex and ever-changing lighting environments. Ultimately, this achieves efficient and accurate visible light and infrared image fusion.

[0101] See Figure 4 This application embodiment can also provide a visible light and infrared image fusion device, such as... Figure 4 As shown, the device may include:

[0102] Image acquisition unit 401 is used to acquire a visible light image and an infrared image to be fused;

[0103] The illumination factor acquisition unit 402 is used to input the visible light image into the illumination sensing gating network to obtain the illumination factor; the illumination sensing gating network includes a multi-layer convolutional neural network composed of residual blocks, and under the constraint of the binary cross-entropy loss function, it learns multimodal image features under different illumination conditions and predicts the illumination distribution in order to obtain the illumination factor.

[0104] The component separation unit 403 is used to convert the visible light image from the RGB color space to the YUV color space and separate the corresponding luminance component, chrominance component and saturation component.

[0105] The preliminary fusion result acquisition unit 404 is used to input the infrared image and the brightness component together into the fusion network based on the illumination perception hybrid expert system, and perform weighted processing in combination with the illumination factor to obtain the preliminary fusion result; the fusion network based on the illumination perception hybrid expert system includes a shallow feature extraction network, a deep feature extraction network, and high and low illumination expert systems.

[0106] The fusion result acquisition unit 405 is used to merge the chromaticity component, the concentration component, and the preliminary fusion result to obtain a fusion result, and to obtain the fusion result...

[0107] This application embodiment can also provide a visible light and infrared image fusion device, the device including a processor and a memory:

[0108] The memory is used to store program code and transmit the program code to the processor;

[0109] The processor is used to execute the steps of the visible light and infrared image fusion method described above according to the instructions in the program code.

[0110] like Figure 5 As shown in the figure, a visible light and infrared image fusion device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0111] In the embodiments of this application, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0112] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiments of the visible light and infrared image fusion method.

[0113] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0114] Acquire a visible light image and an infrared image to be fused;

[0115] The visible light image is input into the illumination sensing gating network to obtain the illumination factor; the illumination sensing gating network includes a multi-layer convolutional neural network composed of residual blocks, and under the constraint of the binary cross-entropy loss function, it learns multimodal image features under different illumination conditions and predicts the illumination distribution in order to obtain the illumination factor.

[0116] The visible light image is converted from the RGB color space to the YUV color space, and the corresponding luminance component, chrominance component and saturation component are separated.

[0117] The infrared image and the brightness component are input together into a fusion network based on a light-sensing hybrid expert system, and weighted by the light factor to obtain a preliminary fusion result; the fusion network based on a light-sensing hybrid expert system includes a shallow feature extraction network, a deep feature extraction network, and high and low light expert systems;

[0118] The chromaticity component, the concentration component, and the preliminary fusion result are combined to obtain a fusion result. The fusion result is then converted from the YUV color space back to the RGB color space to obtain the fusion result of the visible light and infrared image.

[0119] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0120] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0121] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0122] Of course, it should be noted that, Figure 5 The structure shown does not constitute a limitation on the visible light and infrared image fusion device in the embodiments of this application. In practical applications, the visible light and infrared image fusion device may include more than Figure 5 More or fewer components as shown, or combinations of certain components.

[0123] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described visible light and infrared image fusion method.

[0124] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0125] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0126] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0127] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for fusing visible light and infrared images, characterized in that, include: Acquire a visible light image and an infrared image to be fused; The visible light image is input into a light-sensing gating network to obtain the light factor; The illumination-sensing gating network includes a multi-layer convolutional neural network composed of residual blocks, and learns multimodal image features under different illumination conditions under the constraint of a binary cross-entropy loss function to predict the illumination distribution in order to obtain the illumination factor. The visible light image is converted from the RGB color space to the YUV color space, and the corresponding luminance component, chrominance component and saturation component are separated. The infrared image and the brightness component are input together into a fusion network based on a light-sensing hybrid expert system. Weighted processing is performed using the illumination factor to obtain a preliminary fusion result. The fusion network includes a shallow feature extraction network, a deep feature extraction network, and high and low illumination expert systems. Each high and low illumination expert system includes a chiral feature fusion module. In the high illumination expert system, the chiral feature fusion module uses the visible light image as the primary modal input and the infrared image as the auxiliary modal input. In the low illumination expert system, the chiral feature fusion module uses the visible light image as the auxiliary modal input and the infrared image as the primary modal input, ensuring that the inputs of the high illumination expert system and the low illumination expert system are chirally symmetrical. The chromaticity component, the concentration component, and the preliminary fusion result are combined to obtain a fused result. The fused result is then converted from the YUV color space back to the RGB color space to obtain the fusion result of the visible light and infrared images. It also includes the target loss function for training the fusion network based on the illumination-aware hybrid expert system. The target loss function is used to explicitly divide scenes with different illumination intensities, and calculates the loss between the fusion image generated by each expert sub-network and the original image, and weights it in combination with the illumination factor so that each expert sub-network can independently generate a complete fusion image in the corresponding scene.

2. The visible light and infrared image fusion method according to claim 1, characterized in that, The multilayer convolutional neural network composed of residual blocks includes the ResNet18 network structure.

3. The visible light and infrared image fusion method according to claim 1, characterized in that, The shallow feature extraction network is used to perform preliminary feature extraction on the visible light image and the infrared image respectively through a 3×3 convolutional kernel and an activation function with leakage correction linear units.

4. The visible light and infrared image fusion method according to claim 1, characterized in that, The deep feature extraction network includes a Swin-Transformer layer and a residual dense convolutional layer. The Swin-Transformer layer is used to extract deep features from the visible light image, and the residual dense convolutional layer is used to extract deep features from the infrared image.

5. The visible light and infrared image fusion method according to claim 1, characterized in that, The high and low illumination expert system includes an image reconstruction module, which includes three convolutional layers. Two of the convolutional layers consist of 3×3 convolutional kernels and Leaky ReLU activation functions, and one convolutional layer consists of 3×3 convolutional kernels and Tanh activation functions.

6. A visible light and infrared image fusion device, characterized in that, The apparatus for performing the visible light and infrared image fusion method according to any one of claims 1-5, the apparatus comprising: The image acquisition unit is used to acquire a visible light image and an infrared image to be fused. The illumination factor acquisition unit is used to input the visible light image into the illumination sensing gating network to obtain the illumination factor; the illumination sensing gating network includes a multi-layer convolutional neural network composed of residual blocks, and under the constraint of the binary cross-entropy loss function, it learns multimodal image features under different illumination conditions and predicts the illumination distribution in order to obtain the illumination factor. The component separation unit is used to convert the visible light image from the RGB color space to the YUV color space and separate the corresponding luminance component, chrominance component and saturation component. The preliminary fusion result acquisition unit is used to input the infrared image and the brightness component together into the fusion network based on the illumination perception hybrid expert system, and perform weighted processing in combination with the illumination factor to obtain the preliminary fusion result; the fusion network based on the illumination perception hybrid expert system includes a shallow feature extraction network, a deep feature extraction network, and high and low illumination expert systems. The fusion result acquisition unit is used to merge the chromaticity component, the concentration component and the preliminary fusion result to obtain a fusion result, and convert the fusion result from the YUV color space back to the RGB color space to obtain the fusion result of the visible light and infrared image.

7. A visible light and infrared image fusion device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the visible light and infrared image fusion method according to any one of claims 1-5 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the visible light and infrared image fusion method according to any one of claims 1-5.

Images