Image fusion method and system based on cascaded attention mechanism and neural network

Abstract

The application provides an image fusion method and system based on a cascade attention mechanism and a neural network, and the method comprises the following steps: converting a color visible light image located in an RGB space into a YCbCr space; extracting features of the color image and an infrared image respectively and calculating attention maps of the color image and the infrared image; constructing a color fusion result through a fusion module; and converting the YCbCr space into the RGB space, so that the fused image is displayed in the RGB space. The application solves the technical problems of excessive dependence on artificial design, poor applicability and decreased visual effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image fusion, in particular to the field of multi-source image fusion processing technology. BACKGROUND

[0002] Image fusion aims to combine the advantages of two modal images to generate complementary fusion results. In recent years, many methods of infrared and visible light image fusion have been proposed. However, the existing methods are mainly aimed at infrared and gray visible light images, and the gray fusion image cannot maintain the chrominance information, which cannot meet the requirements of different applications. More specifically, chrominance information is an important feature to improve the perception ability of the human visual system. With the development of sensor technology, visible sensors are no longer limited to single-channel or gray space. Many visible sensors can capture images in color space, because color visible sensors can perceive the difference of objects through chrominance, so images in color space can provide more meaningful information and have better visual effects than gray visible light images. Color visible light images contain chrominance information and can provide more background information and detail information than gray visible light images.

[0003] In addition, color visible sensors can help visible light images preserve rich chrominance information by capturing reflected light to produce images. However, when the sensor is in strong light, backlight or weak lighting conditions, the color visible sensor cannot capture target information. While infrared sensors are mainly sensitive to the thermal radiation emitted by objects and can capture target information under any lighting changes or camouflage, they can work during the day or at night.

[0004] According to the fusion strategy, infrared and visible light image fusion can be divided into traditional methods and deep learning methods. Traditional methods include image fusion methods based on multi-scale transformation, image fusion methods based on saliency detection, image fusion methods based on subspace, image fusion methods based on sparse representation, and other methods. Traditional decomposition methods often need to manually design decomposition rules and fusion strategies, which limits their application in many aspects.

[0005] Therefore, deep learning is introduced into the fusion task of infrared and visible light images with its simple fusion strategy and strong feature representation ability. The deep learning-based methods include convolutional neural network-based methods and generative adversarial network (GAN)-based methods. With the convolutional neural network-based method, the features of the fused image are extracted by using a deep network, and then the fused image is reconstructed by using the fused features. However, the attention information cannot be considered in the fusion process. Since the GAN has the ability to extract deep features, it is introduced into the image fusion task. In addition, various GAN variants are used in the image fusion task, such as a dual discriminator GAN fusion model. For example, the existing patent application document with the publication number CN113436128A, “Dual discriminator multi-modal MR image fusion method, system and terminal”, uses a dual discriminator to extract features and classify the fusion image and the MR-T1 image, and the fusion image and the MR-T2 image, to ensure that the fusion image retains the features of the MR-T1 and MR-T2 to the same extent; a U-Net structure is used instead of a general convolutional neural network structure as a generator; based on the cross-layer connection structure of the U-Net, the features of the same level at the bottom are retained during the process of up-sampling the reconstructed image. From the specific embodiments of the prior art, it can be known that in the GAN of the prior scheme, a dual discriminator is set to classify the source images and the fusion images of two modalities, and the purpose of the technical feature is to retain the features in the two images. In addition, some existing schemes also add an attention mechanism to the GAN generator to improve the performance of the fusion model. However, the aforementioned GAN-based fusion method always has the problem of difficult model training. For example, the existing patent application document with the publication number CN113781377A, “Infrared and visible light image fusion method based on adversarial semantic guidance and perception”, includes: 1. generating a fusion network ASGGAN, 2. learning optimization, using a segmentation network as a discriminator, and the segmentation network and the fusion network form a generative adversarial network relationship, and the segmentation network and the fusion network are continuously optimized in the process of adversarial learning, 3. obtaining a global and local GAN network loss function, 4. adding a segmentation label, adding a segmentation label as a discriminator to optimize the spatial selection on the fusion, and 5. comprehensive evaluation. From the specific implementation content in the prior scheme, it can be known that the prior art has similar problems as most existing methods, that is, it mainly focuses on the fusion of infrared and grayscale visible light images, which makes the fused image ignore the chroma information and causes the decline of visual effect.

[0006] In summary, the prior art has the technical problems of excessive dependence on artificial design, poor applicability, and decline in visual effect. SUMMARY

[0007] The technical problem to be solved by the present application is how to solve the technical problems of excessive dependence on manual design, poor applicability and declining visual effect in the prior art.

[0008] The present application solves the above technical problems by adopting the following technical scheme: an image fusion method based on a cascaded attention mechanism and a neural network comprises:

[0009] S1, transmitting a color visible light image in an RGB space to a preset YCbCr space to extract a color visible light luminance of the color visible light image, acquire and convert chrominance information into Cb and Cr channels to complete conversion of the image space;

[0010] S2, acquiring an infrared image, and using a preset cascaded attention mechanism to calculate an infrared effective attention map of the infrared image, wherein the preset cascaded attention mechanism in step S2 comprises:

[0011] S21, using a channel attention feature forced model to reweight the features of the infrared image, calculate initial attention and an initial attention map to focus on key features and filter out irrelevant features, acquire activation values of hidden neurons to measure the importance of the neurons, and perform a maximum value mapping operation on the Cb and Cr channels to acquire the attention map;

[0012] S22, using a preset global attention module to perform a Gaussian kernel convolution operation to capture an infrared effective attention map from the attention map to perfect the initial attention map;

[0013] S3, for the color visible light image, using a first stage of a preset cascaded attention module to capture a color visible light image attention map of the color visible light image;

[0014] S4, using a preset infrared feature extraction module to extract infrared depth features, and using a visible light image feature extraction module to extract image local detail features by using a convolution operation;

[0015] S5, processing the infrared effective attention map, the color visible light image attention map, the infrared depth features of the source image, and the image local detail features to acquire a fusion luminance mapping Y to obtain a fusion luminance map, combining the fusion luminance map with Cb and Cr to reconstruct a color fusion result YCbCr;

[0016] S6, transmitting the color fusion result YCbCr to an RGB space to display the fused image in the RGB space.

[0017] The application provides an infrared and color visible light image end-to-end fusion framework.

[0018] In a more specific technical solution, in step S1, the chrominance information is converted into Cb and Cr channels by the following logic:

[0019]

[0020] In a more specific technical solution, the preset cascade attention mechanism in step S2 is:

[0021] Att=g max (Conv g (g max (ω i *f i ),k)),

[0022] wherein g max represents a max selection strategy of spatial information, which can help the model focus on distinguishing spatial positions in the attention mapping operation; Conv g represents a convolution operation, the kernel is a Gaussian kernel k, and the bias is zero; ω i represents the weight f i of the i-th feature.

[0023] The application introduces a cascade attention mechanism in the fusion network, adopts two optimization stages, and simultaneously calculates the attention map of the visible light image by using the attention mechanism, so as to keep sufficient details. Not only can rich infrared target information be preserved, but also sufficient chrominance, background and details of the color visible light image can be kept. Compared with the gray-scale fusion image, the color fusion image has better visual effect.

[0024] In a more specific technical solution, in step S3, the first stage of the preset cascade attention module is used to capture the attention map of the brightness of the color visible light image.

[0025] In a more specific technical solution, step S4 includes:

[0026] S41, in the preset infrared feature extraction module, ResNet50 is used to extract infrared depth features;

[0027] S42, in the preset visible light image feature extraction module, a convolution operation is used to extract image local detail features.

[0028] In a more specific technical solution, step S5 comprises:

[0029] S51, obtaining fusion brightness according to the fusion brightness map;

[0030] S52, obtaining infrared brightness and color brightness of the infrared image and the color visible light image;

[0031] S53, processing the fusion brightness, the infrared brightness and the color brightness with a preset intensity loss logic to obtain an intensity loss;

[0032] S54, extracting attention maps of the infrared image and the color visible light image, and processing the attention maps with a preset attention loss logic to obtain an attention loss;

[0033] S55, processing the intensity loss and the attention loss to obtain a total loss.

[0034] The present application extracts features and attention maps of the infrared image and the color visible light image through the cascaded attention convolution network, and finds the optimal fusion result through the attention loss and the intensity loss in the fusion module, so as to realize the fusion of the infrared image and the color visible light image.

[0035] In a more specific technical solution, in step S53, the fusion brightness, the infrared brightness and the color brightness are processed with the following logic to obtain the intensity loss:

[0036]

[0037] Wherein, I f-Y , I r and I v-Y are the fusion brightness, the infrared brightness and the color brightness of the image, λ and β are hyperparameters for controlling the balance of the two terms, H and W represent the height and width of the image, and ||·||F F represents the Forbenius norm of the matrix.

[0038] In a more specific technical solution, in step S54, the attention maps are processed with the following logic to obtain the attention loss:

[0039]

[0040] In a more specific technical solution, in step S55, the intensity loss and the attention loss are processed with the following logic to obtain the total loss:

[0041] L=L intensity +L att

[0042] Wherein, L represents the total loss, L intensity and L attrespectively represent intensity loss and attention loss.

[0043] The attention mechanism is introduced into the model of the present application to extract the attention map of the original image, which can force the model of the present application to pay more attention to important areas. Therefore, in order to retain more original image attention area information, the attention loss between the fusion brightness and the original image is designed, the attention mapping difference between the original image and the fusion brightness is punished, and the image fusion effect is optimized.

[0044] In a more specific technical solution, the image fusion system based on the cascaded attention mechanism and the neural network comprises:

[0045] The image space conversion module is used to transmit the color visible light image in the RGB space to the preset YCbCr space, so as to extract the color visible light brightness of the color visible light image, obtain and convert the chroma information into the Cb and Cr channels, and complete the conversion of the image space;

[0046] The infrared image attention map extraction module is used to collect and obtain the infrared image, and a preset cascaded attention mechanism is used to calculate the infrared effective attention map of the infrared image. The infrared image attention map extraction module is connected with the image space conversion module. The preset cascaded attention mechanism in the infrared image attention map extraction module comprises:

[0047] The first optimization stage module is used to use the channel attention feature forced model to reweight the features of the infrared image, calculate the initial attention and the initial attention map, pay attention to the key features, filter out the irrelevant features, obtain the activation value of the hidden neurons, and measure the importance of the neurons, and perform the maximum mapping operation on the Cb and Cr channels to obtain the attention map;

[0048] The second optimization stage module is used to use the preset global attention module to perform the Gaussian kernel convolution operation to capture the infrared effective attention map, so as to perfect the initial attention map. The second optimization stage module is connected with the first optimization stage module.

[0049] The color visible light image attention map extraction module is used to use the first stage processing of the preset cascaded attention module to capture the color visible light image attention map of the color visible light image for the color visible light image. The color visible light image attention map extraction module is connected with the image space conversion module.

[0050] The feature extraction module is used to extract the infrared depth feature by using the preset infrared feature extraction module, and extract the image local detail feature by using the convolution operation in the visible light image feature extraction module. The feature extraction module is connected with the infrared image attention map extraction module and the color visible light image attention map extraction module.

[0051] An image fusion module is used to process the infrared effective attention map, the color visible light image attention map, the infrared depth feature of the source image and the local detail feature of the image, so as to obtain a fusion luminance map Y, so as to obtain a fusion luminance map, and the fusion luminance map is combined with Cb and Cr, so as to reconstruct a color fusion result YCbCr, and the image fusion module is connected with the feature extraction module, the infrared image attention map extraction module and the color visible light image attention map extraction module.

[0052] An image display module is used to transmit the color fusion result YCbCr into the RGB space, so as to display the fused image in the RGB space, and the image display module is connected with the image fusion module.

[0053] Compared with the prior art, the present application has the following advantages: the present application proposes an infrared and color visible light image end-to-end fusion framework. The infrared thermal radiation information of the target is introduced into the color visible light image of the same scene, and the luminance of the color visible light image is refined by using the cascade attention mechanism, which helps to preserve rich infrared target information and visible light image details in the color space. The color fusion image can capture sufficient chrominance, background and detail information, and has good color visual effect.

[0054] The present application introduces a cascade attention mechanism in the fusion network, and adopts two optimization stages, and simultaneously calculates the attention map of the visible light image by using the attention mechanism, so as to preserve sufficient details. Not only can rich infrared target information be preserved, but also sufficient chrominance, background and details of the color visible light image can be preserved. Compared with the gray fusion image, the color fusion image has better visual effect.

[0055] The present application extracts the features and attention maps of the infrared image and the color visible light image by using the cascade attention convolution network, and finds the optimal fusion result in the fusion module by using the attention loss and the intensity loss, so as to realize the fusion of the infrared image and the color visible light image.

[0056] The attention mechanism is introduced into the model of the present application to extract the attention map of the original image, which can force the model of the present application to pay more attention to important areas. Therefore, in order to preserve more attention area information of the original image, the attention loss between the fusion luminance and the original image is designed, the attention mapping difference between the original image and the fusion luminance is punished, and the image fusion effect is optimized. The present application solves the technical problems of excessive dependence on artificial design, poor applicability and decreased visual effect in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 Fig. 1 is a schematic diagram of an infrared and color visible light image fusion framework in the image fusion method based on the cascade attention mechanism and the neural network of the embodiment 1 of the present application.

[0058] Figure 2 The basic step schematic diagram of the image fusion method based on the cascaded attention mechanism and neural network of the embodiment 1 of the present application is shown in the figure.

[0059] Figure 3 The principle diagram of the cascaded attention mechanism of the embodiment 1 of the present application is shown in the figure.

[0060] Figure 4 The specific step schematic diagram of the cascaded attention mechanism of the embodiment 1 of the present application is shown in the figure.

[0061] Figure 5 The specific step schematic diagram of the feature extraction of the embodiment 1 of the present application is shown in the figure.

[0062] Figure 6 The result comparison diagram of the embodiment 2 of the present application and the traditional fusion method and the deep learning method based result is shown in the figure.

[0063] Figure 7a The first quantitative analysis result comparison diagram of the embodiment 2 of the present application is shown in the figure.

[0064] Figure 7b The second quantitative analysis result comparison diagram of the embodiment 2 of the present application is shown in the figure.

[0065] Figure 7c The third quantitative analysis result comparison diagram of the embodiment 2 of the present application is shown in the figure.

[0066] Figure 7d The fourth quantitative analysis result comparison diagram of the embodiment 2 of the present application is shown in the figure. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0068] Embodiment 1

[0069] The image fusion method based on the cascaded attention mechanism and neural network of the present application provides an infrared and color visible light image fusion framework based on the cascaded attention and convolution network. The features and attention maps of the infrared image and the color visible light image are extracted through the cascaded attention convolution network, and the optimal fusion result is found through the attention loss and intensity loss in the fusion module, so as to finally realize the fusion of the infrared image and the color visible light image.

[0070] As Figure 1 and Figure 2As shown in the present embodiment, the image fusion method based on the cascade attention mechanism and neural network of the present application comprises the following steps:

[0071] S1, conversion of image space; in the present embodiment, the color visible light image located in the RGB space is transmitted to the YCbCr space to extract the brightness of the color visible light image, and the chrominance information is converted into the Cb and Cr channels; the conversion formula of RGB and YCbCr is shown in formula (1):

[0072]

[0073] S2, extraction of infrared image attention map; in the present embodiment, for the infrared image, a cascade attention mechanism is used to calculate the effective attention map.

[0074] As shown in Figure 3 and Figure 4 , in the present embodiment, the mechanism comprises the following specific steps:

[0075] S21, in the first stage, the feature is reweighted by using the channel attention feature forced model, the initial attention is calculated, and the initial attention map is calculated to focus on the key features and ignore the irrelevant features. In addition, the importance of the hidden neuron is measured by the activation value of the neuron, and the attention map is calculated by performing maximum mapping operation on the channel;

[0076] S22, in the second stage, a global attention module is designed, and the effective attention map is captured by introducing Gaussian kernel convolution operation to perfect the initial attention map calculated in the first stage. The cascade attention mechanism calculation formula is shown in formula (2):

[0077] Att=g max (Conv g (g max (ω i *f i ),k)),(2)

[0078] Where g max represents the max selection strategy of spatial information, which can help the model focus on the discrimination of spatial position in the attention mapping operation; Conv g represents convolution operation, the kernel is Gaussian kernel k, and the bias is zero; ω i represents the weight f i of the i-th feature.

[0079] S3, extraction of color visible light image attention map; in the present embodiment, for the color visible light image, the first stage of the cascade attention module is used to capture the attention map of the brightness of the color visible light image.

[0080] S4, feature extraction; in the embodiment, two feature extraction modules are adopted.

[0081] As shown in the embodiment, step S4 further comprises the following specific steps: Figure 5

[0082] S41, in the infrared feature extraction module, the application extracts infrared depth features by using ResNet50;

[0083] S42, in the visible light image feature extraction module, convolution operation is adopted to extract image local detail features. The structure of the visible light image feature extraction module is shown in Table 1, wherein I, o, k, s and p represent input size, output size, kernel size, step and padding operation respectively.

[0084] Table 1: visible light image feature extraction module

[0085]

[0086] S5, image fusion; in the embodiment, two attention maps and the depth features of the source image are used to calculate the fusion brightness mapping (fusion Y). Finally, the fused brightness map is combined with Cb and Cr to reconstruct the color fusion result. The structure of the fusion module is shown in Table 2:

[0087] Table 2: fusion module

[0088]

[0089] At the same time, two parts of intensity loss and attention loss are introduced, as shown in formula (3)

[0090] L = L intensity + L att ,(3)

[0091] Wherein L represents the total loss, L intensity and L att represent intensity loss and attention loss respectively.

[0092] Intensity loss: in the fusion method of the application, the target in the infrared image is represented by a high intensity value. Therefore, the application introduces the brightness difference between the infrared brightness and the fusion brightness into the intensity loss. In addition, the brightness of the color visible light image can display the object by the intensity value. Therefore, the difference between the brightness of the color visible light image and the data distribution of the fusion brightness is calculated by using the brightness loss. The design of the intensity loss is shown in formula (4):

[0093]

[0094] Wherein I f-Y , I r and I​v-Y respectively, are the luminance of the fused image, the infrared image and the color visible image. λ and β are hyperparameters that control the balance of the two terms. H and W denote the height and width of the image. F denotes the matrix Forbenius norm.

[0095] Attention Loss: In the method of the present application, the present application introduces an attention mechanism into the model of the present application, extracts the attention map of the original image, which can force the model of the present application to pay more attention to important areas. Therefore, in order to preserve more original image attention area information, the attention loss between the fused luminance and the original image is designed, which punishes the attention mapping difference between the original image and the fused luminance. The attention loss value is shown in formula (5):

[0096]

[0097] S6, conversion of image space; in this embodiment, YCbCr is transmitted into RGB, and the fused image is displayed in RGB space.

[0098] Example 2

[0099] In this embodiment, the model of the present application is trained and tested on the RoadScene dataset 1, which has 221 pairs of aligned infrared and color visible light images, and contains rich scenes such as roads, vehicles, pedestrians, etc. In order to verify the superiority of the proposed scheme over other methods, qualitative and quantitative experiments are carried out

[0100] 1) Qualitative analysis: In this part, the present application uses 50 pairs of images to compare the method of the present application with other 6 methods. That is, the results of traditional fusion methods (CVT, DTCWT and RP) and the results of deep learning based methods (fusionan, DDcGAN and MgAN Fuse) are compared.

[0101] As Figure 6 shown in the comparative example, the fusion results of the traditional method can obtain better fused images, however, compared with the deep learning based method, these results contain more noise or artifacts. In addition, the method of the present application can save more infrared thermal radiation information compared with the deep learning based method and the attention based method.

[0102] 2) Quantitative analysis: Four indicators are used to quantitatively analyze the fusion results. The indicators include SSIM, PSNR, CC and Nabf. For SSIM, PSNR and CC indicators, the larger the value, the better the performance of the method. For the Nabf indicator, the smaller the value, the better the performance. In addition, the present application uses 50 pairs of images to compare the method of the present application with three traditional methods and three deep learning-based methods (including one attention-based method).

[0103] As shown in Figures 7a to 7d The method of the present application is superior to other traditional schemes in the figure in all indicators. Among them, the purpose of the correlation coefficient (CC) is to measure the linear correlation degree of the fused image and the source image, and the model of the present application has the largest value in CC, which shows that it preserves more meaningful information from the source image; the PSNR value of the method is also the largest, indicating that the result of the method has high quality; in addition, the Nabf value of the method is the smallest, indicating that the method has smaller noise and artifacts than other methods.

[0104] From the above quantitative experiment and qualitative experiment, it can be seen that the infrared and color visible light image fusion method based on cascaded attention and convolutional network proposed in the present application can preserve more chrominance information compared with traditional infrared image and grayscale visible light image fusion, and has better visual effect; at the same time, compared with some traditional fusion methods and deep learning fusion methods, the fusion effect of the method proposed in the present application is optimal.

[0105] In summary, the present application proposes an end-to-end fusion framework for infrared and color visible light images. The infrared thermal radiation information of the target is introduced into the color visible light image of the same scene, which refines the brightness of the color visible light image, and helps to preserve rich infrared target information and visible light image details in the color space. The color fusion image can capture sufficient chrominance, background and detail information, and has good color visual effect.

[0106] The present application introduces a cascaded attention mechanism in the fusion network and adopts two optimization stages, and uses the attention mechanism to calculate the attention map of the visible light image to preserve sufficient details. Not only can rich infrared target information be preserved, but also sufficient chrominance, background and details of the color visible light image can be preserved. Compared with the grayscale fusion image, the color fusion image has better visual effect.

[0107] The present application extracts the features and attention maps of the infrared image and the color visible light image through the cascaded attention convolutional network, and finds the optimal fusion result through attention loss and intensity loss in the fusion module, and finally realizes the fusion of the infrared image and the color visible light image.

[0108] The attention mechanism is introduced into the model of the application to extract an attention map of the original image, which can force the model of the application to pay more attention to important areas. Therefore, in order to retain more original image attention area information, the attention loss between the fusion brightness and the original image is designed, the attention mapping difference between the original image and the fusion brightness is punished, and the image fusion effect is optimized. The application solves the technical problems of excessive dependence on artificial design, poor applicability and decline of visual effect in the prior art.

[0109] The above examples are only used to illustrate the technical solutions of the application, but not to limit them; although the application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. An image fusion method based on cascaded attention mechanism and neural network, characterized in that, The method includes: S1. Transmit the color visible light image in RGB space to the preset YCbCr space, thereby extracting the color visible light brightness of the color visible light image, obtaining and converting the chromaticity information into Cb and Cr channels to complete the image space conversion; S2. Acquire an infrared image, and calculate the effective infrared attention map of the infrared image using a preset cascaded attention mechanism. The preset cascaded attention mechanism in step S2 includes: S21. Using the channel attention feature forced model, the features of the infrared image are re-weighted and processed, and the initial attention and initial attention map are calculated to focus on key features and filter out irrelevant features. The activation values ​​of hidden neurons are obtained to measure the importance of neurons. Maximum value mapping operation is performed on the Cb and Cr channels to obtain the attention map. S22. Perform Gaussian kernel convolution operation using a preset global attention module to capture the infrared effective attention map from the attention map, thereby improving the initial attention map; S3. For a color visible light image, the first stage of processing using a pre-set cascaded attention module is used to capture the color visible light image attention map of the color visible light image. S4. Use the preset infrared feature extraction module to extract infrared depth features, and use convolution operation in the preset visible light image feature extraction module to extract local detail features of the image; S5. Process the infrared effective attention map, the color visible light image attention map, the infrared depth features of the source image, and the local detail features of the image to obtain the fusion brightness map Y, thereby obtaining the fusion brightness map. Combine the fusion brightness map with Cb and Cr to reconstruct the color fusion result YCbCr. S6. Image space conversion: The color fusion result YCbCr is transferred to the RGB space to display the fused image in the RGB space.

2. The image fusion method based on cascaded attention mechanism and neural network according to claim 1, characterized in that, In step S1, the chromaticity information is converted into the Cb and Cr channels using the following logic: 。 3. The image fusion method based on cascaded attention mechanism and neural network according to claim 1, characterized in that, The pre-set cascaded attention mechanism in step S2 is as follows: in, The max selection strategy represents spatial information, which helps the model focus on discriminating spatial locations during attention mapping operations; This indicates a convolution operation with a Gaussian kernel k and a zero bias. The weight of the i-th feature .

4. The image fusion method based on cascaded attention mechanism and neural network according to claim 1, characterized in that, In step S3, the attention map of the brightness of the color visible light image is captured using the first stage of the preset cascaded attention module.

5. The image fusion method based on cascaded attention mechanism and neural network according to claim 1, characterized in that, Step S4 includes: S41. In the preset infrared feature extraction module, the infrared depth features are extracted using ResNet50; S42. In the preset visible light image feature extraction module, convolution operation is used to extract the local detail features of the image.

6. The image fusion method based on cascaded attention mechanism and neural network according to claim 1, characterized in that, Step S5 includes: S51. Obtain the fused brightness based on the fused brightness map; S52. Obtain the infrared brightness and color brightness of the infrared image and the color visible light image; S53. Process the fused brightness, the infrared brightness, and the color brightness using a preset intensity loss logic to obtain the intensity loss; S54. Extract the attention maps of the infrared image and the color visible light image, and process the attention maps with preset attention loss logic to obtain attention loss; S55. Process the intensity loss and the attention loss to obtain the total loss.

7. The image fusion method based on cascaded attention mechanism and neural network according to claim 6, characterized in that, In step S53, the fused brightness, the infrared brightness, and the color brightness are processed using the following logic to obtain the intensity loss: in, , and λ and β represent the fused brightness, the brightness of the infrared image, and the brightness of the color visible light image, respectively; λ and β are hyperparameters controlling the balance between the two; and H and W represent the height and width of the image. This represents the Forbenius norm of a matrix.

8. The image fusion method based on cascaded attention mechanism and neural network according to claim 6, characterized in that, In step S54, the attention map is processed using the following logic to obtain the attention loss: 。 9. The image fusion method based on cascaded attention mechanism and neural network according to claim 6, characterized in that, In step S55, the intensity loss and the attention loss are processed using the following logic to obtain the total loss: in, L Represents all losses, and These represent the intensity loss and the attention loss, respectively.

10. An image fusion system based on cascaded attention mechanism and neural network, characterized in that, The system includes: The image space conversion module is used to transmit a color visible light image in RGB space to a preset YCbCr space, thereby extracting the color visible light brightness of the color visible light image, obtaining and converting the chromaticity information into Cb and Cr channels to complete the image space conversion. An infrared image attention map extraction module is used to acquire infrared images and calculate the effective infrared attention map of the infrared images using a preset cascaded attention mechanism. The infrared image attention map extraction module is connected to the image space transformation module. The preset cascaded attention mechanism in the infrared image attention map extraction module includes: The first optimization stage module is used to force the model by utilizing channel attention features, re-weight the features of the infrared image, calculate the initial attention and initial attention map to focus on key features, filter out irrelevant features, obtain the activation values ​​of hidden neurons to measure the importance of neurons, and perform maximum value mapping operation on the Cb and Cr channels to obtain the attention map. The second optimization stage module is used to perform Gaussian kernel convolution operation using a preset global attention module to capture the infrared effective attention map, thereby improving the initial attention map. The second optimization stage module is connected to the first optimization stage module. A color visible light image attention map extraction module is used to capture the color visible light image attention map of the color visible light image by using the first stage processing of a preset cascaded attention module. The color visible light image attention map extraction module is connected to the image space conversion module. The feature extraction module is used to extract infrared depth features using a preset infrared feature extraction module, and to extract local detail features of the image using convolution operations in a preset visible light image feature extraction module. The feature extraction module is connected to the infrared image attention map extraction module and the color visible light image attention map extraction module. An image fusion module is used to process an infrared effective attention map, a color visible light image attention map, the infrared depth features of the source image, and the local detail features of the image to obtain a fused brightness map Y. The fused brightness map is then combined with Cb and Cr to reconstruct the color fusion result YCbCr. The image fusion module is connected to the feature extraction module, the infrared image attention map extraction module, and the color visible light image attention map extraction module. An image display module is used to transmit the color fusion result YCbCr to the RGB space to display the fused image in the RGB space. The image display module is connected to the image fusion module.

Images