A method and system for image fusion based on deep learning

Abstract

The present invention relates to the field of image processing, and the present invention relates to a method and system for image fusion based on deep learning, the method comprising: inputting an infrared image into a first encoder to obtain an infrared feature map; inputting a visible light image into a second encoder to obtain a visible light feature map; inputting an X-ray image into a third encoder to obtain an X-ray feature map; using a CAM cross attention mechanism to preliminarily fuse the infrared feature map, the visible light feature map and the X-ray feature map to obtain an initial fused feature map; inputting the visible light feature map, the infrared feature map, the X-ray feature map and the initial fused feature map into a decoder to generate a fused image. The method of the present invention can effectively improve the fusion effect of visible light images, infrared images and X-ray images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology. More specifically, the present invention relates to a method and system for image fusion based on deep learning. Background Art

[0002] In the field of modern industrial inspection, ensuring product quality and safety is a vital task. Traditionally, X-ray, visible light and infrared images are widely used for defect detection. X-ray images can provide a detailed view of the internal structure and identify internal defects such as cracks, bubbles and voids; visible light images show the state of the product surface, including features such as scratches, dirt and uneven color; and infrared images can effectively detect heat distribution and material properties, identifying problems such as overheating and insulation defects. Although these three images can be used separately to obtain their own information, they still have significant limitations when used alone. For example, X-ray images cannot accurately display surface features, while visible light images and infrared images cannot reveal internal structures. In some cases, internal defects that are visible in X-ray images may appear normal in visible light images and infrared images, and vice versa. Therefore, if one of these three images is used alone to detect the quality of the product, important defect information may be missed, resulting in misjudgment of the overall state and inaccurate inspection results. In addition, analyzing the three images separately when inspecting product quality will take more time and resources, especially when processing high-resolution images, which may reduce the efficiency of defect detection.

[0003] Therefore, when testing product quality, it is usually necessary to use a fusion algorithm to fuse the three images. Although traditional image fusion algorithms, such as weighted average and wavelet transform, can achieve image fusion to a certain extent, they have significant shortcomings. These methods often cannot fully retain all the key information in the input image, resulting in blurred details and poor fusion effect. In addition, traditional algorithms usually rely on manually designed features, lack automation and adaptability, and cannot meet the detection needs in complex industrial environments. For example, a Chinese invention patent with an application authorization announcement number of CN111861957B discloses an image fusion method and device, which uses a wavelet transform method to fuse multiple images to be fused, but the fused image cannot fully retain all the key information in the input image, resulting in blurred details and poor fusion effect. Summary of the invention

[0004] In order to solve the technical problem that the image fusion method in the prior art cannot fully retain all the key information in the input image after fusion, resulting in blurred details and poor fusion effect, the present invention provides solutions in the following aspects.

[0005] In a first aspect, the present invention provides a picture fusion method based on deep learning, comprising:

[0006] Inputting the infrared image into the first encoder to obtain an infrared characteristic map; inputting the visible light image into the second encoder to obtain a visible light characteristic map; inputting the X-ray image into the third encoder to obtain an X-ray characteristic map;

[0007] Using the CAM cross attention mechanism to preliminarily fuse the infrared feature map, the visible light feature map and the X-ray feature map, so as to obtain an initial fused feature map;

[0008] The visible light feature map, the infrared feature map, the X-ray feature map and the initial fused feature map are input into a decoder to generate a fused image; the decoder is used to fuse the visible light feature map, the infrared feature map, the X-ray feature map and the initial fused feature map to obtain a final fused feature map, and classify the pixels of the final fused feature map to generate a fused image.

[0009] The beneficial effects are as follows: although traditional image fusion algorithms, such as weighted average and wavelet transform, can achieve image fusion to a certain extent, they have significant deficiencies; because these methods often cannot fully retain all the key information in the input image, resulting in blurred details and poor fusion effect. The image fusion method based on deep learning of the present invention uses three encoders to extract the features of infrared images, visible light images and X-ray images respectively, and uses the CAM cross-attention mechanism to fuse the features of infrared images, visible light images and X-ray images; since the fused features will lose some original feature information, the method of the present invention further fuses the initial fusion feature map, the original visible light image, the original infrared image and the original X-ray image after obtaining the initial fusion feature map to generate the final fusion image, thereby retaining the original feature information (including deep / shallow features) of the visible light image, infrared image and X-ray image as much as possible, thereby greatly improving the quality of the generated fusion image. Therefore, the method of the present invention can make the fused image retain all the key information in the input image as much as possible, thereby effectively improving the fusion effect of the visible light image, infrared image and X-ray image.

[0010] Preferably, the first encoder includes: a convolution layer, a pooling layer, and four convolution blocks, wherein the first convolution layer is used to extract shallow features of the image, the output of the first convolution layer is connected to the input of the pooling layer, the output of the pooling layer is connected to the input of the first convolution block, the output of the first convolution block is connected to the input of the second convolution block, the output of the second convolution block is connected to the input of the third convolution block, and the output of the third convolution block is connected to the input of the fourth convolution block; the first encoder, the second encoder and the third encoder have the same structure.

[0011] The beneficial effects are as follows: after the infrared image is input into the first encoder, it first passes through the first convolution layer to obtain the shallow features of the infrared image, and the shallow features of the infrared image are pooled by the pooling layer to obtain the pooled shallow feature information; the pooled shallow feature information subsequently passes through four convolution blocks, so that more useful information in the multi-scale features can be retained. As the number of encoder layers becomes deeper and deeper, the deep features begin to focus on significant targets; therefore, by using the first encoder of the present invention, more useful information can be extracted from the infrared image, thereby further improving the fusion effect of visible light images, infrared images and X-ray images.

[0012] Preferably, the convolution block is composed of three convolutions, and the three convolutions are connected by residuals.

[0013] Preferably, the preliminary fusion of the infrared feature map, the visible light feature map and the X-ray feature map includes: respectively obtaining the features to be fused of the infrared feature map, the features to be fused of the visible light feature map and the features to be fused of the X-ray feature map, and adding the above three features to be fused to obtain an initial fused feature map; obtaining the features to be fused of a feature map of a certain modality includes: inputting the feature map of the modality into a first self-attention mechanism to obtain a first feature map; moving the feature position of the first feature map horizontally and vertically, and inputting the first feature map after the feature position is moved into a second self-attention mechanism to obtain a second feature map; resetting the feature position of the second feature map and inputting the reset second feature map into a coordinate attention mechanism to obtain the features to be fused of the feature map of the modality.

[0014] The beneficial effect is as follows: when obtaining the features to be fused of the feature map, the method of the present invention first enhances the internal features of the feature map through the first self-attention mechanism, and then moves the feature position of the feature map after the internal features are enhanced and inputs the second self-attention mechanism, thereby obtaining a more advanced and accurate feature map; and then further processes it using the coordinate attention mechanism, introduces spatial coordinate information into the feature map to enhance the expression ability of the feature map; thereby improving the effect and quality of the initial fused feature map obtained.

[0015] Preferably, the decoder is a four-layer convolution structure, and an upsampling layer is arranged between adjacent convolution layers, and the decoder is jump-connected to the first encoder, the second encoder and the third encoder respectively.

[0016] Preferably, the first encoder is a neural network model, and its training method includes:

[0017] Inputting the infrared image into a first encoder to obtain an infrared feature map;

[0018] Calculating the difference between the infrared feature map and the infrared image input into the first encoder using a first loss function;

[0019] Use the back-propagation algorithm to calculate the gradient of the loss function with respect to the model parameters;

[0020] Using an optimization algorithm to update parameters of the first encoder;

[0021] The parameters of the first encoder are iteratively updated until a preset number of training rounds is reached.

[0022] Preferably, the expression of the first loss function is:

[0023] ;

[0024] in, represents the difference between the infrared feature map and the infrared image input into the first encoder, represents the infrared image input into the first encoder, Infrared characteristic map is the Frobenius norm, which is used to measure the difference between the input image and the reconstructed image at the pixel level. is the structural similarity index, is a balance parameter.

[0025] The beneficial effects are as follows: in the prior art, square damage is usually used when calculating the difference between two images; the first loss function of the present invention comprehensively considers pixel-level loss and the structural similarity of the two images when calculating the gap between the infrared feature map output by the first encoder and the infrared image input to the first encoder. Considering pixel-level loss can avoid the problem of large global error; considering the structural similarity of the two images rather than just the difference in pixel values, compared with square loss, the structural similarity index can better reflect the perceptual characteristics of the human eye, and using the first loss function of the present invention to train the encoder can effectively improve the quality of the infrared feature map output by the first encoder.

[0026] Preferably, the CAM cross attention mechanism and the decoder training method include:

[0027] After the first encoder, the second encoder and the third encoder are trained, the fused image is obtained by using the first encoder, the second encoder, the third encoder, the CAM cross attention mechanism and the decoder;

[0028] Using a second loss function, the loss of the fused image relative to the infrared image input to the first encoder, the visible light feature map input to the second encoder, and the X-ray image input to the third encoder is calculated;

[0029] Using the back propagation algorithm to calculate the gradient of the second loss function with respect to the model parameters;

[0030] Use optimization algorithms to update the parameters of the CAM cross-attention mechanism and decoder;

[0031] Iteratively update the parameters of the CAM cross-attention mechanism and the decoder until the preset number of training rounds is reached.

[0032] Preferably, the expression of the second loss function is:

[0033] ;

[0034] ;

[0035] In the formula, represents the first loss, is a trade-off parameter used to balance the weights of the two loss terms, Indicates the strength loss, represents the gradient loss, F represents the fused image, They are the intensity mask of infrared image, the intensity mask of visible light image and the intensity mask of X-ray image; They are infrared images, visible light images and X-ray images. is the Frobenius norm, which is used to measure the difference between the input image and the reconstructed image at the pixel level.

[0036] The beneficial effects are as follows: by introducing intensity loss and gradient loss in the second loss function, not only the unique features from each modality are retained, but also redundant information is reduced, and the quality of the fused image is improved. By introducing the intensity mask, the fusion process is optimized, redundancy is reduced, and the expressiveness of the fused image is improved. By introducing the gradient loss, the fused image can be guaranteed to be more visually natural and better preserve the contours and details of the target.

[0037] In a second aspect, the present invention provides a deep learning-based image fusion system, comprising: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the deep learning-based image fusion method of the present invention is implemented.

[0038] In summary, the beneficial effect of the present invention is that the method of the present invention can effectively improve the fusion effect of visible light images, infrared images and X-ray images. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0040] Figure 1 The figure schematically shows the process of the image fusion method based on deep learning according to an embodiment of the present invention. Figure 1 ;

[0041] Figure 2 The figure schematically shows the process of the image fusion method based on deep learning according to an embodiment of the present invention. Figure 2 ;

[0042] Figure 3 FIG. 1 schematically illustrates an image fusion system based on deep learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0044] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0045] Example of image fusion method based on deep learning:

[0046] like Figure 1 and Figure 2 As shown, the image fusion method based on deep learning of the present invention includes:

[0047] S101, obtaining an infrared characteristic map, a visible light characteristic map, and an X-ray characteristic map, specifically: inputting an infrared image into a first encoder to obtain an infrared characteristic map; inputting a visible light image into a second encoder to obtain a visible light characteristic map; inputting an X-ray image into a third encoder to obtain an X-ray characteristic map;

[0048] The first encoder, the second encoder and the third encoder are used to extract features from infrared images, visible light images and X-ray images respectively to generate corresponding feature maps. The extracted features may include edges, textures and target contours. Before the infrared images, visible light images and X-ray images are input to the corresponding encoders, the images need to be preprocessed by denoising, contrast enhancement and normalization to improve the quality and consistency of the images; and the images need to be adjusted to the same size and resolution for subsequent processing and fusion.

[0049] S102, obtaining an initial fused feature map, specifically: using a CAM cross attention mechanism to preliminarily fuse the infrared feature map, the visible light feature map, and the X-ray feature map, thereby obtaining an initial fused feature map;

[0050] The Cross-Attention Mechanism is a mechanism that establishes associations between two different input sequences and calculates attention weights. It allows the model to focus on elements in another input sequence when processing elements of one input sequence, thereby capturing the dependency between the two sequences. In image fusion, the cross-attention mechanism allows the model to focus on the features of one modality on the features of another modality, thereby achieving information interaction between different modalities. In the image feature fusion task, the CAM cross-attention mechanism can simultaneously process image data in different representations (such as infrared images, visible light images, and X-ray images) and capture complementary information between these images. The information interaction between images of different modalities helps the model generate more comprehensive and accurate fusion features.

[0051] S103, generating a fused image, specifically: inputting the visible light feature map, the infrared feature map, the X-ray feature map and the initial fused feature map into a decoder to generate a fused image; the decoder is used to fuse the visible light feature map, the infrared feature map, the X-ray feature map and the initial fused feature map to obtain a final fused feature map, and classifying the pixels of the final fused feature map to generate a fused image.

[0052] A skip connection may be set between the decoder and the three encoders respectively, so as to fuse the visible light feature map, the infrared feature map, the X-ray feature map and the initial fusion feature map.

[0053] Traditional image fusion algorithms, such as weighted averaging and wavelet transform, can achieve image fusion to a certain extent, but have significant shortcomings; because these methods often cannot fully retain all the key information in the input image, resulting in blurred details and poor fusion effect. The image fusion method based on deep learning of the present invention uses three encoders to extract the features of infrared images, visible light images and X-ray images respectively, and uses the CAM cross-attention mechanism to fuse the features of infrared images, visible light images and X-ray images; because the fused features will lose some of the original feature information, the method of the present invention further fuses the initial fusion feature map, the original visible light image, the original infrared image and the original X-ray image after obtaining the initial fusion feature map to generate the final fusion image, thereby retaining the original feature information (including deep / shallow features) of the visible light image, infrared image and X-ray image as much as possible, thereby greatly improving the quality of the generated fusion image. In summary, the method of the present invention can effectively improve the fusion effect of visible light images, infrared images and X-ray images.

[0054] In one embodiment, the first encoder includes: a convolution layer, a pooling layer, and four convolution blocks, wherein the first convolution layer is used to extract shallow features of the image, the output of the first convolution layer is connected to the input of the pooling layer, the output of the pooling layer is connected to the input of the first convolution block, the output of the first convolution block is connected to the input of the second convolution block, the output of the second convolution block is connected to the input of the third convolution block, and the output of the third convolution block is connected to the input of the fourth convolution block; the first encoder, the second encoder and the third encoder have the same structure.

[0055] After the infrared image is input into the first encoder, it first passes through the first convolution layer to obtain the shallow features of the infrared image. The shallow features of the infrared image are pooled by the pooling layer to obtain the pooled shallow feature information; the pooled shallow feature information is subsequently passed through four convolution blocks, so that more useful information in the multi-scale features can be retained. As the number of encoder layers becomes deeper and deeper, the deep features begin to focus on significant targets; therefore, by using the first encoder of the present invention, more useful information can be extracted from the infrared image, thereby further improving the fusion effect of visible light images, infrared images and X-ray images.

[0056] In one embodiment, the convolution block is composed of three convolutions, and the three convolutions are connected by residuals.

[0057] In one embodiment, the preliminary fusion of the infrared feature map, the visible light feature map and the X-ray feature map includes: respectively obtaining the features to be fused of the infrared feature map, the features to be fused of the visible light feature map and the features to be fused of the X-ray feature map, and adding the above three features to be fused to obtain an initial fused feature map; obtaining the features to be fused of a feature map of a certain modality includes:

[0058] S201, inputting the feature map of the modality into a first self-attention mechanism to obtain a first feature map; performing an operation of moving the feature position of the first feature map horizontally and vertically, and inputting the first feature map after the feature position is moved into a second self-attention mechanism to obtain a second feature map;

[0059] The self-attention mechanism, also known as the intra-attention mechanism, is an attention mechanism that associates different positions of a single sequence to calculate the representation of the same sequence. In the self-attention mechanism, the model can automatically learn the correlation between different positions in the sequence and generate a weighted vector for each position as the representation of the position in the current context.

[0060] The self-attention mechanism generates different weights for each feature map position, which are dynamically calculated based on the current content of the feature map. This enables the model to adaptively focus on important feature parts, thereby improving the model's representation ability. The self-attention mechanism achieves information interaction on a global scale by calculating the correlation between any two positions in the feature map. This helps the model obtain more contextual information, thereby improving the accuracy of image understanding. After the feature map of this modality is input into the first self-attention mechanism, its internal features can be enhanced.

[0061] Moving the feature position can change the spatial layout of elements in the feature map, thereby promoting interaction and fusion between different features; this interaction and fusion helps the model capture more complex and abstract feature patterns; therefore, after obtaining the first feature map, the feature position is moved horizontally and vertically, and then input into the second self-attention mechanism. The second self-attention mechanism can further capture the global dependencies between features based on these moved feature maps, thereby further refining and combining features to form a more advanced and accurate feature map.

[0062] S202. Reset the feature position of the second feature map and input the reset second feature map into the coordinate attention mechanism, so as to obtain the features to be fused of the feature map of the modality.

[0063] The coordinate attention mechanism can be used to introduce spatial coordinate information to enhance the expressiveness of feature maps. The feature maps processed by the coordinate attention mechanism contain channel information and precise spatial position information, providing richer and more accurate feature representation for subsequent image understanding tasks.

[0064] When acquiring the features to be fused of the feature map, the method of this embodiment first enhances the internal features of the feature map through the first self-attention mechanism, then moves the feature position of the feature map after the internal features are enhanced and inputs into the second self-attention mechanism, thereby obtaining a more advanced and accurate feature map; and then further processes it using the coordinate attention mechanism, introduces spatial coordinate information into the feature map to enhance the expression ability of the feature map; thereby improving the effect and quality of the obtained initial fused feature map.

[0065] In one embodiment, the decoder is a four-layer convolution structure, and an upsampling layer is arranged between adjacent convolution layers. The decoder is jump-connected to the first encoder, the second encoder and the third encoder respectively.

[0066] In one embodiment, the first encoder is a neural network model, and the training method thereof includes:

[0067] S301, inputting the infrared image into a first encoder to obtain an infrared feature map;

[0068] Before training, it is necessary to obtain a sufficient number of X-ray images, infrared images, and visible light images and annotate them as a data set. Make sure that these images are taken of the same object in the same scene or target.

[0069] S302, using a first loss function to calculate the difference between the infrared feature map and the infrared image input into the first encoder;

[0070] S303, using a back propagation algorithm to calculate the gradient of the loss function relative to the model parameters;

[0071] S304, using an optimization algorithm to update parameters of the first encoder;

[0072] S305: Iteratively update the parameters of the first encoder until a preset number of training rounds is reached.

[0073] In one embodiment, the expression of the first loss function is:

[0074] ;

[0075] in, represents the difference between the infrared feature map and the infrared image input into the first encoder, represents the infrared image input into the first encoder, Infrared characteristic map is the Frobenius norm, which is used to measure the difference between the input image and the reconstructed image at the pixel level. is the structural similarity index, is a balance parameter.

[0076] In the prior art, square damage is usually used when calculating the difference between two images. The first loss function of this embodiment comprehensively considers pixel-level loss and the structural similarity of the two images when calculating the gap between the infrared feature map output by the first encoder and the infrared image input to the first encoder. Considering pixel-level loss can avoid the problem of large global error. The structural similarity of the two images is taken into account rather than just the difference in pixel values. Compared with square loss, the structural similarity index can better reflect the perceptual characteristics of the human eye. Using the first loss function of this embodiment to train the encoder can effectively improve the quality of the infrared feature map output by the first encoder.

[0077] In one embodiment, the CAM cross attention mechanism and the decoder training method include:

[0078] S401, inputting the infrared image, the visible light image and the X-ray image into a trained first encoder, a trained second encoder and a trained third encoder respectively, so as to obtain an infrared feature map, a visible light feature map and an X-ray feature map;

[0079] S402, inputting the infrared feature map, the visible light feature map and the X-ray feature map into the CAM cross attention mechanism, thereby obtaining an initial fusion feature map;

[0080] S403, inputting the visible light feature map, the infrared feature map, the X-ray feature map and the initial fused feature map into a decoder, thereby generating a fused image;

[0081] S404, using a second loss function to calculate a first loss of the fused image relative to the infrared image input to the first encoder, the visible light feature map input to the second encoder, and the X-ray image input to the third encoder;

[0082] S405, using a back propagation algorithm to calculate the gradient of the second loss function relative to the model parameters;

[0083] S406, using an optimization algorithm to update the parameters of the CAM cross attention mechanism and the decoder;

[0084] S407, iteratively update the parameters of the CAM cross-attention mechanism and the decoder until a preset number of training rounds is reached.

[0085] In one embodiment, the expression of the second loss function is:

[0086] ;

[0087] ;

[0088] In the formula, represents the first loss, is a trade-off parameter used to balance the weights of the two loss terms, Indicates the strength loss, represents the gradient loss, F represents the fused image, They are the intensity mask of infrared image, the intensity mask of visible light image and the intensity mask of X-ray image; They are infrared images, visible light images and X-ray images. is the Frobenius norm, which is used to measure the difference between the input image and the reconstructed image at the pixel level; It represents the loss of the fused image relative to the infrared image input to the first encoder, the visible light feature map input to the second encoder, and the X-ray image input to the third encoder.

[0089] The intensity loss is used to keep the main parts of the fused image, such as brightness and contours. This information usually does not appear in only one modality. Therefore, an intensity mask is introduced when calculating the intensity loss to ensure that the key information of different modalities is preserved.

[0090] By introducing intensity loss and gradient loss in the second loss function, not only the unique features from each modality are retained, but also redundant information is reduced, and the quality of the fused image is improved. By introducing the intensity mask, the fusion process is optimized, redundancy is reduced, and the expressiveness of the fused image is improved. By introducing the gradient loss, the fused image can be guaranteed to be more visually natural and better preserve the contours and details of the target.

[0091] Example of image fusion system based on deep learning:

[0092] The present invention also provides a picture fusion system based on deep learning. Figure 3 As shown, the deep learning-based image fusion system includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the deep learning-based image fusion method in the above embodiment is implemented.

[0093] The deep learning-based image fusion system also includes other components familiar to those skilled in the art, such as a communication bus and a communication interface, whose settings and functions are known in the art and will not be described in detail here.

[0094] In the present invention, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus or device. For example, a computer-readable storage medium may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by an application, a module, or both. Any such computer storage medium may be part of a device or accessible or connectable to a device. Any application or module described in the present invention may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0095] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

Claims

1. A deep learning-based image fusion method, characterized in that: include: Inputting the infrared image into a first encoder to obtain an infrared feature map; Inputting the visible light image into a second encoder to obtain a visible light feature map; Inputting the X-ray image into a third encoder to obtain an X-ray characteristic map; Using the CAM cross attention mechanism to preliminarily fuse the infrared feature map, the visible light feature map and the X-ray feature map, so as to obtain an initial fused feature map; Inputting the visible light feature map, the infrared feature map, the X-ray feature map and the initial fused feature map into a decoder to generate a fused image; The decoder is used to fuse the visible light feature map, the infrared feature map, the X-ray feature map and the initial fused feature map to obtain a final fused feature map, and classify the pixels of the final fused feature map to generate a fused image; The training method of the CAM cross attention mechanism and decoder includes: after the first encoder, the second encoder and the third encoder are trained, the first encoder, the second encoder, the third encoder, the CAM cross attention mechanism and the decoder are used to obtain a fused image; The first loss of the fused image relative to the infrared image input to the first encoder, the visible light feature map input to the second encoder, and the X-ray image input to the third encoder is calculated using the second loss function; the gradient of the second loss function relative to the model parameters is calculated using the back propagation algorithm; Use optimization algorithms to update the parameters of the CAM cross-attention mechanism and decoder; Iteratively update the parameters of the CAM cross-attention mechanism and the decoder until the preset number of training rounds is reached.

2. The image fusion method based on deep learning as claimed in claim 1, characterized in that: The first encoder includes: a convolution layer, a pooling layer, and four convolution blocks, wherein the first convolution layer is used to extract shallow features of the image, the output of the first convolution layer is connected to the input of the pooling layer, the output of the pooling layer is connected to the input of the first convolution block, the output of the first convolution block is connected to the input of the second convolution block, the output of the second convolution block is connected to the input of the third convolution block, and the output of the third convolution block is connected to the input of the fourth convolution block; the first encoder, the second encoder and the third encoder have the same structure.

3. The image fusion method based on deep learning as claimed in claim 2, characterized in that: The convolution block is composed of three convolutions, and the three convolutions are connected by residuals.

4. The image fusion method based on deep learning as claimed in claim 1, characterized in that: The preliminary fusion of the infrared feature map, the visible light feature map and the X-ray feature map includes: respectively obtaining the features to be fused of the infrared feature map, the features to be fused of the visible light feature map and the features to be fused of the X-ray feature map, and adding the three features to be fused to obtain an initial fused feature map; obtaining the features to be fused of a feature map of a certain modality includes: inputting the feature map of the modality into a first self-attention mechanism to obtain a first feature map; performing an operation of moving the feature position of the first feature map in the horizontal and vertical directions, and inputting the first feature map after the feature position is moved into a second self-attention mechanism to obtain a second feature map; The feature position of the second feature map is reset and the reset second feature map is input into the coordinate attention mechanism to obtain the features to be fused of the feature map of the modality.

5. The image fusion method based on deep learning as claimed in claim 1, characterized in that: The decoder is a four-layer convolution structure, and an upsampling layer is arranged between adjacent convolution layers. The decoder is jump-connected to the first encoder, the second encoder and the third encoder respectively.

6. The image fusion method based on deep learning according to claim 1, characterized in that: The first encoder is a neural network model, and its training method includes: Inputting the infrared image into a first encoder to obtain an infrared feature map; Calculating the difference between the infrared feature map and the infrared image input into the first encoder using a first loss function; Use the back-propagation algorithm to calculate the gradient of the loss function with respect to the model parameters; Using an optimization algorithm to update parameters of the first encoder; The parameters of the first encoder are iteratively updated until a preset number of training rounds is reached.

7. The image fusion method based on deep learning as claimed in claim 6, characterized in that: The expression of the first loss function is: ； in, represents the difference between the infrared feature map and the infrared image input into the first encoder, represents the infrared image input into the first encoder, Infrared characteristic map is the Frobenius norm, which is used to measure the difference between the input image and the reconstructed image at the pixel level. is the structural similarity index, is a balance parameter.

8. The image fusion method based on deep learning as claimed in claim 1, characterized in that: The expression of the second loss function is: ； ； In the formula, represents the first loss, is a trade-off parameter used to balance the weights of the two loss terms, Indicates the strength loss, represents the gradient loss, F represents the fused image, They are the intensity mask of infrared image, the intensity mask of visible light image and the intensity mask of X-ray image; They are infrared images, visible light images and X-ray images. is the Frobenius norm, which is used to measure the difference between the input image and the reconstructed image at the pixel level.

9. A deep learning-based image fusion system, comprising: A processor and a memory, wherein the memory stores computer program instructions, wherein when the computer program instructions are executed by the processor, the deep learning-based image fusion method described in any one of claims 1 to 8 is implemented.

Images