Multimodal image fusion model construction method and multimodal image fusion method

Abstract

The present invention relates to the technical field of optoelectronic pods, and discloses a method for constructing a multimodal image fusion model and a multimodal image fusion method, including: obtaining sample images of each mode collected by different sensors of an optoelectronic pod; inputting the sample images into an initial multimodal image fusion model, performing deep feature extraction and feature reconstruction on the sample images; calculating the gradient loss and pixel loss between the fused image and the sample image; decomposing the fused image using a pre-trained image decomposition network, calculating the auxiliary loss between the reconstructed image and the sample image; calculating the joint loss; and training the initial multimodal image fusion model using an optimizer and the joint loss. The present invention significantly improves the effect of the fusion network and the ability to generalize the fusion of multiple modal images, and also greatly improves the speed and efficiency of image processing, realizes efficient fusion of image data from different modalities, and provides more accurate and comprehensive image information.

Description

Technical Field

[0001] The present invention relates to the technical field of optoelectronic pods, and in particular to a multimodal image fusion model construction method and a multimodal image fusion method. Background Art

[0002] Photoelectric pod technology integrates multiple sensor systems and is a key device in the fields of aviation, aerospace and high-end monitoring. These systems usually include infrared cameras, visible light cameras, etc., which can capture image information from different modalities. In practical applications, in order to obtain more comprehensive and accurate visual information, it is necessary to use efficient image fusion technology to comprehensively process image data from these different modalities.

[0003] In the related technologies, although the multimodal image fusion technology in the field of optoelectronic pod technology has made certain progress, the existing technology still shows some limitations. The current image fusion methods mainly rely on a single deep learning network to solve specific fusion problems, such as infrared and visible light image fusion, multi-exposure image fusion or multi-focus image fusion. These methods are often optimized only for a single or a few cases, and are difficult to be widely used in a variety of different image fusion scenarios, thus limiting the versatility of the technology. In addition, optoelectronic pod systems often need to achieve real-time or near real-time image processing in actual operation, and the existing technology still needs to be improved in terms of processing speed and efficiency. Summary of the invention

[0004] In view of this, the present invention provides a multimodal image fusion model construction method and a multimodal image fusion method to solve the problem that it is difficult to effectively fuse different modal image data collected by an optoelectronic pod.

[0005] In a first aspect, the present invention provides a method for constructing a multimodal image fusion model, the method comprising:

[0006] Obtain sample images of each mode collected by different sensors of the optoelectronic pod;

[0007] The sample image is input into the initial multimodal image fusion model, deep feature extraction and feature reconstruction are performed on the sample image to obtain a fused image corresponding to the sample image;

[0008] Calculate the gradient loss and pixel loss between the fused image and the sample image;

[0009] Decompose the fused image using a pre-trained image decomposition network to obtain the corresponding reconstructed image of the fused image, and calculate the auxiliary loss between the reconstructed image and the sample image;

[0010] Based on gradient loss, pixel loss and auxiliary loss, the joint loss is calculated;

[0011] The initial multimodal image fusion model is trained using the optimizer and the joint loss to obtain the target multimodal image fusion model.

[0012] In the present invention, by utilizing the image fusion network of the decomposition network and the cross-attention mechanism, it is possible to effectively process and fuse image data of different modalities collected from the optoelectronic pod, such as infrared images, visible light images, images of different focal lengths and exposure levels, etc., which significantly improves the generalization and processing capabilities of the constructed multimodal image fusion model for images of multiple modalities, and helps to obtain more comprehensive visual information in complex environments. Image data of different modalities can be processed and fused more accurately, providing richer and more detailed image information, significantly improving the effect of the fusion network and the fusion generalization capabilities of multiple modal images, and greatly improving the speed and efficiency of image processing, achieving efficient fusion of image data of different modalities collected from optoelectronic pods, thereby providing more accurate and comprehensive image information for night vision monitoring, border patrols, military and security fields.

[0013] In an optional implementation, the sample image is input into the initial multimodal image fusion model, deep feature extraction and feature reconstruction are performed on the sample image to obtain a fused image corresponding to the sample image, including:

[0014] Input the sample image into the downsampling module of the initial multimodal image fusion model to extract the deep features of each modality of the sample image;

[0015] The deep features of each modality are deeply fused to obtain fused features;

[0016] The fusion feature and the reconstructed feature obtained by the upsampling module of the previous initial multimodal image fusion model are input into the current upsampling module, and the current reconstructed feature is obtained by reconstructing until the multimodal feature weight is output;

[0017] Based on the multimodal feature weights and the sample image, a fused image corresponding to the sample image is calculated.

[0018] In this way, the multimodal image fusion model with cross-attention mechanism can effectively process and fuse image data of different modalities collected by the optoelectronic pod, such as infrared images, visible light images, images of different focal lengths and exposure levels, etc., which significantly improves the generalization and processing capabilities of the multimodal image fusion model for images of multiple modalities, helps to obtain more comprehensive visual information in complex environments, and can more accurately process and fuse image data of different modalities, providing richer and more detailed image information; the enhancement of real-time processing capabilities significantly improves the speed and efficiency of image processing, meeting the stringent requirements of these application scenarios for real-time image processing.

[0019] In an optional implementation, the image decomposition network includes an encoder, a decoder, and multi-layer skip connections, and the pre-trained image decomposition network is obtained by:

[0020] Obtain a sample fusion image corresponding to the sample image;

[0021] The sample fusion image is input into the encoder, and features are extracted from the sample fusion image to obtain multimodal features;

[0022] The multimodal features are transmitted to the decoder using multi-layer skip connections, and the decoder is used to reconstruct the features and decompose them to obtain the multimodal features;

[0023] The image decomposition network is trained using the loss function between the multimodal features and the sample fusion image to obtain a pre-trained image decomposition network.

[0024] In this method, by training the image decomposition network, the trained image decomposition network can accurately decompose the fused high-quality image into modal features that are close to the distribution of the original image information, which is used to evaluate and guide the fusion effect of the multimodal image fusion model. The image generated by the fusion network is decomposed by the decomposition network, and the difference between the decomposed image information and the original image is quantified through the loss function to guide the training process of the fusion network, so that the fused image after fusion by the multimodal image fusion model contains more multimodal information.

[0025] In an optional implementation, calculating the auxiliary loss between the reconstructed image and the sample image includes:

[0026] The auxiliary loss between the reconstructed image and the sample image is calculated by the following formula:

[0027]

[0028] Among them, l aux is the auxiliary loss, L1 is the reconstruction loss function, F fused To fuse the images, are the reconstructed images of each modality.

[0029] In this method, the fused image generated by the multimodal image fusion model is decomposed by using a decomposition network, and the difference between the decomposed image information and the original image is quantified through auxiliary loss, thereby guiding the training process of the fusion network, causing the fused image to contain more multimodal information, further improving the modal information content of the fused image.

[0030] In a second aspect, the present invention provides a multimodal image fusion method, the method comprising:

[0031] Obtain images of various modes collected by different sensors of the optoelectronic pod;

[0032] The images of each modality are input into a multimodal image fusion model to obtain a fused image corresponding to the images of each modality, wherein the multimodal image fusion model is constructed using the multimodal image fusion model construction method of any one of the first aspects.

[0033] In the present invention, by using the multimodal image fusion model constructed by the above steps, the image data of different modalities collected by the optoelectronic pod system are effectively fused to provide more comprehensive image information, which helps to improve the accuracy of target detection and the ability of environmental perception, and is of great significance for the optoelectronic pod system to perform night vision monitoring, border patrol, military and security fields. Multimodal image fusion realizes more accurate processing and fusion of image data of different modalities, and provides richer and more detailed image information; this is of great significance for improving the advanced visual tasks of optoelectronic pods, such as target detection and environmental perception in application scenarios such as night vision monitoring and border patrol. It meets the needs of optoelectronic pod systems in aviation, aerospace and high-end monitoring fields to realize real-time or near real-time image processing. The speed and efficiency of image processing are greatly improved, and efficient fusion of image data of different modalities collected by optoelectronic pods is realized, thereby providing more accurate and comprehensive image information for night vision monitoring, border patrol, military and security fields.

[0034] In a third aspect, the present invention provides a multimodal image fusion model construction device, the device comprising:

[0035] The first image acquisition module is used to acquire sample images of each mode collected by different sensors of the optoelectronic pod;

[0036] The feature extraction and fusion module is used to input the sample image into the initial multimodal image fusion model, perform deep feature extraction and feature reconstruction on the sample image, and obtain a fused image corresponding to the sample image;

[0037] A loss calculation module is used to calculate the gradient loss and pixel loss between the fused image and the sample image;

[0038] An auxiliary loss calculation module is used to decompose the fused image using a pre-trained image decomposition network to obtain a reconstructed image corresponding to the fused image, and calculate the auxiliary loss between the reconstructed image and the sample image;

[0039] A joint loss calculation module is used to calculate the joint loss based on gradient loss, pixel loss and auxiliary loss;

[0040] The model training module is used to train the initial multimodal image fusion model using the optimizer and the joint loss to obtain the target multimodal image fusion model.

[0041] In a fourth aspect, the present invention provides a multimodal image fusion device, the device comprising:

[0042] The second image acquisition module is used to acquire images of various modes collected by different sensors of the optoelectronic pod;

[0043] The image fusion module is used to input the images of each modality into the multimodal image fusion model to fuse the images of each modality to obtain a fused image corresponding to the images of each modality, wherein the multimodal image fusion model is constructed using the multimodal image fusion model construction device of the third aspect.

[0044] In a fifth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to thereby execute the multimodal image fusion model construction method of the first aspect or any corresponding embodiment thereof, or execute the multimodal image fusion method of the second aspect.

[0045] In a sixth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multimodal image fusion model construction method of the first aspect or any corresponding embodiment thereof, or to execute the multimodal image fusion method of the second aspect.

[0046] In a seventh aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the multimodal image fusion model construction method of the first aspect or any corresponding embodiment thereof, or to execute the multimodal image fusion method of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a flowchart of a method for constructing a multimodal image fusion model according to an embodiment of the present invention.

[0049] Figure 2 The figure is a flow chart of constructing a converged network according to an embodiment of the present invention.

[0050] Figure 3 It is a flowchart of another method for constructing a multimodal image fusion model according to an embodiment of the present invention.

[0051] Figure 4 It is a schematic diagram of a multimodal image fusion network structure based on Transformer cross attention according to an embodiment of the present invention.

[0052] Figure 5 It is a schematic diagram of a fusion module network structure based on Transformer cross attention according to an embodiment of the present invention.

[0053] Figure 6 4 is a schematic diagram of a U-net decomposition network structure according to an embodiment of the present invention.

[0054] Figure 7 It is a flowchart of a multimodal image fusion method according to an embodiment of the present invention.

[0055] Figure 8 It is a structural block diagram of a multimodal image fusion model construction device according to an embodiment of the present invention.

[0056] Fig. 9 4 is a structural block diagram of a multimodal image fusion device according to an embodiment of the present invention.

[0057] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0059] In the related technologies, although the multimodal image fusion technology in the field of optoelectronic pod technology has made certain progress, the existing technology still shows some limitations. The current image fusion methods mainly rely on a single deep learning network to solve specific fusion problems, such as infrared and visible light image fusion, multi-exposure image fusion or multi-focus image fusion. These methods are often optimized only for a single or a few cases, and are difficult to be widely used in a variety of different image fusion scenarios, thus limiting the versatility of the technology. In addition, optoelectronic pod systems often need to achieve real-time or near real-time image processing in actual operation, and the existing technology still needs to be improved in terms of processing speed and efficiency.

[0060] In order to solve the above problems, a multimodal image fusion model construction method is provided in an embodiment of the present invention, which is used in a computer device. It should be noted that its execution subject can be a multimodal image fusion model construction device, which can be implemented as part or all of the computer device through software, hardware, or a combination of software and hardware. The computer device can be a terminal or a client or a server. The server can be a single server or a server cluster composed of multiple servers. The terminal in the embodiment of the present application can be a smart phone, a personal computer, a tablet computer, or other intelligent hardware devices. In the following method embodiments, the execution subject is a computer device as an example for explanation.

[0061] The computer device in this embodiment is suitable for the use scenario of image data fusion of image information of different modalities captured by various types of sensors in the optoelectronic pod system. The present invention provides a method for constructing a multimodal image fusion model. By using the image fusion network of the decomposition network and the cross-attention mechanism, it can effectively process and fuse image data of different modalities collected from the optoelectronic pod, such as infrared images, visible light images, images of different focal lengths and exposure levels, etc., which significantly improves the generalization and processing capabilities of the constructed multimodal image fusion model for images of multiple modalities, and helps to obtain more comprehensive visual information in complex environments. Image data of different modalities can be processed and fused more accurately, providing richer and more detailed image information, significantly improving the effect of the fusion network and the fusion generalization capabilities of multiple modal images, and greatly improving the speed and efficiency of image processing, realizing efficient fusion of image data of different modalities collected from optoelectronic pods, thereby providing more accurate and comprehensive image information for night vision monitoring, border patrols, military and security fields.

[0062] According to an embodiment of the present invention, an embodiment of a method for constructing a multimodal image fusion model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0063] In this embodiment, a multimodal image fusion model construction method is provided, which can be used in the above-mentioned computer device. Figure 1 is a flowchart of a method for constructing a multimodal image fusion model according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0064] Step S101, obtaining sample images of each mode collected by different sensors of the optoelectronic pod.

[0065] In one example, for the training of the multimodal image fusion network, multimodal datasets including infrared visible light datasets (MSRS and LLVIP), multi-exposure image fusion datasets (EMPA HDR), and multi-focus image datasets (Lytro) were used, and batch normalization preprocessing was performed on the sample images of all datasets involved in the training. In order to adapt to different fusion tasks, the corresponding datasets were selected for special training.

[0066] Step S102: input the sample image into the initial multimodal image fusion model, perform deep feature extraction and feature reconstruction on the sample image, and obtain a fused image corresponding to the sample image.

[0067] In one example, the multimodal image fusion model is a multimodal image fusion network structure based on Transformer cross attention. The network includes feature extraction and feature reconstruction parts. After the sample image is input into the multimodal image fusion model, the fused fusion image F is output. fused .

[0068] Step S103, calculating and obtaining the gradient loss and pixel loss between the fused image and the sample image.

[0069] In one example, the gradient loss l grad It helps to maintain the edge and texture information of the fused image and is calculated by the following formula:

[0070]

[0071] Among them, L1(·) is the L1 norm loss function, F fused is the fused image, I i with I j is the multimodal image before fusion, To represent the Sobel operator, it forces the extracted features to contain more texture information with high fidelity.

[0072] Pixel loss pixiv By directly comparing the difference between the fused image and each source image at each pixel, the fused image is forced to be close to the source image at the pixel level, which is calculated by the following formula:

[0073]

[0074] Where N is the total number of pixels in the image, F fused (k) is the intensity value of the fused image at pixel k, I i (k) I j (k) are the intensity values ​​of two different source modality images at pixel k.

[0075] Step S104, using a pre-trained image decomposition network to decompose the fused image, obtain a reconstructed image corresponding to the fused image, and calculate the auxiliary loss between the reconstructed image and the sample image.

[0076] In one example, a decomposition network based on U-net is used to decompose the fused image generated by the fusion network, and the difference between the decomposed image information and the original image is calculated through the auxiliary loss l aux Quantify, and then guide the training process of the fusion network, so that the fusion image contains more multimodal information. It is expressed as:

[0077] The loss value of structural similarity (SSIM) is used as the loss function between the decomposed image and the original image of the fused image, and the L1 norm is used as the reconstruction loss function.

[0078]

[0079] Among them, F fused is the fused image, They are the reconstructed images of different modalities obtained after the U-net decomposition network decomposes the fused image.

[0080] Step S105, calculating the joint loss based on the gradient loss, pixel loss and auxiliary loss.

[0081] In one example, in order to ensure that the fused image can more comprehensively reflect the multimodal image information and improve the performance of the fusion network, the joint loss function l join It includes gradient loss, pixel loss and auxiliary loss. The joint loss function is calculated by the following formula:

[0082] l join =λ1l grad +λ2l pixiv +λ3l aux

[0083] Among them, l grad represents the gradient loss, l pixiv represents pixel loss, l aux Represents the U-net network decomposition loss, i.e., auxiliary loss. The combined use of these three losses can guide the training process of the fusion network to retain more details and information of multimodal images. λ1, λ2, and λ3 are weight coefficients used to adjust the weights of different loss items in the total loss. The weight setting is not limited in the present invention.

[0084] Step S106, using the optimizer and the joint loss to train the initial multimodal image fusion model to obtain the target multimodal image fusion model.

[0085] In one example, the images in the training dataset are preprocessed with batch normalization, and the AdamW optimizer and the joint loss function are used. join To train the Transformer cross-attention based multimodal image fusion network. During the training process, the hyperparameters are continuously optimized to ensure the best network performance. After the training is completed, the model parameters are saved to obtain the target multimodal image fusion model.

[0086] In one implementation scenario, in order to ensure that the image generated by the actual fusion network contains rich multimodal information, if the fused image can restore the original multimodal information, it indicates that the fusion network has high-quality multimodal feature fusion capabilities. Figure 2 is a schematic diagram of a flow chart of a fusion network construction according to an embodiment of the present invention, such as Figure 2 As shown in the figure, the process of building a fusion network can include: A. Constructing a U-net decomposition network, using the latest fusion algorithm to select fusion images with better fusion quality, training the U-net decomposition network, and after the training is completed, using the trained U-net decomposition network to guide the training process of the entire fusion network. B. Constructing a general Transformer-based cross-attention multimodal image fusion network. C. Training of multimodal fusion network: In order to adapt to different fusion tasks, select corresponding data sets for special training; using the corresponding data sets, use the AdamW optimizer and the joint loss function l join To train the multimodal image fusion network based on Transformer cross attention, during the training process, the hyperparameters are continuously optimized to ensure the best network performance. After the training is completed, the model parameters are saved. D. Fusion network test: In the test phase, the network structure used is exactly the same as the multimodal image fusion network structure based on Transformer cross attention described in step B. The key difference is that the U-net decomposition network is no longer used in the test phase, but the model parameters and weights trained and optimized in step C are directly used for image fusion test reasoning.

[0087] The multimodal image fusion model construction method provided in this embodiment can effectively process and fuse image data of different modalities collected from optoelectronic pods, such as infrared images, visible light images, images of different focal lengths and exposure levels, etc., by utilizing an image fusion network with a decomposition network and a cross-attention mechanism, significantly improving the generalization and processing capabilities of the constructed multimodal image fusion model for images of multiple modalities, and helping to obtain more comprehensive visual information in complex environments. Image data of different modalities can be processed and fused more accurately, providing richer and more detailed image information, significantly improving the effect of the fusion network and the fusion generalization capabilities of images of multiple modalities, and greatly improving the speed and efficiency of image processing, achieving efficient fusion of image data of different modalities collected from optoelectronic pods, thereby providing more accurate and comprehensive image information for night vision monitoring, border patrols, military and security fields.

[0088] In this embodiment, a multimodal image fusion model construction method is provided, which can be used in the above-mentioned computer device. Figure 3 is a flowchart of another method for constructing a multimodal image fusion model according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0089] Step S301, obtain sample images of each mode collected by different sensors of the optoelectronic pod. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0090] Step S302: input the sample image into the initial multimodal image fusion model, perform deep feature extraction and feature reconstruction on the sample image, and obtain a fused image corresponding to the sample image.

[0091] Specifically, the above step S302 includes:

[0092] Step S3021: input the sample image into the downsampling module of the initial multimodal image fusion model to extract the depth features of each modality of the sample image.

[0093] Step S3022, deep-level feature fusion is performed on the deep features of each modality to obtain fused features.

[0094] Step S3023, input the fusion features and the reconstructed features obtained by the upsampling module of the previous initial multimodal image fusion model into the current upsampling module, reconstruct and obtain the current reconstructed features until the multimodal feature weights are output.

[0095] Step S3024: Based on the multimodal feature weights and the sample image, a fused image corresponding to the sample image is calculated.

[0096] In one example, Figure 4is a schematic diagram of a multimodal image fusion network structure based on Transformer cross attention according to an embodiment of the present invention, such as Figure 4 As shown, the multimodal image fusion model network includes feature extraction and feature reconstruction parts.

[0097] Specifically, the feature extraction process may include: the input multimodal image passes through four groups of downsampling modules, each group contains a 3×3 convolution layer and a LedkyRelu activation layer, which is used to extract the deep features of the source image at different scales. Taking visible light-infrared image fusion as an example, in the first three downsampling processes, the infrared and visible light feature maps obtained by each downsampling will pass through a Transformer-based fusion module (FB) for feature fusion. Figure 5 is a schematic diagram of a fusion module network structure based on Transformer cross attention according to an embodiment of the present invention. The specific structure of the FB module is as follows Figure 5 As shown in the figure, it uses the cross-attention mechanism of Transformer to realize the interactive fusion of infrared and visible light features, and then shares a multi-layer perceptron (MLP) for deep feature fusion, which not only reduces the number of parameters, but also can mix the features of different modalities again. The fused features output by the FB module will be skipped with the corresponding feature reconstruction layer to transmit multi-scale fusion information. After the fourth downsampling, the features of the two modalities are spliced ​​in the channel dimension, and then a 1×1 convolution layer is used to adjust the number of channels of the spliced ​​feature map to obtain the deepest fusion feature map.

[0098] The feature reconstruction process may include: the feature reconstruction part includes three groups of upsampling modules, each group includes a 3×3 deconvolution layer and a LeakyReLU activation layer. The input of each upsampling module is the reconstructed features of the previous layer and the FB fusion features of the corresponding scale. The output is used as the input of the next upsampling module to supplement the multi-scale detail information and reconstruct the fused features. After passing through three groups of upsampling modules, it passes through a layer of 3×3 convolution and Tanh activation function to output a set of multimodal feature weights W1 and W2. Finally, this set of weights is intelligently multiplied with the corresponding elements of the original image, and then the corresponding elements are added and averaged to obtain the fused image F fused .

[0099] In this way, the multimodal image fusion model with cross-attention mechanism can effectively process and fuse image data of different modalities collected by the optoelectronic pod, such as infrared images, visible light images, images of different focal lengths and exposure levels, etc., which significantly improves the generalization and processing capabilities of the multimodal image fusion model for images of multiple modalities, helps to obtain more comprehensive visual information in complex environments, and can more accurately process and fuse image data of different modalities, providing richer and more detailed image information; the enhancement of real-time processing capabilities significantly improves the speed and efficiency of image processing, meeting the stringent requirements of these application scenarios for real-time image processing.

[0100] Step S303, calculate the gradient loss and pixel loss between the fused image and the sample image. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0101] Step S304, using a pre-trained image decomposition network to decompose the fused image, obtain a reconstructed image corresponding to the fused image, and calculate the auxiliary loss between the reconstructed image and the sample image.

[0102] Specifically, the image decomposition network includes an encoder, a decoder, and multi-layer skip connections. The pre-trained image decomposition network is obtained by:

[0103] Step a1, obtaining a sample fusion image corresponding to the sample image.

[0104] Step a2: input the sample fusion image into the encoder, perform feature extraction on the sample fusion image, and obtain multimodal features.

[0105] In step a3, the multimodal features are transmitted to the decoder using multi-layer skip connections, and the decoder is used to reconstruct the features and decompose them to obtain the multimodal features.

[0106] Step a4, using the loss function between the multimodal features and the sample fusion image, the image decomposition network is trained to obtain a pre-trained image decomposition network.

[0107] In an optional implementation, calculating the auxiliary loss between the reconstructed image and the sample image includes: calculating the auxiliary loss between the reconstructed image and the sample image by the following formula:

[0108]

[0109] Among them, l aux is the auxiliary loss, L1 is the reconstruction loss function, F fused To fuse the images, are the reconstructed images of each modality.

[0110] In one example, in order to ensure that the image generated by the actual fusion network contains rich multimodal information, if the fused image can restore the original multimodal information, it means that the fusion network has high-quality multimodal feature fusion capabilities. An image decomposition network based on the U-net architecture is used to accurately decompose the fused high-quality image into modal features that are close to the distribution of the original image information, which is used to evaluate and guide the fusion effect of the above multimodal image fusion model.

[0111] In the training phase of the multimodal image fusion model, the image generated by the fusion network is decomposed through the decomposition network based on U-net, and the difference between the decomposed image information and the original image is quantified through the loss function, thereby guiding the training process of the fusion network and making the fusion image contain more multimodal information. The design process of the U-net decomposition network is as follows:

[0112] (1) Figure 6 is a schematic diagram of a U-net decomposition network structure according to an embodiment of the present invention, such as Figure 6 As shown, the U-net decomposition network consists of an encoder, a decoder, and multiple layers of skip connections:

[0113] (1.1) Encoder part: The input fused image first enters the encoder part, which includes five layers of downsampling modules. Each module consists of a 3×3 convolution layer and a LeakyReLU activation layer. The stride of the convolution layer is 1 and the mapping padding is 1. The input fused image is effectively compressed and deep features are extracted through five layers of convolution downsampling.

[0114] (1.2) Skip connection: The output of each downsampling module is not only connected to the input of the next layer, but also directly transmitted to the corresponding layer of the decoder through the skip connection. The skip connection of multi-scale features helps the decoder to supplement the detailed information of multi-scale features during the image reconstruction process and improves the feature reconstruction capability of the decoder.

[0115] (1.3) Decoder structure: The decoder part includes five feature reconstruction modules, each of which consists of a deconvolution layer with a convolution kernel of 3 and a LeakyReLU activation layer. The step size of the convolution layer is 1 and the mapping padding is 1. After five layers of feature reconstruction, the original size and number of channels of the image can be gradually restored. In each feature reconstruction step, the feature map from the encoder jump connection is added to the corresponding element of the output of the current layer, and then used as the input of the next feature reconstruction module, so that the subsequent feature reconstruction process can better decompose the image features of different modalities.

[0116] (1.4) Output: After being processed by the last feature reconstruction module, the network outputs two modal features with the same size as the source multimodal image. By training the decomposition network and minimizing the loss function value, the information distribution of these two modal features can be made close to the original sample image, so that the decomposition network can restore the fused image and obtain the source multimodal information.

[0117] (2) For example, taking visible light-infrared fusion image decomposition as an example, 3000 pairs of infrared visible light images are collected from the infrared visible light datasets MSRS and LLVIP, and these images are standardized. In order to obtain high-quality fused images, two currently advanced image fusion technologies, MUFusion and SeAFusion, are used to input the preprocessed sample images into these two fusion networks for image fusion, and 600 high-quality sample fusion images are selected. These sample fusion images not only effectively retain the thermal imaging features in the infrared images, but also fuse the rich details and color information in the visible light images, realizing the efficient fusion of image multimodal information. The 600 high-quality sample fusion images are divided into a training set, a validation set, and a data set at a ratio of 70%, 20%, and 10%, providing rich sample resources for the training and testing of the decomposition network. The number and proportion of sample fusion images are not limited in the present invention.

[0118] (3) Adam is used as the optimizer for decomposition network training, and the decomposition network is trained using the data set in (2). The multimodal features obtained after the sample fusion image is input into the decomposition network are compared with the source image (i.e., the infrared and visible light image before fusion) and the L1 loss is performed to train the decomposition network so that it can decompose the fusion image and obtain modal features with similar information distribution to the original image. The constructed U-net decomposition network will be used to evaluate and guide the training of the above-mentioned Transformer-based cross-attention multimodal image fusion network.

[0119] In this method, by training the image decomposition network, the trained image decomposition network can accurately decompose the fused high-quality image into modal features that are close to the distribution of the original image information, which is used to evaluate and guide the fusion effect of the multimodal image fusion model. The image generated by the fusion network is decomposed by the decomposition network, and the difference between the decomposed image information and the original image is quantified through the loss function, which guides the training process of the fusion network, and promotes the fusion image after the fusion of the multimodal image fusion model to contain more multimodal information. By using the decomposition network to decompose the fused image generated by the multimodal image fusion model, and the difference between the decomposed image information and the original image is quantified through the auxiliary loss, which guides the training process of the fusion network, the fused image is promoted to contain more multimodal information, and the modal information amount of the fused image is further improved.

[0120] Step S305: Based on the gradient loss, pixel loss and auxiliary loss, the joint loss is calculated. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0121] Step S306: Use the optimizer and joint loss to train the initial multimodal image fusion model to obtain the target multimodal image fusion model. Figure 1 Step S106 of the illustrated embodiment will not be described in detail here.

[0122] The multimodal image fusion model construction method provided in this embodiment can effectively process and fuse image data of different modalities collected from the optoelectronic pod, such as infrared images, visible light images, images of different focal lengths and exposure levels, etc., through the multimodal image fusion model of the cross-attention mechanism, significantly improving the generalization and processing capabilities of the multimodal image fusion model for multiple modal images, which is helpful to obtain more comprehensive visual information in complex environments, and can more accurately process and fuse image data of different modalities, providing richer and more detailed image information; the enhancement of real-time processing capabilities significantly improves the speed and efficiency of image processing, and meets the stringent requirements of these application scenarios for real-time image processing. By training the image decomposition network, the trained image decomposition network can accurately decompose the fused high-quality image into modal features similar to the distribution of the original image information, which is used to evaluate and guide the fusion effect of the multimodal image fusion model. The image generated by the fusion network is decomposed and processed by the decomposition network, and the difference between the decomposed image information and the original image is quantified through the loss function, guiding the training process of the fusion network, and prompting the fused image after fusion of the multimodal image fusion model to contain more multimodal information. The fused image generated by the multimodal image fusion model is decomposed by using a decomposition network, and the difference between the decomposed image information and the original image is quantified through auxiliary loss, thereby guiding the training process of the fusion network, prompting the fused image to contain more multimodal information, and further improving the modal information content of the fused image.

[0123] In this embodiment, a multimodal image fusion method is provided, which can be used in the above-mentioned computer device. Figure 7 is a flow chart of a multimodal image fusion method according to an embodiment of the present invention. Figure 7 As shown, the process includes the following steps:

[0124] Step S701, acquiring images of various modes collected by different sensors of the optoelectronic pod.

[0125] Step S702: input the images of each modality into a multimodal image fusion model to obtain a fused image corresponding to the images of each modality.

[0126] In the embodiment of the present invention, the multimodal image fusion model is constructed by using the above-mentioned multimodal image fusion model construction method.

[0127] The multimodal image fusion method provided in this embodiment, by utilizing the multimodal image fusion model constructed by the above steps, realizes the effective fusion of image data of different modalities collected by the optoelectronic pod system, provides more comprehensive image information, helps to improve the accuracy of target detection and the ability of environmental perception, and is of great significance for the optoelectronic pod system to perform night vision monitoring, border patrol, military and security fields.

[0128] In this embodiment, a multimodal image fusion model construction device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0129] This embodiment provides a multimodal image fusion model construction device, such as Figure 8 As shown, including:

[0130] The first image acquisition module 801 is used to acquire sample images of each mode collected by different sensors of the optoelectronic pod. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0131] The feature extraction and fusion module 802 is used to input the sample image into the initial multimodal image fusion model, perform deep feature extraction and feature reconstruction on the sample image, and obtain a fused image corresponding to the sample image. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0132] The loss calculation module 803 is used to calculate the gradient loss and pixel loss between the fused image and the sample image. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0133] The auxiliary loss calculation module 804 is used to decompose the fused image using the pre-trained image decomposition network to obtain the corresponding reconstructed image of the fused image and calculate the auxiliary loss between the reconstructed image and the sample image. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0134] The joint loss calculation module 805 is used to calculate the joint loss based on the gradient loss, pixel loss and auxiliary loss. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0135] The model training module 806 is used to train the initial multimodal image fusion model using the optimizer and the joint loss to obtain the target multimodal image fusion model. Figure 1 Step S106 of the illustrated embodiment will not be described in detail here.

[0136] In some optional implementations, the feature extraction and fusion module 802 includes:

[0137] The image downsampling unit is used to input the sample image into the downsampling module of the initial multimodal image fusion model to extract the deep features of each modality of the sample image.

[0138] The feature fusion unit is used to perform deep-level feature fusion of the deep features of each modality to obtain fused features.

[0139] The feature reconstruction unit is used to input the fusion feature and the reconstructed feature obtained by the upsampling module of the previous initial multimodal image fusion model into the current upsampling module, reconstruct the current reconstructed feature, and output the multimodal feature weight.

[0140] The fused image calculation unit is used to calculate a fused image corresponding to the sample image based on the multimodal feature weights and the sample image.

[0141] In some optional embodiments, the image decomposition network includes an encoder, a decoder, and multi-layer skip connections, and the multimodal image fusion model construction device includes:

[0142] The sample fusion image acquisition unit is used to acquire the sample fusion image corresponding to the sample image.

[0143] The feature extraction unit is used to input the sample fusion image into the encoder, perform feature extraction on the sample fusion image, and obtain multi-modal features.

[0144] The multimodal feature decomposition unit is used to transmit the multimodal features to the decoder using multi-layer skip connections, and use the decoder to reconstruct the features and decompose them to obtain the multimodal features.

[0145] The image decomposition network training unit is used to train the image decomposition network by using the loss function between the multimodal features and the sample fusion image to obtain a pre-trained image decomposition network.

[0146] In some optional implementations, the auxiliary loss calculation module 804 includes:

[0147] The auxiliary loss calculation unit is used to calculate the auxiliary loss between the reconstructed image and the sample image using the following formula:

[0148]

[0149] Among them, l aux is the auxiliary loss, L1 is the reconstruction loss function, F fused To fuse the images, are the reconstructed images of each modality.

[0150] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0151] The multimodal image fusion model construction device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0152] In this embodiment, a multimodal image fusion device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0153] This embodiment provides a multi-modal image fusion device, such as Fig. 9 As shown, including:

[0154] The second image acquisition module 901 is used to acquire images of various modes collected by different sensors of the optoelectronic pod. Figure 7 Step S701 of the illustrated embodiment will not be described in detail here.

[0155] The image fusion module 902 is used to input the images of each modality into the multimodal image fusion model to obtain the fused image corresponding to the images of each modality, wherein the multimodal image fusion model is constructed by using the multimodal image fusion model construction device. Figure 7 Step S702 of the illustrated embodiment will not be described in detail here.

[0156] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0157] The multimodal image fusion device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0158] The embodiment of the present invention also provides a computer device having the above Figure 8 The multimodal image fusion model construction device shown and Fig. 9 The multimodal image fusion device shown.

[0159] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0160] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0161] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0162] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0163] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0164] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0165] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0166] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0167] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0168] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for constructing a multimodal image fusion model, characterized in that: The method comprises: Obtain sample images of each mode collected by different sensors of the optoelectronic pod; Inputting the sample image into an initial multimodal image fusion model, performing deep feature extraction and feature reconstruction on the sample image, and obtaining a fused image corresponding to the sample image; Calculating the gradient loss and pixel loss between the fused image and the sample image; Decomposing the fused image by using a pre-trained image decomposition network to obtain a reconstructed image corresponding to the fused image, and calculating the auxiliary loss between the reconstructed image and the sample image; the image decomposition network includes an encoder, a decoder and a multi-layer skip connection, and the pre-trained image decomposition network is obtained by: obtaining a sample fused image corresponding to the sample image; inputting the sample fused image into the encoder, extracting features from the sample fused image, and obtaining multi-modal features; transmitting the multi-modal features to the decoder by using the multi-layer skip connection, reconstructing features by using the decoder, and decomposing to obtain multi-modal features; training the image decomposition network by using the loss function between the multi-modal features and the sample fused image, and obtaining the pre-trained image decomposition network; Calculate a joint loss based on the gradient loss, the pixel loss and the auxiliary loss; The initial multimodal image fusion model is trained using an optimizer and the joint loss to obtain a target multimodal image fusion model.

2. The method according to claim 1, characterized in that The step of inputting the sample image into an initial multimodal image fusion model, performing deep feature extraction and feature reconstruction on the sample image, and obtaining a fused image corresponding to the sample image includes: Inputting the sample image into the downsampling module of the initial multimodal image fusion model to extract the depth features of each modality of the sample image; Performing deep-level feature fusion on the deep features of each modality to obtain fused features; Input the fusion feature and the reconstruction feature obtained by the upsampling module of the previous initial multimodal image fusion model into the current upsampling module, reconstruct the current reconstruction feature, and output the multimodal feature weight; Based on the multimodal feature weights and the sample image, a fused image corresponding to the sample image is calculated.

3. The method according to claim 1, characterized in that The calculating the auxiliary loss between the reconstructed image and the sample image comprises: The auxiliary loss between the reconstructed image and the sample image is calculated by the following formula: Among them, l aux is the auxiliary loss, L1 is the reconstruction loss function, F fused is the fused image, are the reconstructed images of each modality.

4. A multimodal image fusion method, characterized in that: The method comprises: Obtain images of various modes collected by different sensors of the optoelectronic pod; The images of each modality are input into a multimodal image fusion model to obtain fused images corresponding to the images of each modality, wherein the multimodal image fusion model is constructed using the multimodal image fusion model construction method described in any one of claims 1 to 3.

5. A multimodal image fusion model construction device, characterized in that: The device comprises: The first image acquisition module is used to acquire sample images of each mode collected by different sensors of the optoelectronic pod; A feature extraction and fusion module, used to input the sample image into an initial multimodal image fusion model, perform deep feature extraction and feature reconstruction on the sample image, and obtain a fused image corresponding to the sample image; A loss calculation module, used for calculating the gradient loss and pixel loss between the fused image and the sample image; an auxiliary loss calculation module, for decomposing the fused image by using a pre-trained image decomposition network, obtaining a reconstructed image corresponding to the fused image, and calculating an auxiliary loss between the reconstructed image and the sample image; the image decomposition network comprises an encoder, a decoder and a multi-layer skip connection, and the pre-trained image decomposition network is obtained by: obtaining a sample fused image corresponding to the sample image; inputting the sample fused image into the encoder, performing feature extraction on the sample fused image, and obtaining a multi-modal feature; transmitting the multi-modal feature to the decoder by using the multi-layer skip connection, performing feature reconstruction by using the decoder, and decomposing to obtain a multi-modal feature; and training the image decomposition network by using a loss function between the multi-modal feature and the sample fused image, and obtaining the pre-trained image decomposition network; A joint loss calculation module, used for calculating a joint loss based on the gradient loss, the pixel loss and the auxiliary loss; The model training module is used to train the initial multimodal image fusion model using the optimizer and the joint loss to obtain a target multimodal image fusion model.

6. A multimodal image fusion device, characterized in that: The device comprises: The second image acquisition module is used to acquire images of various modes collected by different sensors of the optoelectronic pod; An image fusion module is used to input the images of each modality into a multimodal image fusion model to fuse the images of each modality to obtain a fused image corresponding to the images of each modality, wherein the multimodal image fusion model is constructed using the multimodal image fusion model construction device described in claim 5.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multimodal image fusion model construction method described in any one of claims 1 to 3 or the multimodal image fusion method described in claim 4 by executing the computer instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multimodal image fusion model construction method described in any one of claims 1 to 3 or the multimodal image fusion method described in claim 4.

9. A computer program product, characterized in that It includes computer instructions, and the computer instructions are used to enable a computer to execute the multimodal image fusion model construction method described in any one of claims 1 to 3 or the multimodal image fusion method described in claim 4.

Images