Multi-modal image fusion method and device based on extended residual attention network

Abstract

The invention provides a multi-modal image fusion method and equipment based on an expanded residual attention network. The method comprises the following steps: carrying out preprocessing of Laplacian enhancement and brightness enhancement on an original image, and extracting a deep feature map from the preprocessed image; fusing the deep feature maps into a fused feature map by adopting a Softmax feature weighting strategy; and determining a loss function for the fusion feature map, generating a fused image, and reducing the number of channels from 64 to 1 by adopting three continuous 3 * 3 convolution layers with output channels. According to the method, the features in the images in different modalities are fused into the same image as much as possible, so that the situation that a clinician has to repeatedly switch bone and joint disc anatomical structures in two modalities when performing related diagnosis on the temporomandibular joint area is avoided, and the fused image has the effects of simplifying the diagnosis process and improving the diagnosis efficiency. And the portability of one picture with two purposes is provided.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of medical image processing technology, and in particular to a multimodal image fusion method and device based on a dilated residual attention network. Background Art

[0002] The progress of medical diagnosis and treatment is inseparable from the assistance of medical imaging. Medical images of different modalities have corresponding diagnosis and treatment purposes and corresponding conditions. When conducting a comprehensive analysis of the special soft and hard tissue structures in a certain area, it is necessary to switch repeatedly between multiple modal images, which is cumbersome and lacks the consistency and comparability of spatial positioning between various image data. The temporomandibular joint is a common disease area in the oral and maxillofacial region. The course of the disease is mostly progressive, and early detection helps the recovery of the disease. Oral medical imaging has different modalities, so it also has its own limitations and advantages. For example, the images of related technologies can clearly see the distribution of soft tissues in the oral cavity, but cannot clearly observe the jaw bone information; while the images of other related technologies are insufficient in soft tissue observation. Combined with the actual needs of oral clinical practice, in order to give full play to the unique advantages of various modal image assistance, avoid clinicians repeatedly switching images, and increase the spatial positioning consistency of each modality data, the development of a multimodal image fusion method and equipment based on an expanded residual attention network can effectively overcome the defects in the above-mentioned related technologies, which has become a technical problem that needs to be solved urgently in the industry. Summary of the invention

[0003] In view of the above problems existing in the prior art, an embodiment of the present invention provides a multimodal image fusion method and device based on a dilated residual attention network.

[0004] In the first aspect, an embodiment of the present invention provides a multimodal image fusion method based on a dilated residual attention network, comprising: preprocessing the original image with Laplacian enhancement and brightness enhancement, extracting a deep feature map from the preprocessed image; fusing the deep feature map into a fused feature map using a Softmax feature weighting strategy; determining a loss function for the fused feature map, generating a fused image, and using three consecutive 3*3 convolutional layers with output channels to reduce the number of channels from 64 to 1.

[0005] Based on the content of the above method embodiments, a multimodal image fusion method based on an expanded residual attention network is provided in an embodiment of the present invention, wherein a deep feature map is extracted from a preprocessed image, comprising: the preprocessed image is first input into a residual attention network with a 1*1 convolutional layer to obtain a 64-dimensional shallow feature image, and then the shallow feature image is passed to the expanded residual attention network to obtain a deep feature map.

[0006] Based on the content of the above method embodiments, the multimodal image fusion method based on the dilated residual attention network provided in the embodiments of the present invention adopts the Softmax feature weighting strategy to fuse the deep feature map into a fused feature map, including: obtaining two output feature maps from the input deep feature map respectively, performing Softmax operation on the two output feature maps, and after obtaining the Softmax output, calculating the sum of the singular values ​​through the nuclear norm of the matrix, taking the maximum weighted average of the nuclear norm to obtain the weight of the fused feature map, and obtaining the fused feature map according to the weight.

[0007] On the basis of the content of the above method embodiment, a multimodal image fusion method based on an expanded residual attention network is provided in an embodiment of the present invention, and the expanded residual attention network includes: a first convolution module, which is connected in parallel with the second convolution module and the third convolution module; a fourth convolution module, which is connected to the first convolution module, the second convolution module, the third convolution module, the pooling module, the fifth convolution module, the sixth convolution module and the seventh convolution module; an upsampling module, which is connected to the pooling module and the ReLu activation function module; an eighth convolution module, which is connected to the fifth convolution module, the sixth convolution module and the tenth convolution module; a ninth convolution module, which is connected to the seventh convolution module and the tenth convolution module; and a ReLu activation function module, which is connected to the fourth convolution module, the fifth convolution module, the eighth convolution module and the tenth convolution module.

[0008] Based on the content of the above method embodiment, the multimodal image fusion method based on the dilated residual attention network provided in the embodiment of the present invention, taking the maximum weighted average of the nuclear norm to obtain the weight of the fusion feature map, includes: ; in, is the jth output feature map The weight map obtained after the Softmax function operation is is the max() function, is the output feature map The weight of is the output feature map The weight of is the matrix Frobenius norm.

[0009] Based on the content of the above method embodiment, the multimodal image fusion method based on the dilated residual attention network provided in the embodiment of the present invention, wherein the fusion feature map is obtained according to the weight, comprises: ; Among them, f is the fused feature map.

[0010] Based on the content of the above method embodiment, the multimodal image fusion method based on the dilated residual attention network provided in the embodiment of the present invention, the loss function is determined for the fusion feature map, including: ; in, is the loss function, M is the number of input images, is the output image, is the kth channel from the pth layer in the VGG16 network, is the kth channel from the qth layer in the VGG16 network, is the matrix 2 norm, MSE is the mean square error of the regression loss function, is the gradient loss, and is the length and width of the input p-layer image, C is the number of channels, Per is the perceptual loss, h is the width of the image, is the length of the image, is the gradient loss of the output image, is the gradient loss of the q-layer image, is the qth layer image, is the weight set of the target optimization network, and It is the weight balancing factor between gradient loss and perceptual loss.

[0011] In the second aspect, an embodiment of the present invention provides a multimodal image fusion device based on an expanded residual attention network, including: a first main module, used to implement preprocessing of the original image by Laplacian enhancement and brightness enhancement, and extract a deep feature map from the preprocessed image; a second main module, used to implement the fusion of the deep feature map into a fused feature map using a Softmax feature weighting strategy; a third main module, used to determine a loss function for the fused feature map, generate a fused image, and use three consecutive 3*3 convolutional layers with output channels to reduce the number of channels from 64 to 1.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, including: At least one processor, at least one memory and a communication interface; wherein, The processor, memory and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the multimodal image fusion method based on the dilated residual attention network provided by any one of the various implementation methods of the first aspect.

[0013] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable a computer to execute a multimodal image fusion method based on a dilated residual attention network provided in any one of the various implementation methods of the first aspect.

[0014] The multimodal image fusion method and device based on the dilated residual attention network provided in the embodiment of the present invention fuses as many features as possible from images of different modalities into the same image, thereby avoiding the need for clinicians to repeatedly switch between two modalities for the anatomical structures of the bones and articular disc when making diagnoses related to the temporomandibular joint area. This fused image simplifies the diagnostic process and provides the portability of a dual-purpose image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of a multimodal image fusion method based on an extended residual attention network provided in an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a multimodal image fusion device based on a dilated residual attention network provided in an embodiment of the present invention; Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention; Figure 4 A schematic diagram of the dilated residual attention network structure provided by an embodiment of the present invention; FIG5 (a) is a schematic diagram of a CBCT imaging effect provided by an embodiment of the present invention; FIG5( b ) is a schematic diagram of an MR imaging effect provided by an embodiment of the present invention; FIG5( c ) is a schematic diagram of the CBCT-MR fusion imaging effect provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical scheme and advantages of the embodiment of the present invention clearer, the technical scheme in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. This combination is not subject to the constraints of the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that the combination of such technical solutions does not exist and is not within the scope of protection required by the present invention. If there are step numbers in the following embodiments, they are only set for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0018] The embodiment of the present invention provides a multimodal image fusion method based on dilated residual attention network, see Figure 1 The method includes: preprocessing the original image by Laplacian enhancement and brightness enhancement, extracting the deep feature map from the preprocessed image; fusing the deep feature map into a fused feature map by using the Softmax feature weighting strategy; determining the loss function for the fused feature map, generating a fused image, and reducing the number of channels from 64 to 1 by using three consecutive 3*3 convolutional layers with output channels.

[0019] Specifically, the image preprocessing module is divided into two parts: Laplacian image enhancement and brightness enhancement of the source image. Laplacian image enhancement is used to sharpen the original image to enhance the grayscale contrast, thereby making the blurred image clearer; brightness enhancement aims to improve the brightness of the original image. Next, the preprocessed CBCT (cone beam CT) and MR (magnetic resonance) images are used as the input of the feature extraction module. The feature extraction module is used to extract the deep semantic features of the input image, and then use it as the input of the feature fusion module. The fusion rule uses the "softmax feature weighting strategy". The feature fusion module aims to fuse two deep semantic feature maps into one feature map, which contains features from two original feature maps. In addition, this module can also determine whether a pixel in the final fused feature map comes from a single I1, I2, or a common part of the two. Finally, the image reconstruction module reconstructs the image based on the output of the feature fusion module, and uses a sequence of convolutional layers to reconstruct the final fused image. Before training the model, all images are normalized in the range of 0~1 [5]. In the training phase, the fusion module is removed and only the feature extraction module and the image reconstruction module are trained. The specific loss function construction method is described later. In terms of the choice of optimizer, this experiment uses the Adam update rule, the learning rate is set to 0.0001, the batch size is set to 4, and the epoch is set to 100. In order to prevent overfitting, this experiment uses the following two methods: one is to monitor the performance of the validation data set during training and stop training in advance when the performance no longer improves; the other is to add L1 regularization to control the complexity of the network and prevent overfitting. The weight balance factors λ1 and λ2 of the image gradient loss and the perceptual loss are set to 0.2, and the output of the VGG block is used to calculate the perceptual loss.

[0020] Based on the content of the above method embodiment, as an optional embodiment, the multimodal image fusion method based on the dilated residual attention network provided in the embodiment of the present invention, wherein the deep feature map is extracted from the preprocessed image, comprising: the preprocessed image is first input into a residual attention network with a 1*1 convolutional layer to obtain a 64-dimensional shallow feature image, and then the shallow feature image is passed to the dilated residual attention network to obtain a deep feature map.

[0021] Based on the content of the above method embodiment, as an optional embodiment, the multimodal image fusion method based on the dilated residual attention network provided in the embodiment of the present invention adopts the Softmax feature weighting strategy to fuse the deep feature map into a fused feature map, including: obtaining two output feature maps from the input deep feature map respectively, performing Softmax operation on the two output feature maps, and after obtaining the Softmax output, calculating the sum of the singular values ​​through the nuclear norm of the matrix, taking the maximum weighted average of the nuclear norm to obtain the weight of the fused feature map, and obtaining the fused feature map according to the weight.

[0022] Specifically, the fusion strategy is used to fuse the deep feature maps of the input image into one feature map. After comparing multiple fusion strategies, the "Softmax feature weighting strategy" was used, which showed the best performance. After the feature extraction module, two output feature maps f1 and f2 are obtained from the input images (I1, I2) respectively. These two output feature maps can be used to generate the corresponding weight maps. In order to obtain the weight map, the output feature maps f1 and f2 are subjected to Softmax operation.

[0023] See also Figure 4 Based on the content of the above method embodiment, as an optional embodiment, the multimodal image fusion method based on the dilated residual attention network provided in the embodiment of the present invention, the dilated residual attention network includes: a first convolution module, which is connected in parallel with the second convolution module and the third convolution module; a fourth convolution module, which is connected to the first convolution module, the second convolution module, the third convolution module, the pooling module, the fifth convolution module, the sixth convolution module and the seventh convolution module; an upsampling module, which is connected to the pooling module and the ReLu activation function module; an eighth convolution module, which is connected to the fifth convolution module, the sixth convolution module and the tenth convolution module; a ninth convolution module, which is connected to the seventh convolution module and the tenth convolution module; and a ReLu activation function module, which is connected to the fourth convolution module, the fifth convolution module, the eighth convolution module and the tenth convolution module.

[0024] Based on the content of the above method embodiment, as an optional embodiment, the multimodal image fusion method based on the dilated residual attention network provided in the embodiment of the present invention, taking the maximum weighted average of the nuclear norm to obtain the weight of the fusion feature map, includes: ; in, is the jth output feature map The weight map obtained after the Softmax function operation is is the max() function, is the output feature map The weight of is the output feature map The weight of is the matrix Frobenius norm.

[0025] Based on the content of the above method embodiment, as an optional embodiment, the multimodal image fusion method based on the dilated residual attention network provided in the embodiment of the present invention, wherein the fusion feature map is obtained according to the weight, comprises: ; Among them, f is the fused feature map.

[0026] Based on the content of the above method embodiment, as an optional embodiment, the multimodal image fusion method based on the dilated residual attention network provided in the embodiment of the present invention, the loss function is determined for the fusion feature map, including: ; in, is the loss function, M is the number of input images, is the output image, is the kth channel from the pth layer in the VGG16 network, is the kth channel from the qth layer in the VGG16 network, is the matrix 2 norm, MSE is the mean square error of the regression loss function, is the gradient loss, and is the length and width of the input p-layer image, C is the number of channels, Per is the perceptual loss, h is the width of the image, is the length of the image, is the gradient loss of the output image, is the gradient loss of the q-layer image, is the qth layer image, is the weight set of the target optimization network, and It is the weight balancing factor between gradient loss and perceptual loss.

[0027] Specifically, in order to make the final fused image as close as possible to the original input image, it is necessary to minimize the distance between the image reconstructed from the deep semantic features and the original input image. The gap between the fused image and the input image is measured by the common regression loss function mean squared error (MSE). In addition, gradient loss is added to model the details of the texture in the reconstructed image, and perceptual loss is added to model the high-level semantic similarity between the reconstructed image and the original input image. MSE, gradient loss, and perceptual loss are shown in the above three equations. The original input image is the patient's left closed-mouth CBCT image as shown in Figure 5 (a) and the MR image as shown in Figure 5 (b). In the CBCT image, the morphology of the patient's condylar cortical bone can be observed, which is information lacking in the MR image; on the other hand, the morphology of the patient's condylar attached muscle, condylar medullary bone, and articular disc can be observed through the MR image, which is a unique feature of the MR image relative to the CBCT image. The fused image in Figure 5 (c) clearly shows the shape of the condylar cortical bone in the CBCT image; the muscles attached to the condylar process shown in the MR image can be observed through the fused image, and the muscle texture can be clearly reflected; the condylar medullary bone in the MR image is clearly presented in the fused image as a black line; the articular disc located above the condyle and between the temporomandibular joint can be seen in the MR image, but not in the CBCT image. However, the shape of the articular disc is clearly reflected in the final CBCT-MR fusion image, and is more obvious than in the MR image.

[0028] The multimodal image fusion method based on the dilated residual attention network provided in the embodiment of the present invention fuses as many features as possible from images of different modalities into the same image, thereby avoiding the need for clinicians to repeatedly switch between two modalities for the anatomical structures of the bones and articular disc when making diagnoses related to the temporomandibular joint area. This fused image simplifies the diagnostic process and provides the portability of one image for two uses.

[0029] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention can be encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a multimodal image fusion device based on an expanded residual attention network, which is used to execute the multimodal image fusion method based on an expanded residual attention network in the above method embodiment. See Figure 2The device includes: a first main module, which is used to implement preprocessing of the original image by Laplace enhancement and brightness enhancement, and extract a deep feature map from the preprocessed image; a second main module, which is used to implement the Softmax feature weighting strategy to fuse the deep feature map into a fused feature map; a third main module, which is used to determine the loss function for the fused feature map, generate a fused image, and use three consecutive 3*3 convolutional layers with output channels to reduce the number of channels from 64 to 1.

[0030] The multimodal image fusion device based on the dilated residual attention network provided by the embodiment of the present invention adopts Figure 2 Several modules in the system fuse as many features from different modalities as possible into the same image, thus avoiding the need for clinicians to repeatedly switch between the two modalities for the anatomical structures of the bone and disc when making diagnoses related to the temporomandibular joint area. This fused image simplifies the diagnostic process and provides the convenience of using one image for two purposes.

[0031] It should be noted that the device in the device embodiment provided by the present invention can be used to implement the method in the above method embodiment as well as the method in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set, and the principle is basically the same as the principle of the above device embodiment provided by the present invention. As long as the technical personnel in the field refer to the specific technical solutions in other method embodiments on the basis of the above device embodiment, obtain the corresponding technical means and the technical solutions composed of these technical means by combining technical features, the device in the above device embodiment can be improved on the premise of ensuring the practicality of the technical solution, thereby obtaining the corresponding device class embodiment, which is used to implement the methods in other method class embodiments. For example: Based on the content of the above-mentioned device embodiment, as an optional embodiment, the multimodal image fusion device based on the dilated residual attention network provided in the embodiment of the present invention also includes: a first sub-module, used to realize the extraction of deep feature maps from the preprocessed images, including: the preprocessed image is first input into a residual attention network with a 1*1 convolutional layer to obtain a 64-dimensional shallow feature image, and then the shallow feature image is passed to the dilated residual attention network to obtain a deep feature map.

[0032] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the multimodal image fusion device based on the dilated residual attention network provided in the embodiment of the present invention also includes: a second sub-module, used to realize the fusion of the deep feature map into a fused feature map by using the Softmax feature weighting strategy, including: obtaining two output feature maps from the input deep feature map respectively, performing Softmax operation on the two output feature maps, and after obtaining the Softmax output, calculating the sum of the singular values ​​through the nuclear norm of the matrix, taking the maximum weighted average of the nuclear norm to obtain the weight of the fused feature map, and obtaining the fused feature map according to the weight.

[0033] Based on the content of the above-mentioned device embodiment, as an optional embodiment, the multimodal image fusion device based on the dilated residual attention network provided in the embodiment of the present invention further includes: a third sub-module, used to implement the dilated residual attention network, including: a first convolution module, connected in parallel with the second convolution module and the third convolution module; a fourth convolution module, connected to the first convolution module, the second convolution module, the third convolution module, the pooling module, the fifth convolution module, the sixth convolution module and the seventh convolution module; an upsampling module, connected to the pooling module and the ReLu activation function module; an eighth convolution module, connected to the fifth convolution module, the sixth convolution module and the tenth convolution module; a ninth convolution module, connected to the seventh convolution module and the tenth convolution module; a ReLu activation function module, connected to the fourth convolution module, the fifth convolution module, the eighth convolution module and the tenth convolution module.

[0034] Based on the content of the above device embodiment, as an optional embodiment, the multimodal image fusion device based on the dilated residual attention network provided in the embodiment of the present invention further includes: a fourth submodule, which is used to implement the maximum weighted average of the nuclear norm to obtain the weight of the fusion feature map, including: ; in, is the jth output feature map The weight map obtained after the Softmax function operation is is the max() function, is the output feature map The weight of is the output feature map The weight of is the matrix Frobenius norm.

[0035] Based on the content of the above device embodiment, as an optional embodiment, the multimodal image fusion device based on the dilated residual attention network provided in the embodiment of the present invention further includes: a fifth submodule, which is used to implement the fusion feature map obtained according to the weight, including: ; Among them, f is the fused feature map.

[0036] Based on the content of the above device embodiment, as an optional embodiment, the multimodal image fusion device based on the dilated residual attention network provided in the embodiment of the present invention further includes: a sixth submodule, which is used to implement the loss function determined for the fusion feature map, including: ; in, is the loss function, M is the number of input images, is the output image, is the kth channel from the pth layer in the VGG16 network, is the kth channel from the qth layer in the VGG16 network, is the matrix 2 norm, MSE is the mean square error of the regression loss function, is the gradient loss, is the length and width of the input p-layer image, C is the number of channels, Per is the perceptual loss, h is the width of the image, is the length of the image, is the gradient loss of the output image, is the gradient loss of the q-layer image, is the qth layer image, is the weight set of the target optimization network, It is the weight balancing factor between gradient loss and perceptual loss.

[0037] The method of the embodiment of the present invention is implemented by relying on electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 3 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein at least one processor, the communication interface, and at least one memory communicate with each other through the communication bus. At least one processor can call the logic instructions in at least one memory to execute all or part of the steps of the method provided by the aforementioned various method embodiments.

[0038] In addition, the logic instructions in the at least one memory mentioned above can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0039] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0040] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0041] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. Based on this understanding, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and sometimes in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0042] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. Any "predetermined threshold", "preset threshold" and other similar expressions that do not indicate specific values ​​can be determined by a person of ordinary skill in the art through simple experiments or corresponding debugging.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal image fusion method based on dilated residual attention network, characterized in that: include: The original image is preprocessed by Laplacian enhancement and brightness enhancement, and the deep feature map is extracted from the preprocessed image; The Softmax feature weighting strategy is used to fuse the deep feature maps into a fused feature map. The loss function is determined for the fused feature map to generate a fused image, and three consecutive 3*3 convolutional layers with output channels are used to reduce the number of channels from 64 to 1.

2. The multimodal image fusion method based on dilated residual attention network according to claim 1, characterized in that: The method of extracting a deep feature map from a preprocessed image includes: first inputting the preprocessed image into a residual attention network with a 1*1 convolutional layer to obtain a 64-dimensional shallow feature image, and then transferring the shallow feature image to an expanded residual attention network to obtain a deep feature map.

3. The multimodal image fusion method based on dilated residual attention network according to claim 2, characterized in that: The Softmax feature weighting strategy is used to fuse the deep feature map into a fused feature map, including: obtaining two output feature maps from the input deep feature map respectively, performing Softmax operation on the two output feature maps, and after obtaining the Softmax output, calculating the sum of singular values ​​through the nuclear norm of the matrix, taking the maximum weighted average of the nuclear norm to obtain the weight of the fused feature map, and obtaining the fused feature map according to the weight.

4. The multimodal image fusion method based on dilated residual attention network according to claim 3 is characterized in that: The dilated residual attention network includes: a first convolution module, which is connected in parallel with the second convolution module and the third convolution module; a fourth convolution module, which is connected to the first convolution module, the second convolution module, the third convolution module, the pooling module, the fifth convolution module, the sixth convolution module and the seventh convolution module; an upsampling module, which is connected to the pooling module and the ReLu activation function module; an eighth convolution module, which is connected to the fifth convolution module, the sixth convolution module and the tenth convolution module; a ninth convolution module, which is connected to the seventh convolution module and the tenth convolution module; and a ReLu activation function module, which is connected to the fourth convolution module, the fifth convolution module, the eighth convolution module and the tenth convolution module.

5. The multimodal image fusion method based on dilated residual attention network according to claim 4, characterized in that: The taking the maximum weighted average of the nuclear norm to obtain the weight of the fused feature map includes: ; in, is the jth output feature map The weight map obtained after the Softmax function operation is: is the max() function, is the output feature map The weight of is the output feature map The weight of is the matrix Frobenius norm.

6. The multimodal image fusion method based on dilated residual attention network according to claim 5, characterized in that: The step of obtaining a fused feature map according to the weights includes: ; Among them, f is the fused feature map.

7. The multimodal image fusion method based on dilated residual attention network according to claim 6, characterized in that: The determining of the loss function for the fused feature map includes: ; in, is the loss function, M is the number of input images, is the output image, is the kth channel from the pth layer in the VGG16 network, is the kth channel from the qth layer in the VGG16 network, is the matrix 2 norm, MSE is the mean square error of the regression loss function, is the gradient loss, and is the length and width of the input p-layer image, C is the number of channels, Per is the perceptual loss, h is the width of the image, is the length of the image, is the gradient loss of the output image, is the gradient loss of the q-layer image, is the qth layer image, is the weight set of the target optimization network, and It is the weight balancing factor between gradient loss and perceptual loss.

8. A multimodal image fusion device based on dilated residual attention network, characterized in that: include: The first main module is used to implement preprocessing of the original image by Laplace enhancement and brightness enhancement, and extract a deep feature map from the preprocessed image; The second main module is used to fuse the deep feature map into a fused feature map using the Softmax feature weighting strategy; the third main module is used to determine the loss function for the fused feature map, generate the fused image, and use three consecutive 3*3 convolutional layers with output channels to reduce the number of channels from 64 to 1.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein, The processor, memory and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which cause a computer to execute the method of any one of claims 1 to 7.

Images