Lightweight medical image fusion method

Abstract

The invention belongs to the technical field of medical image processing, and particularly relates to a lightweight medical image fusion method. The method comprises three stages of feature extraction, feature fusion and feature reconstruction. Firstly, a targeted double-branch extraction path is designed for a feature extraction stage, an independent module design with a structure and a function separated is adopted to distinguish and enhance saliency specific information, and a wavelet convolution and state space module is adopted to complete lightweight extraction of local and global information of different modes; secondly, designing a multi-scale feature fusion unit built by a position coding and state space module in a feature fusion stage, and performing cross-modal fusion on details and global information from two branches; and finally, recovering the fused features to a unified scale in a feature reconstruction stage, and obtaining a final fused image. According to the invention, targeted feature extraction and lightweight medical image fusion are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a lightweight medical image fusion method. Background Art

[0002] Multimodal medical image fusion technology can integrate medical images from different sensors into a unified, complementary fused image, providing important auxiliary support and basis for accurate clinical diagnosis. Typically, multi-source medical imaging methods focus on computed tomography (CT), positron emission tomography (PET), single-photon emission computed tomography (SPECT), and magnetic resonance imaging (MRI). These images are broadly divided into structural images (MRI) and functional images (CT, PET, SPECT) based on their presentation characteristics. For example, PET images can provide information on blood flow and cellular metabolism in tissues and organs, while MRI images provide information on the complete soft tissue structure. Therefore, medical image fusion technology that can effectively integrate this complementary information has become one of the important development directions in the field of medical image processing.

[0003] Currently, deep learning-based methods have become the main method for medical image fusion. These methods mainly include several types based on neural networks, autoencoders (AE), and generative networks. Neural network-based methods usually use deep network structures and carefully designed loss functions to process image features; autoencoder (AE)-based methods use trained encoders to process features of multi-source images under specific fusion tasks; and generative network-based methods explore the mapping relationship between source and target images from the latent space of image features, and obtain fused images through adversarial generation or approximate generation. However, the above three methods still have shortcomings in the targeted acquisition of significant features of different modalities and the design of lightweight network structures, resulting in problems such as redundant computational parameters and loss of complementary information in the fusion model. Summary of the Invention

[0004] The present invention aims to provide a lightweight medical image fusion method to solve the problems of computational redundancy and loss of complementary information in existing image fusion technologies.

[0005] To solve the above technical problems, the present invention provides a lightweight medical image fusion method, comprising the following steps:

[0006] An image fusion network is constructed and trained, and the registered medical images are input into the trained image fusion network for fusion to obtain a fused image. The construction of the image fusion network includes three feature processing stages: feature extraction, feature fusion, and feature reconstruction. First, in the feature extraction stage, the structural feature extraction module and the functional feature extraction module under the dual-branch path are used to perform targeted enhancement and extraction on the structural image and the functional image, thereby obtaining shallow inherent features under each modality, and serving as the input of the lightweight global feature extraction module for deep global feature extraction. Secondly, in the feature fusion stage, the output of the feature extraction stage is first subjected to position encoding interaction of the multi-scale feature fusion unit to obtain two shared features after primary interaction, and then through the three-phase structure of the state space module combination, the modal information is subjected to multi-level cross-modal interaction and fusion processing to obtain a multi-scale high-dimensional feature. Finally, in the feature reconstruction stage, the multi-scale high-dimensional features are gradually restored to a unified scale through the feature reconstruction module and added and integrated into a fused image.

[0007] Preferably, the feature extraction stage in the above method includes: a structural feature extraction module and a functional feature extraction module designed for saliency information extraction, a wavelet convolution module for extracting shallow detail features of different modalities, and a lightweight global feature extraction module;

[0008] The structural feature extraction module is a convolutional neural network based on Gabor filtering and wavelet transform, which includes a layer of Gabor filter, a layer of discrete wavelet transform and an ordinary two-dimensional convolution. The calculation process is as follows:

[0009]

[0010]

[0011]

[0012] The functional feature extraction module is a convolutional neural network based on Laplace filtering and wavelet transform, including a layer of Laplace filter, a layer of discrete wavelet transform and an ordinary two-dimensional convolution. Its calculation process is as follows:

[0013]

[0014]

[0015]

[0016] Among them, x and y are structural image input features and functional image input features respectively. For the enhanced structural characteristics, For enhanced functional features, For The characteristic components extracted after wavelet decomposition in the frequency domain, For The characteristic components extracted after wavelet decomposition in the frequency domain, is the output of the structural feature extraction module, is the output of the corresponding feature extraction module, Indicates a core is 1 1 convolution operation, It is a high-pass filter designed based on the Gabor operator. It is a high-pass filter designed based on the Laplace operator. The convolution kernel is 3 3 two-dimensional convolution, represents the average pooling operation, represents the convolution operation, represents batch normalization, represents the nonlinear activation function (ReLU), and is the wavelet transform and inverse wavelet transform, is the convolution kernel, is the convolution bias;

[0017] Here, the structural feature extraction module is used to strengthen the edge structure information and enhance the local area contrast through Gabor filtering and two-dimensional convolution, and the cross-channel information is extracted through wavelet transform combined with convolution. The functional feature extraction module is used to extract high-frequency texture information and retain basic information such as background and contour through Laplace filtering and average pooling operations. The different processing methods on the two branches enable specific representation of the saliency information between different modalities.

[0018] Preferably, the lightweight global feature extraction module in the above method includes a visual state space module and a state space module, wherein the visual state space module is used to selectively retain the global saliency information output from the upper layer, and the state space module is N=4 2, i.e., it contains two levels of feature extraction, each level consists of four state space modules, which are used to lightweight extract context information of image features under single modality;

[0019] The calculation process of the visual state space module is as follows:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] in, 、 are the implicit state equations of the feature input sequence when the time step is t and the direction is k, and the time step is t-1 and the direction is k, respectively. is the characteristic output equation when the time step is t and the direction is k, 、 、 、 yes The state update parameter matrix when the implementation direction is k perception, is the feature input sequence when the time step is t and the direction is k, is the output from the upper feature extraction module, It is achieved by taking the image feature pixels Linear normalization operation that maps the maximum and minimum values ​​to the standardized range, is a linear mapping, is a depth-wise separable convolution, is the activation function, is the selective scanning equation, " represents the matrix multiplication operation (also known as Hadamard product) of two element sequences arranged in matrix form. is a multi-directional high-dimensional feature. The downlink branch is normalized, linearly mapped and Function activated feature map, first layer global feature This corresponds to the final output of the visual state space module;

[0028] The calculation process of the state space module is as follows:

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] in, is the final output of the upper visual state space module, Indicates that the core is 1 1 convolution operation, Indicates that the core is 3 3 convolution operations, is the feature embedding operation, is the linear normalization, It is a splitting operation along the dimension, is the feature map, is the activation function, 、 The time steps are 、 The implicit state equation of the characteristic input sequence is 、 、 、 is the state update parameter matrix, The time step is The characteristic output equation when For preliminary features, and For Apply layer normalization to standardize feature distribution and pass Operations on preliminary features along the channel dimension The two sets of features obtained by equal division are For the enhanced features, is the output of the current first-layer state space module;

[0036] Here, SS2D scanning can obtain context sequence perception along different directions, and the other branch completely expands the upper-level output to a high-dimensional level. After the matrix multiplication operation between the output features, the final feature map completes the multi-directional global perception at the sequence level. The feature embedding operation converts the two-dimensional feature tensor into a one-dimensional sequence expression, which is conducive to implicit modeling in the state space, reducing the complexity redundancy caused by secondary calculations, and obtaining higher-dimensional spatial features.

[0037] Preferably, the feature fusion stage in the above method is a two-level multi-scale feature fusion unit for feature interaction and fusion, and the multi-scale feature fusion unit includes a position encoding interaction layer and a three-phase structure fusion layer composed of a state space module;

[0038] The calculation process of the position coding interaction layer is as follows:

[0039]

[0040]

[0041] where, is the encoding of the current position channel, is a random 0, 1 encoding, and is the sequence feature of the c-th channel of the output of the N-layer state space module of the two branches, and is the output corresponding to the interaction;

[0042] wherein the three-phase structure fusion layer is combined by 7 state space modules, and the calculation process is:

[0043]

[0044]

[0045]

[0046]

[0047] wherein, and are two output features after position encoding interaction, is the output equation of the state space module, is linear normalization, is 3 3 convolution, , , is the feature output of three different scales, is the final output of the multi-scale feature fusion unit;

[0048] Here, the state space module is used to reduce the calculation amount and increase the receptive field, the linear normalization and the residual connection are used to improve the training stability, and the three-phase network structure is used to make the interlayer information interaction more sufficient, so as to ensure the reservation of complementary information.

[0049] Preferably, the feature reconstruction stage in the above method includes a feature reconstruction module constructed by combining a state space module and a convolution. The feature reconstruction module uses the final output of the multi-scale feature fusion unit as the input of the first layer in the feature reconstruction module, uses the state space module to reduce the dimension of the output, reduces it one by one to a unified dimension and combines it with convolution for integration output, and finally restores the multi-scale image to a fused image of unified scale; the final fused image is an RGB fused image, and the final RGB fused image is obtained by extracting the color channel information in the source functional image and converting it with the fused image through YCrCb color space conversion, thereby converting the obtained grayscale fused image into an RGB fused image;

[0050] Here, when the functional image is a pseudo-color image, such as a PET or SPECT image, the grayscale fused image can be converted into a color space to obtain richer color information, thereby improving the visual perception effect.

[0051] Preferably, when calculating the loss using a loss function during the training of the image fusion network in the above method, the high- and low-frequency features decomposed by wavelet decomposition during the feature extraction process are used as effective constraints, and the loss function is calculated in combination with the feature distribution between the source image and the fused image; the loss function includes high- and low-frequency losses based on wavelet decomposition, structural similarity loss, and contrast loss;

[0052] The constraint design based on frequency components here ensures the effective retention of saliency information in different images, and the constraint design based on overall feature distribution ensures the consistency of the overall image structure; the high and low frequency losses based on wavelet decomposition can implicitly balance the contribution of low and high frequencies to the loss without the need for explicit weighting parameters, which to a certain extent avoids absolute dependence on hyperparameters and realizes the inherent feature constraints of the source image; the structural similarity loss ensures the invariance of the overall structure, and the contrast loss enhances edge details. In general, the joint constraints of the three losses effectively improve the visual perception effect and image quality.

[0053] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0054] The image fusion network used in the present invention is designed with a dual-branch feature extraction path with targeted feature extraction capabilities and a multi-scale feature fusion unit that reduces the loss of complementary information. Among them, one path in the dual-branch feature extraction path includes a structural feature extraction module and a lightweight global feature extraction module for structural medical images, and the other path includes a functional feature extraction module and a lightweight global feature extraction module for functional medical images. The structural feature extraction module and the functional feature extraction module can enhance and capture the significant features of images from different modalities. The lightweight global feature extraction module can realize contextual information perception under the premise of lightweight. The multi-scale feature fusion unit can perform cross-modal interaction and fusion processing of the inherent modal information obtained from the dual-branch extraction path, promote the flow of information between different modalities and reduce the loss of complementary information, so that the medical image fusion task performed using the fusion network can achieve the goals of lightweight and high performance.

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is the multimodal medical image fusion network architecture of the present invention;

[0057] Figure 2 Flowchart for implementing the method of the present invention;

[0058] Figure 3 Schematic diagram of the process of implementing position coding in the feature fusion stage of the present invention;

[0059] Figure 4 Schematic diagram of the three-phase structure implementation process of the feature fusion stage in the present invention;

[0060] Figure 5 Schematic diagram of the three sets of inputs and their corresponding fusion results used in the examples of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] It is understood that the terms "first," "second," and the like may be used herein to describe various output results, but unless otherwise specified, these output results are not limited by these terms. These terms are used solely to distinguish a first element from another element. For example, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script without departing from the scope of this application.

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0064] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0065] Combine Figures 1-2 The process of the lightweight medical image fusion method is as follows: first, an image fusion network is constructed and trained, and then the registered medical image is input into the trained fusion network for fusion to obtain a fused image; wherein, the constructed image fusion network includes three stages: feature extraction, feature fusion, and feature reconstruction; in the feature extraction stage, the structural feature extraction module and the functional feature extraction module under the dual-branch path are first used to perform targeted enhancement and extraction on the structural image and the functional image, so as to obtain the shallow inherent features of each modality, and serve as the input of the lightweight global feature extraction module for deep global feature extraction; then, the output is first subjected to the position encoding interaction of the multi-scale feature fusion unit to obtain two shared features after the primary interaction, and then through the three-phase structure of the state space module combination, the modal information is subjected to multi-level cross-modal interaction and fusion processing to obtain a multi-scale high-dimensional feature; finally, the feature reconstruction module gradually restores the multi-scale complementary information to a unified scale and adds and integrates it into a fused image.

[0066] In this example, functional images and their corresponding structural images from three different imaging modalities were selected as medical image pairs: CT-MRI, PET-MRI, and SPECT-MRI. The following describes the implementation of this method in detail using a fusion network and specific examples.

[0067] Step 1: Use the structural feature extraction module and functional feature extraction module of the dual-branch path to perform targeted feature extraction and enhancement on different inputs.

[0068] The structural feature extraction module is used to process MRI input. This module mainly includes a layer of Gabor filter, a layer of discrete wavelet transform, and an ordinary two-dimensional convolution. The filter is introduced to enhance the edge strength while using two-dimensional convolution to focus on local details. The enhanced features are then normalized and nonlinearly integrated and activated. They are then subjected to a layer of improved wavelet transform convolution for multi-scale feature extraction to obtain the final structural enhancement features. The functional feature extraction module is used to process CT (or PET, SPECT) input. This module mainly includes a layer of Laplace filter, a layer of discrete wavelet transform, and an ordinary two-dimensional convolution. The filter is introduced to enhance high-frequency texture and brightness details while using average pooling to retain effective background information. The enhanced features are then normalized and nonlinearly integrated and activated. They are then subjected to a layer of improved wavelet transform convolution for multi-scale feature extraction to obtain the final functional enhancement features. The specific implementation details of this process are as follows:

[0069] 1) First, the two input images are preprocessed by averaging and scaling to ensure alignment and fusion between different modal features. The formula for the operation steps is:

[0070]

[0071]

[0072]

[0073]

[0074] Among them, A and B are the unprocessed original feature maps, H and W are the sizes of the corresponding images, , is the average intensity value of the original images A and B, and R is a random scaling factor in the range [0,1].

[0075] 2) Secondly, determine whether the input image of the functional branch is a color image (the color image size is , where 3 is the number of channels. When the number of channels is 1, it is a grayscale image). If it is a color image, it needs to be converted into a grayscale image in the YCbCr color space, that is, only the component Y representing the brightness is retained. After preprocessing, two images of size The feature map, that is, x and y. The calculation formula for YCbCr color space conversion is:

[0076]

[0077]

[0078]

[0079] Among them, R, G, and B represent the intensity values ​​of the image in the red, green, and blue color channels respectively, and their range belongs to [0,255]. They together constitute the original representation of the image in the RGB color space, while Y corresponds to the brightness component, and Cb and Cr are the chrominance components.

[0080] 3) Subsequently, the structural image input feature x and the functional image input feature y are input to the structural feature extraction module and the functional feature extraction module respectively. Among them, the feature x is respectively filtered by Gabor and a convolution kernel size of The convolution is performed locally, and then linear normalized and ReLU activated to obtain the enhanced structural features. , and then use wavelet decomposition to Perform feature extraction in the frequency domain and obtain the output of the structural feature extraction module by summing the multi-scale convolution of the decomposed frequency components The calculation process is:

[0081]

[0082]

[0083]

[0084] The processing of y is similar to that of x, except that the filter used in the enhancement process of y is Laplace filtering. The convolution is also replaced by the average pooling operation, and finally the output of the feature extraction module is obtained through the subsequent same operation. The calculation process is:

[0085]

[0086]

[0087]

[0088] Among them, x and y are structural image input features and functional image input features respectively. For the enhanced structural characteristics, For enhanced functional features, For The characteristic components extracted after wavelet decomposition in the frequency domain, For The characteristic components extracted after wavelet decomposition in the frequency domain, is the output of the structural feature extraction module, is the output of the corresponding feature extraction module, Indicates a core is 1 1 convolution operation, It is a high-pass filter designed based on the Gabor operator. It is a high-pass filter designed based on the Laplace operator. The convolution kernel is 3 3 two-dimensional convolution, represents the average pooling operation, represents the convolution operation, represents batch normalization, represents the nonlinear activation function (ReLU), and is the wavelet transform and inverse wavelet transform, is the convolution kernel, is the convolution bias.

[0089] Step 2: Use a lightweight global feature extraction unit to extract global features under each mode.

[0090] The lightweight feature extraction unit consists of a visual state space module and several layers of state space modules. The visual state space module is used to selectively retain the global saliency information from the upper layer output, while the state space module uses N=4 2, that is, it contains two levels of feature extraction, each level consists of four state space modules, which are used to extract the context information of single-modal image features in a lightweight way. The specific implementation of this process is as follows:

[0091] 1) First, and Each of them will pass through a visual state space module to selectively extract global information from the saliency information. In this process, the operations on the two features are the same. They are first normalized by the upstream branch of the visual state space module, and then linearly projected to a higher dimensional space. Then, depthwise separable convolution is used to perform independent convolution operations on each channel to extract cross-channel features, and then pass through the The activation function generates a weight map after nonlinear activation and obtains multi-directional high-dimensional features under the selective scanning mechanism of SS2D. and through Perform linear normalization operation. Then normalize, linearly map and Feature map of function activation After matrix multiplication operation and upper layer features Multiply the matrix again to get the first layer of global features , and its calculation process is:

[0092] for and , abbreviated as

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100] in, 、 are the implicit state equations of the feature input sequence when the time step is t and the direction is k, and the time step is t-1 and the direction is k, respectively. is the characteristic output equation when the time step is t and the direction is k, 、 、 、 yes The state update parameter matrix when the implementation direction is k perception, is the feature input sequence when the time step is t and the direction is k, is the output from the upper feature extraction module, It is achieved by taking the image feature pixels Linear normalization operation that maps the maximum and minimum values ​​to the standardized range, is a linear mapping, is a depth-wise separable convolution, is the activation function, is the selective scanning equation, " represents the matrix multiplication operation (also known as Hadamard product) of two element sequences arranged in matrix form. is a multi-directional high-dimensional feature. The downlink branch is normalized, linearly mapped and Function activated feature map, first layer global feature This corresponds to the final output of the visual state space module.

[0101] 2) Then, the first layer of global features After a The state space stacking blocks of the layer are used to extract deep context features. , first by The convolution is flattened; then the Patch Embedding method is used to embed the flattened feature representation, that is, to encode the features of the local area into a fixed-dimensional representation to obtain the preliminary features. ; Then, Layer normalization is applied to standardize the feature distribution and the features are evenly divided along the channel dimension by chunking to obtain two sets of features. and ; Among them, a set of features After the convolution kernel size is After the convolution operation, the SiLU activation function is applied to extract nonlinear features. Another set of features The SiLU activation function is directly applied to maintain its original information; then, the above two sets of activation features are multiplied element by element, and the result is input into the state space output equation (SSM) to enhance the temporal dependency and context modeling, and the feature is obtained. ;Finally, the enhanced features The output feature is obtained by restoring the feature map to the original dimension and performing residual connection with the initial feature. ;against repeat The final output of the feature extraction stage is obtained The calculation process is:

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108] in, is the final output of the upper visual state space module, Indicates that the core is 1 1 convolution operation, Indicates that the core is 3 3 convolution operations, is the feature embedding operation, is the linear normalization, It is a split operation along the dimension, is the feature map, is the activation function, 、 The time steps are 、 The implicit state equation of the characteristic input sequence is 、 、 、 is the state update parameter matrix, The time step is The characteristic output equation when For preliminary features, and For Apply layer normalization to standardize feature distribution and pass Operations on preliminary features along the channel dimension The two sets of features obtained by equal division are For the enhanced features, is the output of the current first-layer state-space module.

[0109] Step 3: Use the multi-scale feature fusion module to fusion the final features of the feature extraction stage and Interact and integrate.

[0110] 1) First, combine Figure 3 ,right and Perform layered position encoding on the channel, that is, for the c channel and , perform random assignment of 0 or 1, and add and concatenate the interactive features after encoding assignment to obtain the final shallow shared features and The calculation formula of shallow shared features is as follows:

[0111]

[0112]

[0113] in, is the code of the current position channel, is a random 0 or 1 code. and is the sequence feature of the cth channel output by the N-layer state space module under the two branches, and is the output corresponding to the interaction.

[0114] 2) Secondly, for shared features and Conduct multi-stage deep interaction and fusion, that is, use the three-phase structure designed by the state space module combination to process it. Figure 4, first two features each enter a state space module for feature extraction, the extracted features are added and become an independent branch as a fusion branch for cross-modal feature fusion; then the first level feature output is normalized and then input into the state space module for processing, and local convolution is introduced to adjust the channel to obtain the second level feature; then the second level two feature outputs and the output in the fusion branch are added again to obtain the second level feature of the fusion branch; finally, the three second level features are normalized and processed by the state space module and added to obtain the final fusion feature . The calculation formula of the three-phase structure feature fusion process is as follows:

[0115]

[0116]

[0117]

[0118]

[0119] wherein, and are two output features after position coding interaction, is the output equation of the state space module, is linear normalization, is 3 3 convolution, , , are three feature outputs of different scales, is the final output of the multi-scale feature fusion unit.

[0120] Step four, using the feature reconstruction module to reconstruct and restore its scale.

[0121] 1) For the fusion feature , four layers of state space modules are used for sequential structured modeling to further enhance the fusion feature;

[0122] 2) Then the fusion feature is operated by Patch Unembedding for feature patching. Specifically, Patch Unembedding restores multiple linearly embedded small block features to a two-dimensional image form of the fusion feature map according to their spatial positions, so as to realize spatial remodeling of feature dimensions and restoration of image structure;

[0123] 3) Finally, through the convolution of , the feature map is reduced and integrated, and the fusion image with the same size as the original image is directly output by applying the Sigmoid activation function .

[0124] Step five, if the judgment result of the functional input in step one is color, the fused image further color information processing, that is, the color components Cb and Cr in the original functional image are assigned , and restored to the RGB fused image through color space conversion. The calculation formula for restoring to the RGB space is as follows:

[0125]

[0126]

[0127]

[0128] wherein R, G, B represent the intensity values of the image in red, green and blue color channels respectively, and 128 is subtracted to control the Cb and Cr chroma components in [0, 255] to prevent color distortion.

[0129] Step six, iterative training of the fusion network using loss constraint.

[0130] In the design of the model structure, a variety of loss joint methods are adopted to constrain the training and generation of the fusion result, so as to ensure the visual perception effect and image quality of the fusion result. The implementation process of the specific constraint is as follows:

[0131] 1) In the feature extraction stage, aiming at the difference between the saliency information in the frequency component and the frequency distribution of the fused image, the features and under the two feature extraction branches are divided into and according to high and low frequencies. Among them, contains , . denoted as , . The optimizes the details and edges of the image, and the pays attention to the global structure of the image. The specific constraint form is as follows:

[0132]

[0133]

[0134]

[0135] wherein, represents the high-frequency loss. represents the gradient form of the high-frequency component, is the gradient representation of the high-frequency component of the fused image. It represents the maximum selection strategy for effectively aggregating the texture details of the source images. The purpose is to fuse the gradient information of the source images and retain the high-intensity detail information in the two source images by maximizing the gradient in each direction. Finally, express Loss. It indicates low frequency loss, represents the low-frequency component of the source image, is the low-frequency component of the fused image. Finally, the loss constraint of the feature extraction stage is defined as .

[0136] 2) Secondly, using structural similarity loss Balance and constrain the consistency of image feature structure during the entire network training process. The specific constraint form is:

[0137]

[0138]

[0139] in, and is the average brightness of image xs,y, and represents the variance of image x,y, Represents the covariance of image x, y, and C is a constant. and represents the source image, To fuse the images, is the structural similarity index, and Both are set to 0.5 to balance the structural differences between different modes.

[0140] 3) In addition, contrast loss is used Constrain the brightness distribution to guide the generation of a fused image with appropriate visual effects. The specific constraint form is:

[0141]

[0142] in, is the image size, It is an element-wise aggregation operation that finds the best distribution of brightness values ​​based on the pixel distribution at the image size.

[0143] 4) The total loss constraint used by the fusion network during training is , and its calculation formula is:

[0144]

[0145] in As the weighted sum of all constraints in the entire network, and Is a hyperparameter that controls the balance between structural similarity loss and contrast loss. In this instance, , .

[0146] Based on the three sets of input images used in the training network and their corresponding fusion results, Figure 5 shown.

[0147] Based on the present invention, a computational complexity analysis is performed on the key components of the overall network structure. Specifically, a modular analysis is performed on the key components of the network structure according to the three stages of feature processing. The specific computational complexity analysis is as follows:

[0148] In the feature extraction stage,

[0149]

[0150]

[0151]

[0152]

[0153]

[0154]

[0155]

[0156] In the feature fusion stage,

[0157]

[0158] In the feature reconstruction phase,

[0159]

[0160] For the entire network,

[0161]

[0162] Among them, B is the batch size, H and W are the image sizes, and C is the number of channels. is the convolution kernel size, and r is the channel compression ratio. The computational size of the two specificity extraction modules and wavelet transform convolution in the feature extraction stage is represented. and They represent the computational size of the state space stacking block and the visual state space module in the global feature extraction module respectively. 、 、 They correspond to the total computational size in the three stages respectively. is the total computational size of the complete network. In this embodiment, (B, C, H, W) is set to (12, 1, 64, 64). , According to the above formula analysis, the computational complexity of this method is mainly reflected in the modeling of the state-space module and the operation of the convolution module. Although the computational complexity will increase with the increase of image size and channel dimension, this is inevitable in the existing model. Compared with other models, this method is lightweight in terms of computational complexity.

[0163] In summary, the present invention extracts significant local and global information from different modalities through a targeted feature extraction module and a lightweight global feature extraction unit, and uses a multi-scale feature fusion module to perform multi-level interaction and fusion of the extracted features to retain complementary information, and finally reshapes the final fused image through a reconstruction module. In addition, a loss constraint based on wavelet decomposition combined with structure and contrast is designed to guide the extraction and fusion of target features during the training process. Therefore, the present invention effectively ensures the targeted extraction of modality-inherent features and the retention of complementary information under the premise of lightweight, and can construct a multimodal medical fusion image with good visual effects.

[0164] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A lightweight medical image fusion method, characterized in that: include: An image fusion network is constructed and trained, and the registered medical images are input into the trained image fusion network for fusion to obtain a fused image. The construction of the image fusion network includes three feature processing stages: feature extraction, feature fusion, and feature reconstruction. First, in the feature extraction stage, the structural feature extraction module and the functional feature extraction module under the dual-branch path are used to perform targeted enhancement and extraction on the structural image and the functional image, thereby obtaining shallow inherent features under each modality, and serving as the input of the lightweight global feature extraction module for deep global feature extraction. Secondly, in the feature fusion stage, the output of the feature extraction stage is first subjected to position encoding interaction of the multi-scale feature fusion unit to obtain two shared features after primary interaction, and then through the three-phase structure of the state space module combination, the modal information is subjected to multi-level cross-modal interaction and fusion processing to obtain a multi-scale high-dimensional feature. Finally, in the feature reconstruction stage, the multi-scale high-dimensional features are gradually restored to a unified scale through the feature reconstruction module and added and integrated into a fused image.

2. A lightweight medical image fusion method according to claim 1, characterized in that: The feature extraction stage includes: a structural feature extraction module and a functional feature extraction module designed for extracting salient information, a wavelet convolution module for extracting shallow detail features of different modalities, and a lightweight global feature extraction module; The structural feature extraction module is a convolutional neural network based on Gabor filtering and wavelet transform, which includes a layer of Gabor filter, a layer of discrete wavelet transform and an ordinary two-dimensional convolution. The calculation process is as follows: The functional feature extraction module is a convolutional neural network based on Laplace filtering and wavelet transform, including a layer of Laplace filter, a layer of discrete wavelet transform and an ordinary two-dimensional convolution. Its calculation process is as follows: Among them, x and y are structural image input features and functional image input features respectively. For the enhanced structural characteristics, For enhanced functional features, For The characteristic components extracted after wavelet decomposition in the frequency domain, For The characteristic components extracted after wavelet decomposition in the frequency domain, is the output of the structural feature extraction module, is the output of the corresponding feature extraction module, Indicates a core is 1 1 convolution operation, It is a high-pass filter designed based on the Gabor operator. It is a high-pass filter designed based on the Laplace operator. The convolution kernel is 3 3 two-dimensional convolution, represents the average pooling operation, represents the convolution operation, represents batch normalization, represents the nonlinear activation function (ReLU), and is the wavelet transform and inverse wavelet transform, is the convolution kernel, is the convolution bias.

3. The lightweight medical image fusion method according to claim 2, characterized in that: The lightweight global feature extraction module includes a visual state space module and a state space module. The visual state space module is used to selectively retain the global saliency information output from the upper layer. The state space module is N=4 2, i.e., it contains two levels of feature extraction, each level consists of four state space modules, which are used to lightweight extract context information of image features under single modality; The calculation process of the visual state space module is as follows: in, 、 are the implicit state equations of the feature input sequence when the time step is t and the direction is k, and the time step is t-1 and the direction is k, respectively. is the characteristic output equation when the time step is t and the direction is k, 、 、 、 yes The state update parameter matrix when the implementation direction is k perception, is the feature input sequence when the time step is t and the direction is k, is the output from the upper feature extraction module, It is achieved by taking the image feature pixels Linear normalization operation that maps the maximum and minimum values ​​to the standardized range, is a linear mapping, is a depth-wise separable convolution, is the activation function, is the selective scanning equation, " " represents the matrix multiplication operation (also known as Hadamard product) of two element sequences arranged in matrix form. is a multi-directional high-dimensional feature. The downlink branch is normalized, linearly mapped and Function activated feature map, first layer global feature This corresponds to the final output of the visual state space module; The calculation process of the state space module is as follows: in, is the final output of the upper visual state space module, Indicates that the core is 1 1 convolution operation, Indicates that the core is 3 3 convolution operations, is the feature embedding operation, is the linear normalization, It is a splitting operation along the dimension, is the feature map, is the activation function, 、 The time steps are 、 The implicit state equation of the characteristic input sequence is 、 、 、 is the state update parameter matrix, The time step is The characteristic output equation when For preliminary features, and For Apply layer normalization to standardize feature distribution and pass Operations on preliminary features along the channel dimension The two sets of features obtained by equal division are For the enhanced features, is the output of the current first-layer state-space module.

4. The lightweight medical image fusion method according to claim 1, characterized in that: The feature fusion stage is a two-level multi-scale feature fusion unit for feature interaction and fusion. The multi-scale feature fusion unit includes a position encoding interaction layer and a three-phase structure fusion layer composed of a state space module. The calculation process of the position coding interaction layer is as follows: in, is the code of the current position channel, is a random 0 or 1 code. and is the sequence feature of the cth channel output by the N-layer state space module under the two branches, and is the output corresponding to the interaction; The three-phase structure fusion layer is composed of seven state space modules, and its calculation process is as follows: in, and are the two output features after the position encoding interaction, is the output equation of the state-space module, is the linear normalization, 3 3 convolutions, 、 、 are feature outputs of three different scales, It is the final output of the multi-scale feature fusion unit.

5. The lightweight medical image fusion method according to claim 1, characterized in that: The feature reconstruction stage includes a feature reconstruction module constructed by combining a state space module and a convolution. The feature reconstruction module uses the final output of the multi-scale feature fusion unit as the input of the first layer in the feature reconstruction module, uses the state space module to reduce the dimension of the output, reduces it one by one to a unified dimension, and combines it with convolution for integrated output, and finally restores the multi-scale image to a unified scale fused image; The final fused image is an RGB fused image, which is obtained by extracting the color channel information from the source functional image and converting the fused image into a YCrCb color space to obtain the final RGB fused image, thereby converting the obtained grayscale fused image into an RGB fused image.

6. The lightweight medical image fusion method according to claim 1, characterized in that: When using the loss function to calculate the loss during the training process of the image fusion network, the high and low frequency features decomposed by wavelet in the feature extraction process are used as effective constraints, and the loss function is calculated in combination with the feature distribution between the source image and the fused image; the loss function includes high and low frequency loss based on wavelet decomposition, structural similarity loss and contrast loss.

Images