A medical image super-resolution reconstruction method based on multi-modal fusion

Abstract

The application discloses a kind of medical image super-resolution reconstruction methods based on multi-modal fusion, it is related to the technical field of image processing, the method is first to the input multi-modal data is preprocessed and splicing fusion, make different modalities between spatial structure alignment, resolution and channel number consistent, then utilize multiscale convolution to input data feature extraction, and with mesoscale feature as intermediary cross-scale feature fusion, by the feature of transformer encoder fusion carries out global feature modeling, using self-attention mechanism enhances global context perception, obtains the feature map with stronger semantic representation ability by global context information enhancement, then the feature map is input into pre-trained generator, and makes generator generate super-resolution image;The application effectively solves the problem of details blur, obvious artifact and insufficient use of multi-modal data in medical image reconstruction, improves the visual quality and clinical diagnostic value of reconstructed image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method for super-resolution reconstruction of medical images based on multimodal fusion. Background Technology

[0002] With the continuous development of deep learning in image processing, image processing technology has significant advantages in the field of super-resolution image reconstruction. Image super-resolution is an image processing technique that transforms low-resolution (LR) images into high-resolution (HR) images by scaling them up to a target ratio. With the continuous improvement of modern medicine and the rapid development of medical imaging technology, super-resolution image reconstruction has also made significant progress in the field of medical imaging. Medical imaging technologies such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Electrical Impedance Tomography (EIT) have become important tools for doctors to diagnose and treat diseases. It is worth noting that multimodal medical data can provide complementary diagnostic information; however, most existing super-resolution methods are designed for single-modality imaging and fail to effectively utilize the correlation features between modalities, resulting in incomplete detail reconstruction and feature alignment deviations. Medical imaging plays an important role in clinical diagnosis, but it is often limited by factors such as equipment resolution, noise interference, and motion artifacts, leading to low image resolution and blurred details. Traditional super-resolution methods, such as bicubic interpolation, suffer from edge blurring. While convolutional neural network-based methods can improve resolution, they are prone to over-smoothing and loss of texture details.

[0003] In recent years, Generative Adversarial Networks (GANs) have demonstrated significant advantages in the field of image super-resolution, but problems such as training instability and loss of detail still exist. While GANs perform excellently in super-resolution of natural images, their ability to align inter-modal features and restore details is insufficient when directly applied to multimodal medical images. Therefore, there is an urgent need for a high-precision medical image super-resolution reconstruction method that can fuse multimodal data and combine local feature extraction with global dependency modeling. Summary of the Invention

[0004] The purpose of this invention is to provide a medical image super-resolution reconstruction method based on multimodal fusion, which solves the problems of detail loss, obvious artifacts, and insufficient utilization of multimodal data in existing medical image super-resolution reconstruction methods, so as to improve the visual quality and clinical diagnostic value of the reconstructed images.

[0005] The technical solution of this invention is: to provide a medical image super-resolution reconstruction method based on multimodal fusion, the method comprising:

[0006] Step 1: Obtain multimodal medical images from different data sources, preprocess and stitch the multimodal medical images to form standardized low-resolution images;

[0007] Step 2: Set up an MSIMM module containing multiple convolutional kernels of different sizes, use the MSIMM module to extract multi-scale features of low-resolution images, and use intermediate-scale features as an intermediary to perform cross-scale modulation and fusion of multi-scale features.

[0008] Step 3: Construct a Transformer encoder consisting of multiple identical stacked encoder layers. Each encoder layer includes a self-attention layer and a feedforward neural network. The feedforward neural network is composed of two linear transformation functions and one nonlinear activation function. The Transformer encoder is used to perform global feature modeling on cross-scale fused features, specifically including:

[0009] Step 31: Convert the cross-scale fusion features into a sequence representation, and introduce positional encoding into the sequence to form the input features;

[0010] Step 32: Iterate through each encoder layer in sequence. For a single encoder layer, first process the input features through a self-attention layer, and then perform residual connections and layer normalization. Then input the normalized features into the feedforward neural network, process them through two linear transformation functions and a nonlinear activation function, and then perform residual connections and layer normalization again.

[0011] Step 33: A gating mechanism is introduced after each self-attention layer. The normalized features of the self-attention layer output are processed by the gating mechanism to obtain weight gating. The output features of a single encoder layer are then weighted and fused based on the weight gating.

[0012] Step 34: Pass the weighted fused features to the next layer, and repeat the process from Step 32 to Step 33 until all encoder layers have completed the calculation, and finally output the first fused feature;

[0013] Step 4: Construct a generative adversarial network (GAN) containing a generator and a discriminator. Obtain high-resolution sample images from the database and downsample them to obtain low-resolution sample images. Use the high-resolution sample images and low-resolution sample images as samples to pre-train the GAN. At the same time, use a multi-objective loss function that combines adversarial loss, perceptual loss and TV loss to jointly optimize the generator and discriminator.

[0014] Step 5: Extract shallow features from the low-resolution image through a preset convolutional layer, concatenate the global context features and shallow features in the channel dimension to obtain enhanced fusion features, input the enhanced fusion features into the trained generator, and use the generator to generate a high-resolution image.

[0015] The beneficial effects of this invention are:

[0016] First, this invention establishes a joint modeling mechanism for multimodal medical images. During the input stage, multimodal medical images from different data sources are fused to generate low-resolution images, enabling effective collaboration among different modalities at the feature level. Multi-scale features of the low-resolution images are then extracted and cross-scale modulation fusion is performed to enhance feature representation, allowing the network to simultaneously consider both detailed textures and overall anatomical information. Global feature modeling is then performed based on a Transformer encoder, using a self-attention mechanism to model the global dependencies of the cross-scale fused features. A gating mechanism is introduced after each self-attention layer, further combining the gating mechanism to dynamically select and weight important features, suppressing redundant and irrelevant information. This allows the network to effectively capture the correlation between distant pixels in medical images, improving the stability and efficiency of network training. Finally, the modeled images are used to obtain... This invention combines global contextual features with shallow features of low-resolution images, and inputs the combined enhanced features into a pre-trained generator for high-resolution medical image reconstruction. This achieves collaborative reconstruction that balances global semantic information and local texture information, significantly improving the visual quality, structural accuracy, and clinical interpretability of the generated high-resolution medical images. Through the synergy of the above technical solutions, this invention can fully exploit the complementary information between multimodal medical images, overcoming the problem of insufficient utilization of modal correlation in existing single-modal super-resolution methods. It significantly improves the structural consistency and information integrity of reconstructed images, alleviates the problems of edge blurring, detail loss, and scale information fragmentation in traditional super-resolution methods, and enhances the clarity of reconstructed images. This improves the reliability and reference value of medical images in clinical diagnosis, demonstrating good generalization ability and engineering practical value.

[0017] Secondly, this invention utilizes a generative adversarial network that combines a generator and a multi-scale discriminator to generate high-resolution images. By employing a multi-objective optimization strategy that combines adversarial loss, perceptual loss, and total variation loss, the generation process is constrained and guided, achieving the collaborative reconstruction of global semantic information and local texture information. This further improves the quality of high-resolution medical images and solves the problems of detail distortion, excessive texture smoothing, and unrealistic structure that are prone to occur in existing medical image super-resolution methods. It enables the reconstructed image to maintain the overall structural realism while taking into account local details. Attached Figure Description

[0018] The advantages of the above and additional aspects of the present invention will become apparent and readily understood in the description of the embodiments in conjunction with the following drawings, wherein:

[0019] Figure 1 This is a schematic flowchart of a medical image super-resolution reconstruction method based on multimodal fusion according to an embodiment of the present invention;

[0020] Figure 2 This is a flowchart of medical image super-resolution reconstruction based on multimodal fusion and Transformer-ESRGAN according to an embodiment of the present invention;

[0021] Figure 3 This is a generator network structure diagram according to an embodiment of the present invention;

[0022] Figure 4 This is a discriminator network structure diagram according to an embodiment of the present invention;

[0023] Figure 5 This is a diagram of a multi-scale convolutional parallel structure according to an embodiment of the present invention;

[0024] Figure 6 This is a comparison chart of four lung image samples according to Example 1 of the present invention, and the reconstructed high-resolution medical images corresponding to the traditional bicubic interpolation method, super-resolution convolutional neural network, and the method of the present invention.

[0025] Figure 7 This is a local comparison image of the reconstructed high-resolution medical image corresponding to the conventional bicubic interpolation method, the super-resolution convolutional neural network, and the method of the present invention, according to Example 1 of the present invention. Detailed Implementation

[0026] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0027] In the following description, many specific details are set forth in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0028] like Figures 1 to 5 As shown, this embodiment provides a medical image super-resolution reconstruction method based on multimodal fusion, the method comprising:

[0029] Step 1: Obtain multimodal medical images from different data sources, and preprocess the multimodal medical images, including image registration and intensity normalization, to ensure that the images of each modality are consistent in spatial alignment and grayscale distribution. Then, stitch and fuse the preprocessed multimodal medical images to form a single standardized low-resolution image.

[0030] Specifically, this embodiment only processes 1- or 3-channel medical images, such as EIT images (Electrical Impedance Tomography), CT images (Computed Tomography), and MRI images (Magnetic Resonance Imaging). Preprocessing of multimodal medical images from different data sources includes high-precision spatial registration, unifying resolution and channel number, and normalization. Specifically, feature point matching and the RANSAC algorithm are first used to align pixels in multimodal medical images to ensure consistent anatomical structure positions in different modalities. Then, bicubic interpolation and upsampling techniques are used to unify low-resolution modal images to the scale of high-resolution modalities. Channel duplication is used to convert single-channel images into three-channel images (color space conversion is performed for modalities with inconsistent channel numbers). Finally, mean and variance normalization is performed on each channel of the image to normalize the data to the range of [0,1] to eliminate device differences and noise interference, providing standardized input for subsequent feature fusion.

[0031] The channel duplication method is used to convert a single-channel image into a three-channel image, specifically including:

[0032] Single-channel image data is a two-dimensional matrix, represented as Where H is the image height and W is the image width; the single-channel image data is copied by copying the two-dimensional matrix three times along the channel dimension (the third dimension), and then the three copied channel data are concatenated to form a new three-channel image data tensor, represented as... For each pixel (i,j) in a single-channel image, its pixel value is denoted as p. After conversion, the value of this pixel in all three channels is set to p, as shown in the expression:

[0033] ;

[0034] In the formula, c is the channel index.

[0035] The preprocessed multimodal medical images are stitched and fused along the channel dimension to form a standardized low-resolution image to be input. Specifically, EIT images, CT images, and MRI images are used as examples. The image data of each modality is a three-dimensional matrix, and the data format is as follows: H is the image height, W is the image width, and C1 is the number of channels corresponding to a single modality image data. The EIT image, CT image, and MRI image are stitched and fused along the channel dimension to obtain a new image, represented as:

[0036] ;

[0037] In the formula, Indicates channel splicing. These represent EIT images, CT images, and MRI images, respectively. Standardized low-resolution images obtained by stitching and fusing multimodal medical images.

[0038] In this embodiment, feature point matching and RANSAC algorithm are used to align pixels of multimodal medical images. Bicubic interpolation and upsampling techniques are used to unify low-resolution modal images to the scale of high-resolution modal images. Channel-wise mean-variance normalization is also performed. These are all commonly used processing methods and will not be described in detail here.

[0039] Step 2: Set up an MSIMM module containing multiple convolutional kernels of different sizes. Use the MSIMM module to extract multi-scale features from low-resolution images and use intermediate-scale features as an intermediary to perform cross-scale modulation and fusion of multi-scale features.

[0040] Specifically, an MSIMM module with three parallel convolutional kernels (3×3, 5×5, and 7×7) is configured. The original low-resolution image is first processed through a 3×3 convolutional layer. Shallow feature extraction is performed to obtain the original input feature x. The original input feature x is then input into the MSIMM module, where three parallel convolutional kernels of 3×3, 5×5, and 7×7 are used to extract features from it to capture features of different receptive fields and obtain local detail features. Features of intermediate scale Global context features , is represented as:

[0041] ;

[0042] In the formula, Corresponding to a 3×3 convolution kernel, Corresponding to a 5×5 convolution kernel, The corresponding 7×7 convolution kernel, x is the input feature corresponding to the low-resolution image.

[0043] Cross-scale modulation and fusion of multi-scale features using intermediate-scale features as an intermediary includes:

[0044] Features at intermediate scale Using element-wise multiplication as an intermediary, local detail features are respectively... and global context features Interactive modulation and fusion are represented as:

[0045] ;

[0046] In the formula, for and The fused features represent local details in the enhanced intermediate-scale features; for and The fused features represent the injection of global contextual information into the features at intermediate scales; symbols This represents element-wise multiplication (referring to multiplying elements at the same position in two tensors or matrices); it also refers to the modulation and fusion of features. and The concatenation is performed along the channel dimension, and then a 1×1 convolutional layer is used to compress the concatenated features back to the original number of channels to complete the fusion of multi-scale information, as shown below:

[0047] ;

[0048] ;

[0049] In the formula, for and The fusion characteristics after splicing To compress the fusion features back to the original number of channels, This represents the splicing operation along the channel dimension. Represents a 1×1 convolutional layer; fusion features compressed back to the original number of channels. The original input features x are subjected to residual connections to obtain the final cross-scale fused features, represented as:

[0050] ;

[0051] In the formula, For the final cross-scale fusion features, C2 represents cross-scale fusion features. The number of channels, A matrix representing the real number field.

[0052] It should be noted that by performing interactive modulation and feature fusion between different scales, the feature representation of the image can be improved, making it richer and more refined. When the subsequent model performs feature processing, it can simultaneously capture the local details, intermediate scale information and global context of the image. It is more comprehensive in processing multi-level information, which enhances the expressive power and robustness of the model. This helps to improve the accuracy and effectiveness of image processing, target recognition and image understanding, so as to better adapt to various complex visual tasks.

[0053] In this embodiment, the MSIMM module (Multi-Scale Interaction Modulation Module) supports setting various convolutional kernels of different sizes, allowing for flexible configuration according to actual processing needs. The size of the convolutional kernels can be adjusted according to specific application scenarios and performance requirements, thereby effectively extracting features at different scales and enhancing the module's adaptability and expressiveness.

[0054] Step 3: Construct a Transformer encoder consisting of multiple stacked identical encoder layers. Each encoder layer includes a self-attention layer and a feedforward neural network. The feedforward neural network is composed of two linear transformation functions and one nonlinear activation function. The Transformer encoder is used to perform global feature modeling on cross-scale fused features; specifically including:

[0055] Step 31: Convert the cross-scale fusion features into a sequence representation, and introduce positional encoding into the sequence to form the input features.

[0056] Specifically, cross-scale feature fusion Convert to sequence representation:

[0057] ;

[0058] In the formula, X represents the cross-scale fusion feature. The corresponding sequence, where n is the sequence length (n = H × W), and d is the feature dimension (d = C²). This represents sequence transformation.

[0059] To preserve the spatial location information corresponding to the features, a learnable location code P is introduced into the sequence to form the input features. , represented as , The position code P is generated using a cosine-sine coding method, represented as:

[0060] ;

[0061] ;

[0062] In the formula, pos is the position index of the current feature in the sequence (starting from 0 and ending at n-1), which is used to represent the order of the current feature in the sequence; i is the dimension index in the position encoding, which is used to distinguish the different dimensions in each position encoding. The generated position encoding is constructed by alternating sine and cosine functions.

[0063] Step 32: Iterate through each encoder layer in sequence. For a single encoder layer, first process the input features through a self-attention layer to enhance global context awareness, and then perform residual connections and layer normalization. Then, input the normalized features into the feedforward neural network, process them through two linear transformation functions and a nonlinear activation function, and perform residual connections and layer normalization again.

[0064] Specifically, the input features are processed through a self-attention layer, followed by residual connections and layer normalization, including:

[0065] Input features The input is fed into the self-attention layer in the current encoder layer. The self-attention layer consists of multiple self-attention modules. For a single self-attention module, the input features are first processed through different parameter matrices. The mapping is a set of queries, keys, and values, with each set of query, key, and value mappings forming a single header, represented as:

[0066] ;

[0067] In the formula, Q is the query matrix, K is the key matrix, and V is the value matrix. , , These are the learnable parameters corresponding to the query, key, and value matrices, respectively; then, based on the query, key, and value, the self-attention output corresponding to the current self-attention module (i.e., the output corresponding to a single head) is calculated independently, as follows:

[0068] ;

[0069] ;

[0070] In the formula, Represents the self-attention mechanism. This represents the self-attention output corresponding to the j-th self-attention module; finally, the self-attention outputs corresponding to all self-attention modules are concatenated together and mapped through a weight matrix to obtain the final output of the self-attention layer, represented as:

[0071] ;

[0072] In the formula, The final output of the self-attention layer, This is the weight matrix used to map the splicing result. It is a learnable parameter matrix that is optimized along with other parameters during training.

[0073] Will Recorded as The final output of the self-attention layer Residual connections and layer normalization are performed to enhance the training stability of the model, as follows:

[0074] ;

[0075] In the formula, Z represents the intermediate representation of the current encoder layer after passing through the self-attention layer, residual connections, and layer normalization. Representation layer normalization.

[0076] The normalized feature Z is input into the feedforward neural network in the current encoder layer and processed through two linear transformation functions and one nonlinear activation function, as follows:

[0077] ;

[0078] In the formula, , These are the weight matrices corresponding to the linear transformation functions of the first and second layers, respectively. , These are the bias vectors corresponding to the linear transformation functions of the first and second layers, respectively. The ReLU activation function is expressed as follows:

[0079] ;

[0080] In the formula, Activation function The independent variable; the output of the feedforward neural network

[0081] Residual connections and layer normalization are performed to enhance the training stability of the model, as follows:

[0082] ;

[0083] In the formula, These are the features in the current encoder layer after passing through the feedforward neural network, residual connections, and layer normalization.

[0084] Step 33: A gating mechanism is introduced after each self-attention layer. The normalized features of the self-attention layer output are processed by the gating mechanism to obtain weight gating. The output features of a single encoder layer are then weighted and fused based on the weight gating.

[0085] Specifically, the normalized feature Z output from the self-attention layer is processed using a gating mechanism to obtain the weight gating, which is expressed as:

[0086] ;

[0087] In the formula, G is the weighted gating factor. , Here, we have the learnable weight matrix and bias vector corresponding to the gating mechanism, which are used to project the feature Z into the gating space. The Sigmoid activation function is used to compress the projected values ​​to a range between 0 and 1. Its expression is:

[0088] ;

[0089] In the formula, Activation function The corresponding independent variables; features of the feedforward neural network output after residual connections and layer normalization based on weight gating. Dynamic, adaptive weighted fusion is performed, represented as:

[0090] ;

[0091] In the formula, G and (1-G) represent the weighted gating that allows passage and prohibits passage, respectively.

[0092] It should be noted that, to improve network stability and training efficiency, this embodiment also introduces a gating mechanism after each encoder layer. This gating mechanism is used to process complex local structural characteristics in medical images. This mechanism can enhance the network's selectivity for important features while suppressing interference from irrelevant information. The output of each encoder layer is passed to the next layer. After multiple layers of computation, a feature representation that integrates global information is finally obtained for use in subsequent tasks.

[0093] Step 34: Pass the weighted fused features to the next layer, repeating steps 32 and 33 until all encoder layers have completed computation, finally outputting the global context features, denoted as... .

[0094] It should be noted that the Transformer encoder in this embodiment can adopt the existing standard Transformer architecture, including a self-attention mechanism and a feedforward neural network. The role of each encoder layer is to capture the global dependencies between positions in the input data through the self-attention mechanism, and simultaneously enhance the model's expressive power by performing non-linear transformations on the features through the feedforward neural network. In this embodiment, the feedforward neural network is further processed by using two linear transformations and a ReLU activation function to further process the features extracted through the self-attention layer.

[0095] It should also be noted that in computer vision, Transformer encoders typically use around 12 encoder layers, with the specific number depending on the resolution of the input image and the requirements of the task.

[0096] Step 4: Construct a generative adversarial network (GAN) containing a generator and a discriminator. Obtain high-resolution sample images from the database and downsample them to obtain low-resolution sample images. Use the high-resolution and low-resolution sample images as samples to pre-train the GAN. The generator generates high-resolution images based on enhanced fusion features (i.e., performs image reconstruction). The discriminator evaluates the realism of the high-resolution images. At the same time, a multi-objective loss function combining adversarial loss, perceptual loss, and TV loss is used to jointly optimize the generator and discriminator, thereby guiding the generator to generate high-resolution images with realistic local details and global structure.

[0097] Specifically, a generative adversarial network is constructed, comprising a generator and a discriminator, including:

[0098] The generator's architecture includes an initial feature extraction layer, a deep residual network architecture (the core of the generator), a backbone convolutional layer, an upsampling module, and a post-processing layer. The initial feature extraction layer consists of a single convolutional layer used to extract initial features from the input low-resolution sample image, capturing low-level features. The deep residual network architecture consists of 16 concatenated Residual Dense Residual Blocks (RRDBs). Each RRDB is formed by stacking multiple convolutional layers and the LeakyReLU activation function (i.e., multiple convolutional layers + LeakyReLU combined and stacked), where each layer sets a residual... The differential connection is used for deep feature extraction and nonlinear transformation of low-level features. Through multiple residual dense blocks, multi-stage feature reconstruction is performed to complete high-quality feature map reconstruction (the number of channels and resolution remain unchanged). The upsampling module is used to map the feature map output by the deep residual network architecture to high resolution. The post-processing layer consists of two convolutional layers and a LeakyReLU activation function. The first convolutional layer is used to further refine and extract the multi-channel features after upsampling. The LeakyReLU activation function is used to normalize the values ​​corresponding to the channels to the range of [-1, 1]. The second convolutional layer is used to compress the number of normalized feature channels to 3 (RGB) and output the final high-resolution image (SR image).

[0099] The generator performs multi-stage feature reconstruction using a deep residual network architecture composed of multiple concatenated Residual Dense Blocks (RRDBs). Each RRDB is composed of multiple convolutional layers and LeakyReLU activation functions stacked together. The LeakyReLU activation function is expressed as follows:

[0100] ;

[0101] In the formula, For the LeakyReLU activation function, It is a constant. <0, The value can be 0.01 or 0.1, and is used to prevent the output of some neurons from being zero during training, thus maintaining the training effect of the network.

[0102] The discriminator employs a multi-scale discriminator, which consists of multiple independent branches, each responsible for processing features at different scales (resolutions). In this embodiment, three independent branches are used to process image features at the original scale, half-scale, and quarter-scale, respectively. Each independent branch has the same structure and parameters and shares parameters. Each independent branch contains an input layer, a deep convolutional layer, a fully connected layer, and an output layer. The input layer, composed of convolutional layers, performs preliminary feature extraction on the generated and real images, capturing low-level features of both types of images. The convolutional layers downsample by setting stride=2, reducing the resolution to half. The deep convolutional layer consists of cascaded discriminator blocks, set to eight blocks. Each discriminator block contains a convolutional layer (using a 3×3 or 4×4 convolutional kernel), a normalization layer (BatchNorm), and a LeakyReLU activation function. The convolutional layers in the first discriminator block use a stride of 2 for downsampling. In subsequent convolutional blocks, the stride is also set to 2 to progressively downsample the image, extracting deep features and gradually reducing the resolution to obtain a high-dimensional deep feature map. The fully connected layer module includes a 1024-neuron fully connected layer and uses the LeakyReLU activation function to non-linearly transform the features output by the deep convolutional layer, flattening the multidimensional feature map into a one-dimensional vector. The output layer consists of a fully connected layer (usually with one neuron) and a Sigmoid activation function, used to compress the output of the fully connected layer to the [0, 1] interval, generating a probability output. Each independent branch outputs a probability, and the probabilities from different scales are averaged to obtain the final probability result.

[0103] A predetermined number of real high-resolution sample images are obtained from a publicly available medical image database. The high-resolution sample images are downsampled and Gaussian noise (following a normal distribution and with a normalized standard deviation ranging from 0.01 to 0.1) is added to obtain low-resolution sample images. Each high-resolution sample image and its corresponding low-resolution sample image are used as a single training sample to construct a training set.

[0104] A set of training samples is extracted from the training set and used to train the generative adversarial network. For a single set of training samples, the low-resolution sample image and preset condition information (such as defining it as a high-resolution image) are used as input to the generator, and the generator outputs the corresponding high-resolution sample image (i.e., image reconstruction), so that the generator learns the process of converting low-resolution sample images into high-resolution sample images. The generated image and the real high-resolution sample image are subjected to multi-scale downsampling processing to obtain three sets of images at preset scales. The first set retains the original resolution, the second set is downsampled to 1 / 2 of the original resolution, the third set is downsampled to 1 / 4 of the original resolution, and then the three sets of images are processed. The generated image and three sets of real high-resolution sample images are input into the discriminator. The discriminator independently judges the generated image at each scale and outputs a probability value, which represents the probability that the generated image is a real image. This allows the discriminator to accurately distinguish the difference between the generated image and the real high-resolution sample images. Finally, the multi-target loss of the generative adversarial network is calculated at multiple scales, and the losses at multiple scales are aggregated. The aggregated loss is used to update the network parameters of the generator and discriminator through backpropagation and the Adam optimizer to achieve joint optimization of the generator and discriminator. This training process is executed iteratively. When the maximum number of training epochs is reached, the iteration stops, and the network parameters of the generator are frozen.

[0105] It should be noted that downsampling of high-resolution sample images can be performed using existing methods, such as max pooling (2×2 window, with stride=2 to achieve ×1 / 2 downsampling), which is a commonly used method and will not be described in detail here.

[0106] The multi-target loss of the generative adversarial network is calculated at multiple scales, and the losses at multiple scales are aggregated, specifically including:

[0107] Define a multi-objective loss function that includes adversarial loss, perceptual loss, and TV loss (Total Variation Loss). Specifically, the least squares loss has the property of smoothing gradients. Introducing the least squares loss function as an adversarial loss can improve the training stability and output quality of generative adversarial networks in image generation tasks. For the generator, it attempts to make the discriminator's judgment of the generated samples close to 1. The least squares loss function of the generator is:

[0108] ;

[0109] In the formula, Represents generator, Represents the discriminator. The input generator is the kth sample (i.e., the kth low-resolution sample image). Let M be the output probability of the discriminator for the generated sample, and M be the total number of samples.

[0110] The discriminator attempts to make the discrimination of real samples close to 1 and the discrimination of generated samples close to 0. The loss function of the discriminator is:

[0111] ;

[0112] In the formula, The k-th real sample (i.e., the corresponding k-th high-resolution sample image) is input to the discriminator.

[0113] To improve the feature extraction capability of the generator network, a perceptual loss function is used to measure the similarity of the feature space between generated samples and real samples. The perceptual loss function of the generator is:

[0114] ;

[0115] In the formula, This represents the feature representation of the VGG-19 network output. , Let the width and height be the width and height of the nth and mth feature maps in the VGG-19 network. The index of the column in the feature map (i.e., the width of the feature map) (position index on) The index of the row in the feature map (i.e., the feature map height) (position index on) For the generated high-resolution sample images, These are real, high-resolution sample images.

[0116] The TV loss function is introduced to eliminate interference signals generated during the image generation process. The TV loss function of the generator is as follows:

[0117] ;

[0118] In the formula, Indicates the midpoint of the generated high-resolution sample image Pixel value at that location, , This represents the column and row coordinates in the generated high-resolution sample image. The L1 norm represents the summation of the absolute values ​​of the differences between each pair of adjacent pixels. The TV loss function is used to penalize excessive gradient changes by measuring the changes in adjacent pixels in the image, thereby helping the generator avoid producing obvious noise or artifacts and making the generated high-resolution image smoother and more realistic.

[0119] The generator's total loss function is:

[0120] ;

[0121] In the formula, Weighting coefficients to counteract losses, The weighting coefficients for perceived loss, The weighting coefficients for TV regularization.

[0122] It should be noted that the VGG-19 network is a classic deep convolutional neural network (CNN) architecture. When used to extract image features, it does not require training. It can be used directly by loading ImageNet pre-trained weights and freezing the parameters. The ImageNet pre-trained weights were trained on ImageNet by the VGG authors and are publicly available, reusable standard weights. The use of the VGG-19 network is a common practice and will not be described in detail here.

[0123] Step 5: Extract shallow features from the low-resolution image through a preset convolutional layer, concatenate the global context features and shallow features in the channel dimension to obtain enhanced fusion features, input the enhanced fusion features into the trained generator, and use the generator to generate a high-resolution image based on the enhanced fusion features.

[0124] Specifically, shallow features of low-resolution images are extracted through pre-defined convolutional layers. Global context features and shallow features are then concatenated along the channel dimension to obtain enhanced fusion features, including:

[0125] low-resolution images Input a preset convolutional layer, extract its shallow features, and obtain , is represented as:

[0126] ;

[0127] In the formula, This represents the convolution operation corresponding to the preset convolutional layer; the global context features finally output by the Transformer encoder. Given a two-dimensional sequence, rearrange it into the form shown above. The matched two-dimensional feature map is represented as In the formula, This represents a transformation operation from a sequence to the shape of a two-dimensional feature map; it transforms shallow features... Global context features of the shape of a 2D feature map By splicing along the channel dimension, enhanced fusion features are obtained. , is represented as:

[0128] ;

[0129] In the formula, This indicates a concatenation operation along the channel dimension.

[0130] Will enhance fusion features The input is used in the trained generator to enhance the fusion features. Generate a high-resolution image, represented as , This is the generated high-resolution image.

[0131] Example 1:

[0132] High-resolution medical images were reconstructed using the traditional bicubic interpolation method (BICUBIC), the super-resolution convolutional neural network (SRCNN), and the method of this invention (OURS). BICUBIC reconstructs high-resolution medical images using bicubic interpolation, while SRCNN reconstructs high-resolution medical images using a pre-trained convolutional neural network.

[0133] Figure 6 The first row contains four low-resolution medical images to be reconstructed (i.e., four lung image samples: Sample1, Sample2, Sample3, and Sample4). Figure 6 The second row shows the reconstruction results corresponding to the cubic interpolation method (BICUBIC). Figure 6 The third row displays the reconstruction results corresponding to the Super-Resolution Convolutional Neural Network (SRCNN). Figure 6 The fourth row shows the reconstruction results corresponding to the method of the present invention. By comparing the processing effects of the four image samples in the figure under different super-resolution models, it can be seen that the results of the method of the present invention are significantly better than other methods (or models). Specifically, the image spot generated by the method of the present invention exhibits a more delicate imaging effect, and the spot has a clear boundary, indicating that it can better restore details and maintain the coherence of the overall structure. In contrast, BICUBIC, due to the limitations of traditional interpolation, results in blurred spots and loss of a large number of details. Although SRCNN improves the detail density through deep learning, local colors are blurred and the boundaries are rough, and noise or over-sharpening is introduced in the reconstruction.

[0134] The reconstruction results of each method were compared using two key metrics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), to demonstrate the significant advantages of the method of this invention in super-resolution reconstruction of lung medical images. The results are shown in Table 1 below:

[0135] Table 1

[0136]

[0137] As shown in Table 1, the PSNR and SSIM values ​​of the reconstructed images on the four lung samples by the method of the present invention are higher than those of traditional bilinear interpolation and the deep learning-based SRCNN model. In Sample 1, the PSNR of the method of the present invention (32.08 dB) is higher than that of SRCNN (29.43 dB) and BICUBIC (26.79 dB), indicating that the generated images are not only closer to real images, but also more in line with human visual perception in terms of structure, texture and brightness distribution, and can more accurately preserve the subtle structural features of lung tissue.

[0138] For the reconstruction results corresponding to each method, the PI value (perceptual index, the lower the value, the better the perceptual quality of the image reconstruction) is compared. Figure 7 As shown, the method of this invention exhibits the lowest perception index (PI) value in the reconstruction of four lung samples, significantly outperforming the BICUBIC and SRCNN models. This indicates that the lung images reconstructed by the method of this invention are closer to natural images, can more accurately present local lung details, and have a stronger visual realism. In contrast, although BICUBIC and SRCNN perform well on the RMSE index (the most commonly used objective indicator to measure the pixel-level numerical difference between the reconstructed image and the original high-resolution image), their reconstructed images are blurry and lack details; while the method of this invention significantly outperforms the perception index PI, and the reconstruction results have higher visual realism and clinical usability.

[0139] The steps in this invention can be adjusted, combined, or deleted according to actual needs.

[0140] The units in the device of the present invention can be merged, divided, or reduced according to actual needs.

[0141] In this invention, the terms "installation," "connection," "linking," and "fixing" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; "linking" can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of these terms in this invention according to the specific circumstances.

[0142] The shapes of the components in the accompanying drawings are schematic and may differ from their actual shapes. The drawings are only used to illustrate the principles of the present invention and are not intended to limit the present invention.

[0143] Although the invention has been disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and not intended to limit the application of the invention. The scope of protection of the invention is defined by the appended claims and may include various modifications, alterations, and equivalents made to the invention without departing from the scope and spirit of the invention.

Claims

1. A method for super-resolution reconstruction of medical images based on multimodal fusion, characterized in that, The method includes: Step 1: Obtain multimodal medical images from different data sources, preprocess and stitch the multimodal medical images to form standardized low-resolution images; Step 2: Set up an MSIMM module containing multiple convolutional kernels of different sizes, use the MSIMM module to extract multi-scale features of low-resolution images, and use intermediate-scale features as an intermediary to perform cross-scale modulation and fusion of multi-scale features. Step 3: Construct a Transformer encoder consisting of multiple identical stacked encoder layers. Each encoder layer includes a self-attention layer and a feedforward neural network. The feedforward neural network is composed of two linear transformation functions and one nonlinear activation function. The Transformer encoder is used to perform global feature modeling on cross-scale fused features, specifically including: Step 31: Convert the cross-scale fusion features into a sequence representation, and introduce positional encoding into the sequence to form the input features; Step 32: Iterate through each encoder layer in sequence. For a single encoder layer, first process the input features through a self-attention layer, and then perform residual connections and layer normalization. Then input the normalized features into the feedforward neural network, process them through two linear transformation functions and a nonlinear activation function, and then perform residual connections and layer normalization again. Step 33: A gating mechanism is introduced after each self-attention layer. The normalized features of the self-attention layer output are processed by the gating mechanism to obtain weight gating. The output features of a single encoder layer are then weighted and fused based on the weight gating. Step 34: Pass the weighted fused features to the next layer, and repeat the process from Step 32 to Step 33 until all encoder layers have completed the calculation, and finally output the first fused feature; Step 4: Construct a generative adversarial network (GAN) containing a generator and a discriminator. Obtain high-resolution sample images from the database and downsample them to obtain low-resolution sample images. Use the high-resolution sample images and low-resolution sample images as samples to pre-train the GAN. At the same time, use a multi-objective loss function that combines adversarial loss, perceptual loss and TV loss to jointly optimize the generator and discriminator. Step 5: Extract shallow features from the low-resolution image through a preset convolutional layer, concatenate the global context features and shallow features in the channel dimension to obtain enhanced fusion features, input the enhanced fusion features into the trained generator, and use the generator to generate a high-resolution image.

2. The medical image super-resolution reconstruction method based on multimodal fusion as described in claim 1, characterized in that, Step 1 specifically includes: Preprocessing of multimodal medical images from different data sources includes high-precision spatial registration, unified resolution and channel count, and normalization. Specifically, a channel duplication method is used to convert single-channel images into three-channel images, including: Single-channel image data is a two-dimensional matrix, represented as Where H is the image height and W is the image width; the single-channel image data is copied by copying the two-dimensional matrix three times along the channel dimension, and then the three copied channel data are concatenated to form a new three-channel image data, represented as... For each pixel (i,j) in a single-channel image, its pixel value is denoted as p. After conversion, the value of this pixel in all three channels is set to p, as shown in the expression: ； In the formula, c is the channel index; the preprocessed multimodal medical images are stitched and fused along the channel dimension to form a standardized low-resolution image to be input, denoted as c. .

3. The medical image super-resolution reconstruction method based on multimodal fusion as described in claim 2, characterized in that, Step 2 specifically includes: An MSIMM module with three parallel convolutional kernels (3×3, 5×5, and 7×7) is configured. First, a 3×3 convolutional layer is used to perform shallow feature extraction on the original low-resolution image to obtain the original input feature x. Then, the original input feature x is input into the MSIMM module, where the three parallel convolutional kernels (3×3, 5×5, and 7×7) are used to extract features, resulting in local detail features. Features of intermediate scale Global context features , is represented as: ； In the formula, Corresponding to a 3×3 convolution kernel, Corresponding to a 5×5 convolution kernel, Corresponding to a 7×7 convolution kernel; Features at intermediate scale Using element-wise multiplication as an intermediary, local detail features are respectively... and global context features Interactive modulation and fusion are represented as: ； In the formula, for and The fused features represent local details in the enhanced intermediate-scale features; for and The fused features represent the injection of global contextual information into the features at intermediate scales; symbols Represents element-wise multiplication; modulated and fused features and The concatenation is performed along the channel dimension, and then a 1×1 convolutional layer is used to compress the concatenated features back to the original number of channels to complete the fusion of multi-scale information, as shown below: ； ； In the formula, for and The fusion characteristics after splicing To compress the fusion features back to the original number of channels, This represents the splicing operation along the channel dimension. Represents a 1×1 convolutional layer; fusion features compressed back to the original number of channels. The original input features x are subjected to residual connections to obtain the final cross-scale fused features, represented as: ； In the formula, For the final cross-scale fusion features, C2 represents cross-scale fusion features. The number of channels, A matrix representing the real number field.

4. The medical image super-resolution reconstruction method based on multimodal fusion as described in claim 1, characterized in that, Step 31 specifically includes: Cross-scale fusion features Convert to sequence representation: ； In the formula, X represents the cross-scale fusion feature. The corresponding sequence, where n is the sequence length (n = H × W), and d is the feature dimension (d = C²). Represents sequence transformation; A positional code P is introduced into the sequence to form the input feature. , represented as Wherein, the position code P is represented as: ； ； In the formula, pos is the position index of the current feature in the sequence, and i is the dimension index in the position encoding; the position encoding P is constructed by alternating sine and cosine functions.

5. The medical image super-resolution reconstruction method based on multimodal fusion as described in claim 4, characterized in that, In step 32, the input features are processed through a self-attention layer, and residual connections and layer normalization are performed, specifically including: Features The input is fed into the self-attention layer in the current encoder layer. The self-attention layer consists of multiple self-attention modules. For a single self-attention module, the input features are first processed through different parameter matrices. The mapping is a set of queries, keys, and values, with each set of query, key, and value mappings forming a single header, represented as: ； In the formula, Q is the query matrix, K is the key matrix, and V is the value matrix. , , These are the learnable parameters corresponding to the query, key, and value matrices, respectively; then, based on the query, key, and value, the self-attention output corresponding to the current self-attention module is independently calculated, as follows: ； ； In the formula, This represents the self-attention output corresponding to the j-th self-attention module; finally, the self-attention outputs corresponding to all self-attention modules are concatenated together and mapped through a weight matrix to obtain the final output of the self-attention layer, represented as: ； In the formula, This is the weight matrix used to map the splicing result; Will Recorded as The final output of the self-attention layer Performing residual connections and layer normalization is represented as follows: ； In the formula, Z represents the intermediate representation of the current encoder layer after passing through the self-attention layer, residual connections, and layer normalization. Representation layer normalization.

6. The medical image super-resolution reconstruction method based on multimodal fusion as described in claim 5, characterized in that, Step 32 further includes: The normalized feature Z is input into the feedforward neural network in the current encoder layer and processed through two linear transformation functions and one nonlinear activation function, as follows: ； In the formula, , These are the weight matrices corresponding to the linear transformation functions of the first and second layers, respectively. , These are the bias vectors corresponding to the linear transformation functions of the first and second layers, respectively. The ReLU activation function is expressed as follows: ； In the formula, Activation function The independent variable; the output of the feedforward neural network Performing residual connections and layer normalization is represented as follows: ； In the formula, These are the features in the current encoder layer after passing through the feedforward neural network, residual connections, and layer normalization.

7. The medical image super-resolution reconstruction method based on multimodal fusion as described in claim 6, characterized in that, Step 33 specifically includes: We use a gating mechanism to process the normalized feature Z output from the self-attention layer to obtain the weight gating: ； In the formula, , Here are the learnable weight matrix and bias vector corresponding to the gating mechanism. The Sigmoid activation function is expressed as follows: ； In the formula, Activation function The corresponding independent variables; features of the feedforward neural network output after residual connections and layer normalization based on weight gating. Dynamic, adaptive weighted fusion is performed, represented as: ； In the formula, G and (1-G) represent the weighted gating that allows and prohibits passage, respectively.

8. The medical image super-resolution reconstruction method based on multimodal fusion as described in claim 1, characterized in that, Step 4 specifically includes: A predetermined number of real high-resolution sample images are obtained from a publicly available medical image database. The high-resolution sample images are downsampled and Gaussian noise is added to obtain low-resolution sample images. Each high-resolution sample image and its corresponding low-resolution sample image are used as a single training sample to construct a training set. Training samples are extracted from the training set to train the generative adversarial network (GAN). For a single training sample, the low-resolution sample image and preset condition information are used as input to the generator, which outputs the corresponding high-resolution sample image. The generated image and the real high-resolution sample image are then subjected to multi-scale downsampling to obtain three sets of images at preset scales. The resolution of the first set remains unchanged, the second set is downsampled to 1 / 2 of the original resolution, and the third set is downsampled to 1 / 4 of the original resolution. The three sets of generated images and the three sets of real high-resolution sample images are then input into the discriminator, which independently discriminates the generated image at each scale and outputs a probability value. Finally, the multi-target loss of the GAN is calculated at multiple scales, and the losses at multiple scales are aggregated. The aggregated loss is used to update the network parameters of the generator and discriminator through backpropagation and the Adam optimizer. This training process is executed iteratively, and the iteration stops when the maximum number of training epochs is reached, and the network parameters of the generator are frozen.

9. The medical image super-resolution reconstruction method based on multimodal fusion as described in claim 8, characterized in that, The process of calculating the multi-target loss of the generative adversarial network at multiple scales and aggregating the losses at multiple scales specifically includes: Define a multi-objective loss function that includes adversarial loss, perceptual loss, and TV loss, where the least squares loss function of the generator is: ； In the formula, Represents generator, Represents the discriminator. For the k-th sample input to the generator, Let M be the output probability of the discriminator for each generated sample, and M be the total number of samples. The loss function of the discriminator is: ； In the formula, The k-th real sample is input to the discriminator; The perceptual loss function is used to measure the feature space similarity between generated samples and real samples. The perceptual loss function of the generator is: ； In the formula, This represents the feature representation of the VGG-19 network output. , Let the width and height be the width and height of the nth and mth feature maps in the VGG-19 network. For the column index in the feature map, This is the index of the row in the feature map. For the generated high-resolution sample images, These are real, high-resolution sample images; The TV loss function is introduced to eliminate interference signals generated during the image generation process. The TV loss function of the generator is as follows: ； In the formula, Indicates the midpoint of the generated high-resolution sample image Pixel value at that location, , This represents the column and row coordinates in the generated high-resolution sample image. It is an L1 norm; The generator's total loss function is: ； In the formula, Weighting coefficients to counteract losses, The weighting coefficients for perceived loss, The weighting coefficients for TV regularization.

10. The medical image super-resolution reconstruction method based on multimodal fusion as described in claim 2, characterized in that, Step 5 specifically includes: low-resolution images Input a preset convolutional layer, extract its shallow features, and obtain , is represented as: ； In the formula, This represents the convolution operation corresponding to the preset convolutional layer; the global context features finally output by the Transformer encoder. Given a two-dimensional sequence, rearrange it into the form shown above. The matched two-dimensional feature map is represented as In the formula, This represents a transformation operation from a sequence to the shape of a two-dimensional feature map; it transforms shallow features... Global context features of the shape of a 2D feature map By stitching along the channel dimension, enhanced fusion features are obtained. , is represented as: ； In the formula, This represents a concatenation operation along the channel dimension; it will enhance the fusion features. The input is used in the trained generator to enhance the fusion features. Generate a high-resolution image, represented as , This is the generated high-resolution image.

Images