Brain MRI (Magnetic Resonance Imaging) super-resolution cross fusion adaptive feature enhancement Transform method

Abstract

The invention discloses a brain MRI (Magnetic Resonance Imaging) super-resolution cross fusion adaptive feature enhancement Transform method, and relates to the technical field of image restoration in computer vision of artificial intelligence. According to the method, the utilization of high-frequency information is remarkably improved, high-frequency detail information is effectively supplemented by designing a high-frequency enhancement block, the limitation that a traditional Transform model tends to pay attention to low-frequency information is overcome, texture details and edge quality of a magnetic resonance image are remarkably improved, a reconstruction result is clearer and more natural, and the reconstruction efficiency is improved. According to the method, global information and local information are efficiently fused, a mixed attention block is introduced, a displacement rectangular window attention module and a channel attention module are combined, the local fitting capacity is enhanced while the global long-distance dependency relationship is captured, an ASOAB block is designed, and the method has the advantages that the local fitting capacity is improved; the main function of the ASOAB block is to adaptively reserve key information, remove redundant information in the reconstruction process and enhance direct interaction of features between different windows.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration in computer vision of artificial intelligence, and specifically to a brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method. Background Technique

[0002] High-resolution brain MRI images play a crucial role in the diagnosis and treatment of brain diseases. However, due to the inherent limitations of the imaging mechanism, MRI images are often interfered by scattering and noise. In addition, the long scanning process and the limitation of the patient's body position will also cause motion artifacts and distortion in the images. To solve these problems, super-resolution technology has gradually become a research hotspot. The SR technology aims to reconstruct high-resolution (HR) images from low-resolution (LR) images to make up for the deficiencies in the imaging process.

[0003] Traditional SR reconstruction methods mainly rely on interpolation and learning-based techniques. However, in the face of different scaling ratios, perspective changes, and complex lighting conditions, these methods have poor adaptability and are difficult to meet the needs of practical applications. In recent years, deep learning-based natural image super-resolution (Single Image Super-Resolution, SISR) methods have made remarkable progress in overcoming the limitations of traditional methods. Inspired by this, deep learning-based MRI medical image super-resolution reconstruction methods have gradually increased. Some networks enhance the performance of convolutional neural networks by introducing residual connections and dense connections; others capture long-range dependencies and context relationships through attention mechanisms to further optimize the model performance. These advanced methods have significantly improved the effect of MRI super-resolution reconstruction and provided more potential solutions for practical applications.

[0004] Although the attention mechanism performs well in capturing long-range dependencies, existing studies have shown that the attention mechanism tends to focus more on low-frequency information. Most current work focuses on the design of the Transformer module, ignoring the effective utilization of high-frequency information and lacking an efficient fusion strategy for high-frequency and low-frequency information. This deficiency leads to problems such as texture distortion, detail blurring, and uneven reconstruction results in the reconstruction. In addition, some networks adopt a window-based attention mechanism to more efficiently capture long-range dependencies, but they lack direct interaction between adjacent window features. To solve this problem, some studies have introduced an overlapping window attention mechanism to enhance the interaction between adjacent windows. However, overlapping windows not only calculate regions with insufficient information in dense attention calculations but also introduce redundant or irrelevant features, resulting in noise interference and performance degradation.

[0005] Based on this, the following issues will be specifically addressed:

[0006] (1) Existing methods focus more on low-frequency information and ignore the use of high-frequency information. This results in the details and edge information in the reconstructed image not being clear enough, affecting the overall image quality and diagnostic accuracy. Although low-frequency information contains the main structure of the image, high-frequency information is the key to understanding image details and texture.

[0007] (2) Existing methods lack effective extraction and integration of low-frequency information and high-frequency information. An effective information integration strategy is the key to improving the quality of MRI image reconstruction. It is necessary to design more sophisticated algorithms to process these two types of information simultaneously in order to restore more accurate and natural images.

[0008] (3) It is difficult to remove redundant information in the image during reconstruction and it is difficult to enhance the direct interaction between different windows in the window-based attention module. In the reconstruction process of MRI images, existing algorithms are often unable to effectively distinguish important information from unnecessary repetitive information. This redundancy not only consumes additional computing resources, but may also affect the quality of the final image and the accuracy of the analysis. In addition, it is also an important problem that it is difficult to enhance the direct interaction between different windows in the window-based attention module. This module usually processes information within a local window and ignores the association between windows, resulting in the emergence of information islands, which limits the integration and utilization of global information. This limitation hinders the model's ability to capture a wider range of contextual information, which in turn affects the coherence and detail representation of the reconstructed image. Summary of the invention

[0009] The purpose of the present invention is to provide a brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method, which obtains a super-resolution image by processing a low-resolution image, and designs and trains a model through a deep learning method to get rid of the image quality degradation caused by hardware limitations, so as to solve the technical problems raised in the background technology.

[0010] To achieve the above object, the present invention provides the following technical solution: a brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method, comprising at least the following steps:

[0011] S1: Slice the 3D image in the original dataset and save the middle part;

[0012] S2: Downsample the sliced ​​data by different scale factors to make an LR-HR paired dataset, and divide the dataset into training set, validation set, and test set;

[0013] S3: Design the corresponding modules in the CAFET network architecture;

[0014] S4: Design the loss function and Adam optimizer for optimizing the network 1 ;

[0015] S5: Feed the partitioned data into the model, conduct training, validation, and testing, and evaluate the model according to corresponding metrics. The model training is completed.

[0016] Furthermore, the S1 at least includes the following steps: Read the nii files in the original dataset, slice them along the x-axis or y-axis or z-axis, convert them into 2D pictures, save them in PNG format, and only keep the middle pictures.

[0017] Furthermore, the S2 at least includes the following steps:

[0018] Randomly shuffle the pictures in the original dataset made after slicing, and use bicubic downsampling to perform ×2 and ×4 times downsampling on them to obtain low-resolution images to simulate degradation in the real world, and save them in PNG format;

[0019] Finally, make paired low-resolution and high-resolution data pairs. Subsequently, divide the shuffled paired data according to the ratio of 7:2:1 into a training set, a validation set, and a test set.

[0020] Furthermore, the CAFET network architecture in the S3 at least includes shallow feature extraction, deep feature extraction, and image reconstruction;

[0021] The input low-resolution image First, obtain shallow features through shallow feature extraction where represents the number of input channels, and C represents the number of intermediate feature channels;

[0022] Next, the shallow features pass through N residual enhanced attention fusion blocks (Residual Enhanced Attention Fusion Blocks, REAFB) and a 3×3 convolutional layer to extract deep features F d ;

[0023] After that, through global residual connection, the shallow features F s are fused with the deep features F d Finally, a high-resolution image is generated through the image reconstruction module;

[0024] The residual enhanced attention fusion block (REAFB) consists of a hybrid fusion module (HFB), an adaptive sparse overlapping attention module (ASOAB), and a 3×3 convolutional layer with global residual connection;

[0025] In the image reconstruction module, the Pixel Shuffle method is adopted to upsample the fused features to generate a high-resolution image.

[0026] Furthermore, the hybrid fusion module consists of a High-Frequency Enhancement Block (HFEB), a Hybrid Attention Block (HAB), and a Large Window Fusion Attention Block (LWFAB);

[0027] The High-Frequency Enhancement Block is used to effectively capture edge detail information. After the input features pass through the first Layernorm layer, the feature F is obtained. in , and then the feature F in is divided into two parts along the channel dimension and sent to two branches respectively for high-frequency feature extraction and enhancement to obtain the F M feature and the F N feature;

[0028] For the F M feature, local features are obtained using a 3×3 convolution and the GELU activation function.

[0029] For the F N feature, first, the high-frequency part in the feature is obtained using max pooling, then the high-frequency information is enhanced through 1×1 convolution on the high-frequency information and the GELU activation, and finally, the information of the two branches is fused deeply by concatenating along the channel dimension and using 1×1 convolution.

[0030] Furthermore, in the hybrid attention block, the Shifted Rectangle Window Attention Module is used in parallel with the Channel Attention Mechanism (CAM) to obtain global information and enhance the representation ability of the network. The Shifted Rectangle Window Attention Module is the SRWAB module;

[0031] The SRWAB module consists of the Shifted Rectangle Window Attention and a Multi-Layer Perceptron (MLP). To more effectively capture information in different regions, the SRWAB module is divided into vertical window attention and horizontal window attention;

[0032] The vertical window attention is when the window height r h is greater than the width r w , which is simply referred to as HSAB;

[0033] When the window width r w is greater than the height r h , it is simply referred to as WSAB;

[0034] In the specific implementation, using multiple attention heads, HSAB and WSAB are respectively and parallelly assigned to the same number of attention heads for independent calculation for a given input F in ;

[0035] First, it is divided into non-overlapping rectangular windows;

[0036] For each rectangular window feature its projection is obtained to get query, key, value (F Q , F K , F V ) vectors, and then, rectangular window attention calculation is performed on them:

[0037]

[0038] where d represents the channel dimension, B represents the dynamic position encoding, represents the transposed vector of the F K vector;

[0039] Subsequently, attention operations are performed on all windows, and the final feature X att ;

[0040] In addition, a channel attention module, namely CAB, is embedded in the model to enhance the expression ability of visual features;

[0041] To avoid conflicts between the CAB and SRWAB modules that may occur during network optimization, the output of the CAB is multiplied by a small scaling factor α to ensure the stability and consistency of the model;

[0042] Finally, the output of the scaled channel attention is added to the output of the SRWAB module to obtain the output feature F att .

[0043] Furthermore, the construction of the large window fusion attention block includes at least the following steps:

[0044] The output features F in and F att of the high-frequency enhancement block and the hybrid attention block are respectively divided into M non-overlapping windows;

[0045] Then, they are encoded through a linear layer to generate two query vectors;

[0046] Subsequently, these two query vectors are initially fused to obtain the final query vector Q;

[0047] At the same time, the output F att of the hybrid attention block is subjected to the same window division operation, and then the features are projected and their number of channels is compressed to obtain the key matrix and the value matrix where r is the token reduction factor, M is the number of samples, and L 2 is the window spatial dimension;

[0048] Next, perform permutation operations on K and V to generate a permuted key matrix and value matrix

[0049] After that, send the fused feature F out into the feed-forward network to fully aggregate the features. Use 1×1 convolution to aggregate channel information, 3×3 convolution to aggregate spatial information, and adopt a gating mechanism to enhance information encoding to obtain a fully fused feature

[0050] Furthermore, the adaptive sparse overlapping attention module consists of an adaptive sparse overlapping attention mechanism and a multi-layer perceptron (MLP);

[0051] The adaptive sparse overlapping attention module is implemented by expanding the input feature into blocks of different window sizes and performing shifted window attention calculations with a kernel size of M0 and a stride of M on these blocks of different window sizes;

[0052] The input feature is projected into Q, K, and V vectors;

[0053] Among them, the query vector Q is divided into non-overlapping windows of size R×R;

[0054] While the K and V vectors are divided into overlapping windows of size R 0 ×R 0 ;

[0055] At the same time, to ensure the consistency of the window size, zero-padding operations of size are used for the K and V vectors. γ is a coefficient used to control the overlapping size of the windows, and γ = 0.5;

[0056] Among them, the calculation of R 0 is as follows:

[0057] R 0 =(1 + γ)R (2)

[0058] In addition, to ensure the sparsity of the attention mechanism, an attention mechanism based on the RELU activation function is introduced, aiming to filter out features with relatively low matching scores between the query and the key;

[0059] And square the RELU output to enhance the positive correlation;

[0060] Finally, to prevent excessive sparsity, a dual-branch attention structure is proposed to ensure that the attention mechanism has both sparsity and density;

[0061] To adaptively fuse these two branches and thus retain key information to the greatest extent, adaptive attention weights are introduced to impose constraints on the sparse branch, enabling the network to dynamically adjust and retain key information;

[0062] The final attention calculation result X final is expressed as:

[0063] X final = Att d + w 0 · Att s

[0064] where w 0 is the adaptive weight used to balance the contributions of the sparse and dense attention branches, Att d is the calculation result of the Softmax branch, and Att s is the calculation result of the RELU branch.

[0065] Furthermore, the S4 at least includes the following steps:

[0066] L 1 The loss function is used to evaluate the difference between the predicted value and the true value;

[0067] In the super-resolution task, the L 1 loss function can help the model learn to predict the pixel values in the high-resolution image, minimizing the absolute error between the predicted high-resolution image and the true high-resolution image. Its value ranges from 0 to 1, where the smaller the value, the lower the pixel loss of the image reconstructed by the network, which means a smaller error;

[0068] The advantages of the Adam optimizer's momentum and adaptive learning rate can help the model converge faster and improve the training efficiency;

[0069] In the super-resolution task, the Adam optimizer can effectively adjust the model parameters, enabling the model to better learn the mapping relationship from the low-resolution image to the high-resolution image.

[0070] Furthermore, the principle of evaluating the model by the corresponding metrics in the S5 is as follows:

[0071] Peak signal-to-noise ratio (PSNR) is used to measure the similarity between the reconstructed image and the original image;

[0072] The peak signal-to-noise ratio evaluates the image quality by comparing the pixel values between the original image and the reconstructed image. The calculation formula of PSNR is as follows:

[0073]

[0074] Among them, MAX is the maximum value of the image pixel value. For example, for an 8-bit grayscale image, the maximum value is usually 255. MSE (Mean Squared Error) is the mean squared error, representing the sum of the squares of the differences between each pixel of the original image and the reconstructed image, as shown in the formula:

[0075]

[0076] Where M is the number of samples, y i is the actual observed value, is the corresponding predicted value;

[0077] The peak signal-to-noise ratio only cares about the difference between the input and reconstructed pixels. Generally speaking, the higher the peak signal-to-noise ratio value, the smaller the difference between the reconstructed image and the original image, and the higher the reconstruction quality;

[0078] The Structural Similarity Index (SSI) aims to comprehensively consider luminance, contrast, and structural information to more comprehensively evaluate image quality. Refer to the following formula:

[0079]

[0080] Among them, and μ I represent the mean luminance of the generated image and the real image respectively, and σ I represent the luminance variances of the generated image and the real image respectively, represents the covariance of luminance, C 1 and C 2 is a small constant used to avoid the denominator being zero;

[0081] When the value of the Structural Similarity Index (SSI) is close to 1, it indicates that the structural similarity between the reconstructed image and the original image is relatively high and the quality is good;

[0082] The structural similarity is often used together with other evaluation metrics (such as PSNR) to comprehensively evaluate the performance of the super-resolution algorithm.

[0083] Compared with the prior art, the beneficial effects of the present invention are:

[0084] 1. The significant improvement in the utilization of high-frequency information of the present invention. By designing a High-Frequency Enhancement Block (HFEB), the present invention effectively supplements high-frequency detail information, overcomes the limitation that traditional Transformer models tend to focus on low-frequency information. This design significantly improves the texture details and edge quality of magnetic resonance images, making the reconstruction results clearer and more natural, thereby enhancing the presentation effect of key regions for disease diagnosis;

[0085] 2. In the present invention, for the efficient fusion of global and local information, a Hybrid Attention Block (HAB) is introduced, which combines a Shifted Rectangle Window Attention Module and a Channel Attention Module. While capturing long-range global dependencies, it enhances the local fitting ability. In addition, the Large Window Fusion Attention Block effectively integrates the feature information extracted from the High-Frequency Enhancement Block and the Hybrid Attention Block by expanding the receptive field, ensuring the structural consistency of the reconstructed image, significantly reducing the computational complexity at the same time, and designing an ASOAB block. The main role of the ASOAB block is to adaptively retain key information, remove redundant information during the reconstruction process, and enhance the direct interaction of features between different windows;

[0086] 3. For the effective removal of redundant information and the improvement of computational efficiency, the proposed Adaptive Sparse Overlapping Attention Module enhances the direct interaction between windows through the dynamic fusion of sparse and dense attention, eliminating redundant or irrelevant information in the traditional overlapping window mechanism. This design not only reduces noise interference but also optimizes the utilization efficiency of computing resources, ensuring the scalability of the model when processing large-scale data;

[0087] 4. For enhancing the anti-distortion and detail retention capabilities, the present invention integrates the complementary features of CNN and Transformer, which can not only capture global context relationships but also retain delicate local details. This fusion design effectively avoids image distortion in traditional methods under conditions such as scale changes, perspective complexity, and noise interference, ensuring the high fidelity of the reconstructed MRI images;

[0088] 5. For comprehensively surpassing experimental performance, the experimental results on the BraTs and IXI datasets show that the present invention has achieved significant performance improvements in the MR image reconstruction tasks of different anatomical structures. Compared with the current state-of-the-art technical methods, under the ×2 upsampling factor of the BraTs dataset, the PSNR has increased by 2.4 dB. At the same time, the reconstructed images have reached the leading level in the industry in terms of texture details, global consistency, and key lesion regions, providing more reliable technical support for practical clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0090] Figure 1 It is the overall network implementation flowchart provided by the present invention;

[0091] Figure 2Schematic diagram of the CAFET network architecture provided by the present invention;

[0092] Figure 3 Schematic diagram of the LWFAB module provided by the present invention;

[0093] Figure 4 Schematic diagram of the ASOAB module provided by the present invention;

[0094] Figure 5 Schematic diagram of the test effect provided by the present invention. Detailed implementation manners

[0095] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0096] The method disclosed by the present invention is not limited to brain MRI images and can be applied to different implementation fields and environments in different embodiments;

[0097] When applied to brain MRI images, the following ideas are proposed:

[0098] High-resolution magnetic resonance imaging (MRI) is a core tool for doctors to diagnose and treat brain diseases, and its quality has a direct impact on medical decisions. In recent years, the Transformer method has shown great potential in single-image super-resolution reconstruction (SISR) of MRI by capturing long-range dependencies. However, the existing Transformer-based MRI reconstruction methods still have the following technical bottlenecks: (1) strong dependence on low-frequency information and failure to fully utilize high-frequency information, thus affecting the restoration of image details; (2) lack of an effective integration mechanism for low-frequency and high-frequency information, resulting in disharmony between the structure and texture of the reconstruction results; (3) difficulty in effectively removing redundant information in the image, resulting in low information utilization efficiency and limiting the improvement of model performance.

[0099] To solve the above key problems, an innovative Cross-fusion Adaptive Feature Enhancement Transformer (CAFET) is proposed, aiming to fully explore and utilize the potential of high-frequency information, improve the interaction between global and local features, and at the same time improve the ability to remove redundant information, so as to achieve higher-quality MRI image reconstruction. The present invention not only effectively solves the core bottlenecks in the existing MRI reconstruction technology, but also shows significant advantages in terms of image quality, model efficiency, and practical applicability, providing an innovative and efficient solution for high-resolution medical image reconstruction.

[0100] Please refer to Figure 1 , a cross-fusion adaptive feature enhancement Transformer method for brain MRI super-resolution, which at least includes the following steps:

[0101] S1: Slice the 3D images in the original dataset and save the middle part;

[0102] S2: Downsample the sliced data with different scale factors respectively to form an LR-HR paired dataset, and divide the dataset into a training set, a validation set, and a test set;

[0103] S3: Design the corresponding modules in the CAFET network architecture;

[0104] S4: Design the L 1 loss function and Adam optimizer for optimizing the network;

[0105] S5: Feed the divided data into the model for training, validation, and testing, and evaluate the model according to the corresponding metrics. The model training is completed.

[0106] S1 includes at least the following steps: Read the nii file in the original dataset, slice it along the x-axis or y-axis or z-axis to turn it into 2D pictures, save them in PNG format and only keep the middle pictures.

[0107] S2 includes at least the following steps:

[0108] Randomly shuffle the pictures in the original dataset made after slicing, and use bicubic downsampling to perform ×2 and ×4 times downsampling on them to obtain low-resolution images to simulate the degradation in the real world, and save them in PNG format;

[0109] Finally, form paired low-resolution and high-resolution data pairs. Subsequently, divide the shuffled paired data according to the ratio of 7:2:1, and divide the dataset into a training set, a validation set, and a test set.

[0110] As Figure 2 shown, the CAFET network architecture in S3 includes at least shallow feature extraction, deep feature extraction, and image reconstruction;

[0111] The input low-resolution image First passes through shallow feature extraction to obtain shallow features where represents the number of input channels, and C represents the number of intermediate feature channels;

[0112] Next, the shallow features pass through N residual enhanced attention fusion blocks (Residual Enhanced Attention Fusion Blocks, REAFB) and a 3×3 convolutional layer to extract deep features F d ;

[0113] After that, through global residual connection, the shallow features Fs Fused with the deep feature F d and finally, a high-resolution image is generated through the image reconstruction module;

[0114] The Residual Enhanced Attention Fusion Block (REAFB) consists of a Hybrid Fusion Block (HFB), an Adaptive Sparse Overlapping Attention Block (ASOAB), and a 3×3 convolutional layer with a global residual connection;

[0115] In the image reconstruction module, the Pixel Shuffle method is used to upsample the fused features to generate a high-resolution image.

[0116] The Hybrid Fusion Block consists of a High-Frequency Enhancement Block (HFEB), a Hybrid Attention Block (HAB), and a Large Window Fusion Attention Block (LWFAB);

[0117] The High-Frequency Enhancement Block is used to effectively capture edge detail information. After the input feature passes through the first Layernorm layer, the feature F in is obtained. Subsequently, the feature F in is divided into two parts along the channel dimension and sent to two branches respectively for high-frequency feature extraction and enhancement to obtain the F M feature and the F N feature;

[0118] For the F M feature, a 3×3 convolution and a GELU activation function are used to obtain local features

[0119] For the F N feature, first, the high-frequency part of the feature is obtained by max pooling, then the high-frequency information is enhanced by 1×1 convolution and GELU activation on the high-frequency information, and finally, the information of the two branches is connected along the channel dimension and depth-fused using 1×1 convolution.

[0120] In the Hybrid Attention Block, the Shifted Rectangle Window Attention Module is used to parallel the Channel Attention Mechanism (CAM) to obtain global information and enhance the representation ability of the network. The Shifted Rectangle Window Attention Module is the SRWAB module;

[0121] The SRWAB module consists of Shifted Rectangle Window Attention and a Multi-Layer Perceptron (MLP). To more effectively capture information in different regions, the SRWAB module is divided into Vertical Window Attention and Horizontal Window Attention;

[0122] The Vertical Window Attention is when the window height r h is greater than the width r w , which is simply called HSAB;

[0123] When the window width rw Greater than height r h When it is, it is simply called WSAB;

[0124] In a specific implementation, using multiple attention heads, HSAB and WSAB are respectively and parallelly assigned to the same number of attention heads for independent calculation. For a given input F in ;

[0125] First, divide it into non-overlapping rectangular windows;

[0126] For each rectangular window feature Project it to obtain query, key, value (F Q , F K , F V ) vectors, and then perform rectangular window attention calculation on it:

[0127]

[0128] where d represents the channel dimension, B represents the dynamic position encoding, and represents the transpose of the F K vector;

[0129] Subsequently, perform attention operations on all windows and obtain the final feature X through the MLP layer att ;

[0130] In addition, a channel attention module is embedded in the model to enhance the expression ability of visual features. The channel attention module is the CAB;

[0131] To avoid conflicts between the CAB and SRWAB modules that may occur during network optimization, a small scaling factor α is multiplied by the output of the CAB to ensure the stability and consistency of the model;

[0132] Finally, add the scaled channel attention output to the output of the SRWAB module to obtain the output feature F of the hybrid attention module att .

[0133] As Figure 3 shown, the construction of the large window fusion attention block includes at least the following steps:

[0134] Divide the output features F in and F att of the high-frequency enhancement block and the hybrid attention block into M non-overlapping windows respectively;

[0135] Then, encode them through a linear layer to generate two query vectors;

[0136] Subsequently, preliminarily fuse these two query vectors to obtain the final query vector Q;

[0137] Meanwhile, for the output F of the hybrid attention block att the same window partitioning operation is performed, and then the features are projected and their number of channels is compressed to obtain the key matrix and the value matrix where r is the token reduction factor, M is the number of samples, and L 2 is the window spatial dimension;

[0138] Next, permutation operations are performed on K and V to generate the permuted key matrix and the value matrix

[0139] After that, the fused feature F out is fed into the feed-forward network to fully aggregate the features. 1×1 convolution is used to aggregate channel information, 3×3 convolution is used to aggregate spatial information, and a gating mechanism is adopted to enhance information encoding to obtain the fully fused feature

[0140] The adaptive sparse overlapping attention module consists of an adaptive sparse overlapping attention mechanism and a multi-layer perceptron (MLP), and its structure is as Figure 4 shown;

[0141] The adaptive sparse overlapping attention module is implemented by expanding the input features into blocks of different window sizes and performing shifted window attention calculations with a kernel size of M0 and a stride of M on these blocks of different window sizes;

[0142] The input features are projected into Q, K, and V vectors;

[0143] where the query vector Q is divided into non-overlapping windows of size R×R;

[0144] while the K and V vectors are divided into overlapping windows of size R 0 ×R 0 ;

[0145] Meanwhile, to ensure the consistency of the window size, zero-padding operations of size are used for the K and V vectors. γ is a coefficient used to control the overlapping size of the windows, and γ = 0.5;

[0146] where the calculation of R 0 is as follows:

[0147] R 0 = (1 + γ)R (2)

[0148] In addition, to ensure the sparsity of the attention mechanism, an attention mechanism based on the RELU activation function is introduced, aiming to filter out features with relatively low matching scores between the query and the key;

[0149] And the output of RELU is squared to enhance the positive correlation;

[0150] Finally, to prevent excessive sparsity, a dual-branch attention structure is proposed to ensure that the attention mechanism has both sparsity and density;

[0151] To adaptively fuse these two branches and thus retain key information to the greatest extent, adaptive attention weights are introduced to impose constraints on the sparse branch, enabling the network to dynamically adjust and retain key information;

[0152] The final attention calculation result X final is expressed as:

[0153] X final = Att d + w 0 · Att s

[0154] where w 0 is the adaptive weight used to balance the contributions of the sparse and dense attention branches, Att d is the calculation result of the Softmax branch, and Att s is the calculation result of the RELU branch.

[0155] S4 includes at least the following steps:

[0156] L 1 The loss function is used to evaluate the difference between the predicted value and the true value;

[0157] In the super-resolution task, the L 1 loss function can help the model learn to predict the pixel values in the high-resolution image, minimizing the absolute error between the predicted high-resolution image and the true high-resolution image. Its value ranges from 0 to 1, where a smaller value indicates a lower pixel loss in the image reconstructed by the network, meaning a smaller error;

[0158] The Adam optimizer combines the advantages of momentum and adaptive learning rate, which can help the model converge faster and improve the training efficiency;

[0159] In the super-resolution task, the Adam optimizer can effectively adjust the model parameters, enabling the model to better learn the mapping relationship from the low-resolution image to the high-resolution image.

[0160] The principle of evaluating the model by the corresponding metrics in S5 is as follows:

[0161] The Peak Signal-to-Noise Ratio (PSNR) is used to measure the similarity between the reconstructed image and the original image;

[0162] The Peak Signal-to-Noise Ratio evaluates the image quality by comparing the pixel values between the original image and the reconstructed image. The calculation formula of PSNR is as follows:

[0163]

[0164] Where MAX is the maximum value of the image pixel values. For example, for an 8-bit grayscale image, the maximum value is usually 255. MSE (Mean Squared Error) is the mean squared error, representing the sum of the squares of the differences between each pixel of the original image and the reconstructed image, as shown in the formula:

[0165]

[0166] Where M is the number of samples, y i is the actual observed value, is the corresponding predicted value;

[0167] The Peak Signal-to-Noise Ratio only cares about the difference between the input and reconstructed pixels. Generally speaking, the higher the Peak Signal-to-Noise Ratio value, the smaller the difference between the reconstructed image and the original image, and the higher the reconstruction quality;

[0168] The Structural Similarity Index (SSI) aims to comprehensively consider brightness, contrast, and structural information to more comprehensively evaluate the image quality. Refer to the following formula:

[0169]

[0170] Where, and μ I represent the mean brightness of the generated image and the real image respectively, and σ I represent the brightness variances of the generated image and the real image respectively, represents the covariance of brightness, C 1 and C 2 is a small constant used to avoid the denominator being zero;

[0171] If the value of the Structural Similarity Index (SSI) is close to 1, it indicates a high structural similarity between the reconstructed image and the original image, and good quality;

[0172] The Structural Similarity is often used together with other evaluation metrics (such as PSNR) to comprehensively evaluate the performance of the super-resolution algorithm.

[0173] To train and test the network architecture of the present invention and verify the super-resolution performance of the proposed network on different MRI weighted images, 10,228 T1-weighted image slices were randomly selected from the BraTS2019 brain tumor segmentation dataset, and the resolution of all slices was 240×240. The dataset was divided into a training set, a validation set, and a test set in a ratio of 7:2:1. In addition, to evaluate the reconstruction ability of the model on T2-weighted images, 8,925 slices were randomly selected from the T2-weighted images of the IXI dataset, with a resolution of 256×256. To meet the requirements of network training, bicubic downsampling was performed on the high-resolution images to generate paired data with ×2 and ×4 downsampling for the training and testing of the super-resolution task of the model.

[0174] The CAFET method was compared with multiple brain MRI super-resolution methods. The PSNR and SSIM measurement results at ×2 and ×4 upsampling factors on the IXI and BraTs datasets are shown in Table 1, and the effect comparison diagrams are as Figure 5 shown, presenting the super-resolution (SR) reconstruction results at 2× and 4× upsampling. The first and second rows correspond to the 2× upsampling reconstruction effects of the T1 and T2 modalities respectively, and the third and fourth rows show the 4× upsampling reconstruction effects of the T1 and T2 modalities. To more intuitively compare the reconstruction performance of each method, error maps are also provided. The darker the color in the error map, the smaller the reconstruction error, that is, the better the recovery effect, and vice versa. Refer to Figure 5 and Table 1.

[0175] Table 1 Quantitative comparison of the PSNR and SSIM metrics with the current best method at *2 and *4 upsampling factors on the BraTs and IXI datasets

[0176]

[0177]

[0178] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be regarded as limiting the claimed rights.

Claims

1. A brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method, characterized by: At least the following steps are included: S1: Slice the 3D image in the original dataset and save the middle part; S2: Downsample the sliced ​​data by different scale factors to make an LR-HR paired dataset, and divide the dataset into training set, validation set, and test set; S3: Design the corresponding modules in the CAFET network architecture; S4: Design the L1 loss function and Adam optimizer for optimizing the network; S5: The divided data is sent into the model for training, verification and testing, and the model is evaluated according to the corresponding indicators. The model training is completed.

2. According to claim 1, a brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method is characterized by: The S1 at least includes the following steps: reading the nii file in the original data set, and slicing it along the x-axis, y-axis or z-axis, converting it into a 2D image, saving it in PNG format and only retaining the intermediate image.

3. The brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method according to claim 1, characterized in that: The S2 at least comprises the following steps: The images in the original dataset after slicing are randomly shuffled, and bicubic downsampling is used to obtain low-resolution images by ×2 and ×4 times to simulate the degradation in the real world, and then saved in PNG format; Finally, paired low-resolution and high-resolution data pairs are generated. Subsequently, the shuffled paired data are divided into training set, validation set, and test set in a ratio of 7:2:

1.

4. The brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method according to claim 1, characterized in that: The CAFET network architecture in S3 at least includes shallow feature extraction, deep feature extraction and image reconstruction; Input low-resolution image First, shallow features are extracted to obtain shallow features. in represents the number of input channels, and C represents the number of intermediate feature channels; Next, the shallow features are fused through N residual enhanced attention blocks and a 3×3 convolutional layer to extract the deep features F d ; Afterwards, the shallow features F are connected through the global residual connection s With the deep feature F d The fusion is performed and finally a high-resolution image is generated through an image reconstruction module; The residual enhanced attention fusion block consists of a hybrid fusion module, an adaptive sparse overlapping attention module, and a 3×3 convolutional layer with a global residual connection; In the image reconstruction module, the pixel shuffling method is used to upsample the fused features to generate a high-resolution image.

5. The brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method according to claim 4, characterized in that: The hybrid fusion module consists of a high-frequency enhancement block, a hybrid attention block, and a large-window fusion attention block; The high-frequency enhancement block is used to effectively capture edge detail information. After the input feature passes through the first Layernorm layer, the feature F is obtained. in , and then the feature F in It is divided into two parts according to the channel dimension and sent to two branches for high-frequency feature extraction and enhancement to obtain F M Features and F N feature; for F M Features, using 3×3 convolution and GELU activation function to obtain local features For F N First, we use the maximum pooling to obtain the high-frequency part of the feature, and then enhance the high-frequency information by performing 1×1 convolution and GELU activation on the high-frequency information. Finally, we connect along the channel dimension and use 1×1 convolution to deeply fuse the information of the two branches.

6. The brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method according to claim 5, characterized in that: In the hybrid attention block, a shifted rectangular window attention module and a parallel channel attention mechanism are used to obtain global information and enhance the representation ability of the network. The shifted rectangular window attention module is the SRWAB module. The SRWAB module consists of a shifted rectangular window attention and a multi-layer perceptron. To more effectively capture information in different regions, the SRWAB module is divided into vertical window attention and horizontal window attention. Vertical window attention is when the window height r h Greater than width r w When, it is referred to as HSAB; When the window width r w Greater than height r h When, it is referred to as WSAB; In the specific implementation, multiple attention heads are used to assign HSAB and WSAB to the same number of attention heads in parallel for independent calculation. For a given input F in ; First divide it into non-overlapping rectangular windows; For each rectangular window feature Project it to get query, key, value (F Q ,F K ,F V ) vector, and then perform rectangular window attention calculation on it: Where d represents the channel dimension, B represents the dynamic position encoding, Indicates F K The transpose of a vector; Then, attention operation is performed on all windows, and the final feature X is obtained through the MLP layer. att ; In addition, a channel attention module is embedded in the model to improve the expressiveness of visual features. The channel attention module is CAB. In order to avoid possible conflicts between CAB and SRWAB modules during network optimization, a small scaling factor α is multiplied on the output of CAB to ensure the stability and consistency of the model; Finally, the scaled channel attention output is added to the output of the SRWAB module to obtain the output feature F of the hybrid attention module. att .

7. The brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method according to claim 6, characterized in that: The construction of the large window fusion attention block includes at least the following steps: The output features F of the high-frequency enhancement block and the hybrid attention block are combined in and F att , divided into M non-overlapping windows respectively; Next, it is encoded through a linear layer to generate two query vectors; Subsequently, the two query vectors are initially fused to obtain the final query vector Q; At the same time, the output F of the hybrid attention block att Perform the same window division operation, then project the features and compress their channels to obtain the key matrix Sum Matrix Where r is the token reduction factor, M is the number of samples, and L 2 is the window space dimension; Next, perform permutation operations on K and V to generate the permuted key matrix Sum Matrix Afterwards, the fused feature F out It is sent to the feedforward network to fully aggregate features, using 1×1 convolution to aggregate channel information, 3×3 convolution to aggregate spatial information, and a gating mechanism to enhance information encoding to obtain fully fused features.

8. The brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method according to claim 4, characterized in that: The adaptive sparse overlapping attention module is composed of an adaptive sparse overlapping attention mechanism and a multi-layer perceptron; The adaptive sparse overlapping attention module is implemented by expanding the input features into blocks of different window sizes and performing shifted window attention calculations with a kernel size of M0 and a stride of M on these blocks of different window sizes; The input features are projected into Q, K, and V vectors; The query vector Q is divided into non-overlapping windows of size R×R; The K and V vectors are divided into Overlapping windows of size R0×R0; At the same time, in order to ensure the consistency of window size, the K and V vectors are used with a size of Zero padding operation, γ is a coefficient used to control the overlap size of the window, γ = 0.5; The calculation of R0 is as follows: R0=(1+γ)R (2) In addition, to ensure the sparsity of the attention mechanism, an activation function-based attention mechanism is introduced, which aims to filter out features with low matching scores between queries and keys; And the RELU output is squared to enhance the positive correlation; Finally, in order to prevent excessive sparsity, a dual-branch attention structure is proposed to ensure that the attention mechanism is both sparse and dense; In order to adaptively fuse the two branches and retain the key information to the greatest extent, adaptive attention weights are introduced to impose constraints on the sparse branches so that the network can dynamically adjust and retain key information; The final attention calculation result X final It is expressed as: X final =To d +w0·To s Among them, w0 is the adaptive weight used to balance the contribution of sparse and dense attention branches, Att d is the calculation result of the Softmax branch, Att s is the calculation result of the RELU branch.

9. The brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method according to claim 1, characterized in that: The S4 at least comprises the following steps: The L1 loss function is used to evaluate the difference between the predicted value and the true value; In the super-resolution task, the L1 loss function can help the model learn to predict the pixel values ​​in the high-resolution image so that the absolute error between the predicted high-resolution image and the real high-resolution image is minimized. Its value is in the range of 0-1, where the smaller the value, the lower the pixel loss of the reconstructed image of the network, which means a smaller error. The advantages of Adam optimizer momentum and adaptive learning rate can help the model converge faster and improve training efficiency; In the super-resolution task, the Adam optimizer can effectively adjust the model parameters so that the model can better learn the mapping relationship from low-resolution images to high-resolution images.

10. The brain MRI super-resolution cross-fusion adaptive feature enhancement Transformer method according to claim 1, characterized in that: The principle of the corresponding indicator evaluation model in S5 is as follows: Peak signal-to-noise ratio is used to measure the similarity between the reconstructed image and the original image; Peak signal-to-noise ratio evaluates image quality by comparing the pixel values ​​between the original image and the reconstructed image. The calculation formula of PSNR is as follows: Among them, MAX is the maximum value of the image pixel value, and MSE is the mean square error, which represents the sum of the squares of the difference between each pixel of the original image and the reconstructed image, as shown in the formula: Where M is the number of samples, y i is the actual observed value, is the corresponding predicted value; The peak signal-to-noise ratio only cares about the difference between the input and reconstructed pixels. Generally speaking, the higher the peak signal-to-noise ratio value, the smaller the difference between the reconstructed image and the original image, and the higher the reconstruction quality; Structural similarity aims to comprehensively consider brightness, contrast, and structural information to more comprehensively evaluate image quality, see the following formula: in, and μ I Represent the mean brightness of the generated image and the real image respectively, and σ I Represent the brightness variance of the generated image and the real image respectively, Represents the covariance of brightness, C1 and C2 are small constants used to avoid the denominator being zero; The value of the structural similarity index close to 1 indicates that the structural similarity between the reconstructed image and the original image is high and the quality is good; Structural similarity is often used together with other evaluation indicators to comprehensively evaluate the performance of super-resolution algorithms.

Images