Multi-modal large model guided individualized magnetic resonance fast sampling and reconstruction method

Abstract

The application discloses a multi-modal large model guided individualized magnetic resonance rapid sampling and reconstruction method, and specific steps are as follows: step 1, constructing a multi-modal large model as a guided nuclear magnetic resonance image reconstruction model; step 2, solving the multi-modal large model as the guided nuclear magnetic resonance image reconstruction model constructed in step 1 in two stages; step 3, training the multi-modal large model as the guided nuclear magnetic resonance image reconstruction model; and step 4, applying the trained multi-modal large model as the guided nuclear magnetic resonance image reconstruction model to perform individualized nuclear magnetic resonance imaging and pathological positioning. The image reconstructed by the method has good interpretability and can adapt to individual pathological characteristics.

Description

Technical Field

[0001] This invention belongs to the technical field of medical magnetic resonance imaging methods, specifically relating to a multimodal large-model-guided method for individualized rapid magnetic resonance sampling and reconstruction. Background Technology

[0002] Magnetic resonance imaging (MRI) is a widely used non-invasive, radiation-free imaging technique in clinical practice. It provides high-resolution, soft tissue contrast images while revealing rich anatomical and functional information, offering crucial information for medical diagnosis. However, it still faces significant challenges in clinical settings. Excessive scanning time for fully sampled images often leads to patient discomfort and motion artifacts. Furthermore, current imaging protocols tend to optimize image quality at the voxel level, neglecting the balance between imaging effectiveness and the detectability of clinically relevant abnormalities (such as lesions), which is precisely the core concern in clinical diagnosis and treatment.

[0003] To shorten scan time and improve imaging efficiency, researchers have developed various magnetic resonance imaging (MRI) reconstruction methods, particularly those for accelerating imaging, which significantly reduce scan time. These methods can be broadly categorized into traditional optimization-based methods and deep learning-based methods. Traditional methods, such as compressed sensing and low-rank models, often struggle to achieve high-quality results in high-acceleration environments due to their reliance on manually designed priors, making it difficult to meet clinical needs. In recent years, deep learning technologies, represented by convolutional neural networks (CNNs), have achieved significant breakthroughs in computer vision, with their applications continuously expanding and demonstrating outstanding performance across multiple interdisciplinary fields. In the field of MRI image reconstruction, researchers have not only achieved significant results by introducing CNN technology but also fully demonstrated its broad application prospects. Overall, deep learning-based MRI image reconstruction methods are mainly divided into two categories: data-driven methods and model-driven methods. Data-driven methods heavily rely on large datasets and complex network architectures for end-to-end training to achieve high-quality reconstruction, but their interpretability is limited. In contrast, model-driven approaches expand iterative optimization algorithms into deep networks, combining the advantages of model-based reconstruction with the learning capabilities of deep learning, thereby providing enhanced interpretability and reliability.

[0004] However, despite significant advancements in accelerating imaging, these methods have focused on overall voxel-level quality, neglecting the personalized imaging needs of clinically critical abnormalities—crucial for developing precise diagnostic and treatment plans. While directly improving the reconstruction quality of MRI images is an intuitive approach, the true clinical relevance lies in their performance on downstream tasks. Recent research has attempted to address this by integrating downstream task information into the reconstruction process to enhance the visibility of clinically relevant abnormalities. These methods often rely on empirical network designs with limited interpretability, thus restricting their consistency and reliability in enhancing pathological features. Other studies have explored learnable sampling patterns, but these patterns are typically fixed and shared across all inputs, failing to adapt to individual pathological characteristics. This limits their effectiveness in optimizing image acquisition for specific clinical needs. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal large model-guided method for individualized rapid magnetic resonance imaging sampling and reconstruction, which solves the problems of poor interpretability of reconstructed images and inability to adapt to individual pathological characteristics in existing methods.

[0006] The technical solution adopted in this invention is a multimodal large model-guided individualized rapid magnetic resonance sampling and reconstruction method, the specific steps of which are as follows: Step 1: Construct a multimodal large model-guided nuclear magnetic resonance image reconstruction model; Step 2: Solve the NMR image reconstruction model guided by the multimodal large model constructed in Step 1 in two stages; Step 3: Train a multimodal large model as a guided nuclear magnetic resonance image reconstruction model; Step 4: Apply the trained multimodal large model as a guided MRI image reconstruction model to perform individualized MRI imaging and pathological localization.

[0007] The invention is further characterized by: In step 1, the multimodal large model-guided MRI image reconstruction model is as follows: (1) In equation (1), This is a mapping operator used to map fully sampled MRI images. Mapped to an undersampled k-space, , For coil sensitivity information, For Fast Fourier Transform, It is a learnable sampling matrix; and It is a regularization parameter used to balance data items and multiple regularization terms; These are learnable parameters; For noise reduction regularization; For local constraint regularization; in: (2) In equation (2), It is a personalized sampling network; These are global sampling parameters; Low-frequency image; Learnable sampling matrix If we set it to the form of a learnable probability mask combined with a threshold operation, then we have: (3) In equation (3), It is a sigmoid function operation; A learnable probability mask; Then we have: (4) In equation (4), , for each individual learnable probability parameter; If we only consider a one-dimensional imaging scene, then The values ​​of each column of parameters in the table are the same, that is After simplification, you only need to learn Introduce a uniform distribution To implement a pseudo-Bernoulli variable, then Represented as: (5) In equation (5), It's a sigmoid function operation. To standardize operations; As an acceleration factor, .

[0008] Denoising Regularization for: (6) In equation (6), As the first denoising network, Let be the learnable parameters in this denoising network, and obtain their values ​​at the th . The Taylor expansion formula for the step is as follows: (7) In equation (7) For the first denoising network about Jacobian matrix, let To obtain information about denoising regularization Approximate estimate: (8) When the disturbance ∆ Enough hours Approaching 0, based on this, we can draw the following conclusions: (9).

[0009] Local constraint regularization for: (10) In equation (10), For multimodal large models; This is the second denoising network; Similarly, for the k-th iteration, This can be processed using a univariate Taylor expansion, resulting in: (11) In equation (11), For multimodal large models The corresponding Jacobian matrix, let ,So This can be expressed as:

[0010] (12) For the second denoising network about Jacobian matrix, when the disturbance ∆ Enough hours If the value approaches 0, then based on the above formula, the following approximate estimate can be obtained: (13).

[0011] First noise reduction network With the second denoising network Both have the same structure, using a ResNet network, consisting of 7 cascaded skip-connected modules, with the first 6 modules each consisting of a single layer. The convolutional layer consists of one BN layer and one ReLU activation layer. The 7th module consists of one layer. It consists of a convolutional layer and a BN layer.

[0012] Multimodal large model An improved CLIP network is adopted, specifically: the original CLIP network includes an image feature extraction module and a language feature extraction module. The image feature extraction module has 16 layers, and a fully connected layer is added after every 4 layers, for a total of 4 fully connected layers. The language feature extraction module remains unchanged.

[0013] Personalized sampling network Specifically, the coordinate information is cascaded after being encoded by sine and cosine respectively, and then passed through a fully connected layer to obtain coordinate features; the low-frequency image is input into a three-layer network A to obtain image features; the image features and coordinate features are cascaded and passed through a three-layer network B, and the obtained features are input into a global average pooling and fully connected layer to obtain the final high-frequency sampling information; Each layer of the three-layer network A consists of one layer. A convolutional layer consists of one GN layer and one LeakyReLU activation layer; In the three-layer network B, the first two layers each consist of a single layer. A convolutional layer, a GN layer, and a LeakyReLU activation layer; the last layer consists of a single layer. It consists of a convolutional layer and a GN layer.

[0014] The specific process of step 2 is as follows: Solving the first part: Solving equation (14) yields the parameters of the multimodal large model-guided nuclear magnetic resonance image reconstruction model and : (14) For convenience, the mapping operator is still described as follows: Instead ; nuclear magnetic resonance images As a noise reduction network Given an input image and an output image with noise and aliasing artifacts removed, we have: (15) Substituting formula (15) into the first equation of formula (14), the first equation expands and solves to obtain: (16) We obtain a globally denoised solution; The second equation in formula (14) is solved using the ADMM optimization algorithm. The number of iterations of the ADMM algorithm is set to n. The iterative formulas are: (17) in, , and These are all intermediate quantities generated during the ADMM iterative solution process. Set to 1, and in the first iteration ; As can be seen from formula (17), ,and ; For the first equation in formula (17), its analytical solution is: (18) In equation (18), I is the identity matrix, which is obtained by using... Approximate iterative update; For the second equation in formula (17), expand Then, its analytical solution is: (19); Second stage solution: Will Decomposed into low-frequency sampling parameters With high frequency sampling parameters The sum (i.e., Equation 20) will include the low-frequency sampling parameters. Low-frequency images acquired Coordinate information as an individualized sampling mask network The input and output are high-frequency sampling information and global sampling parameters. Substituting into formula (2) yields the learnable parameters. The final undersampled image is obtained using formula (22). Then As input to a multimodal large model-guided nuclear magnetic resonance image reconstruction model, the final reconstruction result can be obtained. Formula (20) is as follows: (20) In equation (20), These are low-frequency sampling parameters. These are high-frequency sampling parameters; Low-frequency images It can be expressed as: (twenty one) In equation (21), and These are the Fast Fourier Transform and the Inverse Fast Fourier Transform, respectively. This refers to the low-frequency sampling mask trajectory generated based on the low-frequency sampling information parameters; It is a real image; The final undersampled image The expression is: (twenty two).

[0015] The specific process of step 3 is as follows: Step 3.1, construct the dataset; The dataset consists of multiple data sets, each of which includes fully sampled k-space data reconstructed MRI images and lesion pixel-level labels for the MRI images; Step 3.2: Train a multimodal large model guided by the dataset to reconstruct the nuclear magnetic resonance image. The specific process is as follows: Step 3.2.1: Using the reconstructed MRI image from the fully sampled k-space data and the lesion pixel-level labels from the MRI image as input, the Dice, DEC, and Focal losses are applied to... Fine-tuning is performed on the pre-defined fully connected layers until their localization maps achieve the pre-defined performance, at which point fine-tuning stops, resulting in a well-trained multimodal large model. The trained multimodal large model The parameters are frozen and integrated into the multimodal large model-guided MRI image reconstruction model to provide pathological site information and guide the training of the multimodal large model-guided MRI image reconstruction model. Step 3.2.2, will The undersampled information collected from the corresponding sampling trajectory is used as the initial input. Then, the gradient of the parameters of the multimodal large-scale model-guided MRI image reconstruction model is updated using the loss function. Finally, the gradient is used as the input of the Adam algorithm to update the parameters of the reconstruction model, thereby obtaining the optimal parameters of the multimodal large-scale model-guided MRI image reconstruction model. Fix the optimal parameters; The loss function is: (twenty three) In equation (23), Num is the number of samples; Real images; To reconstruct the model in The final reconstructed image of the stage (full-sample MRI image); for The location map output at stage K. ; Step 3.2.3, extract the low-frequency image. Input to individualized sampling network In this process, individualized sampling parameters are obtained, and these individualized sampling parameters are then compared with the global sampling parameters. The learnable parameters are obtained through formula (2). Then, the undersampled image is obtained using formula (22). undersampled image The multimodal large model with fixed optimal parameters is used for training in the MRI image reconstruction model to obtain the trained multimodal large model with MRI image reconstruction model. Training individualized sampling network The loss function is: (twenty four) in, (25) In equation (25), This refers to the high-frequency sampling mask portion for sampling. To reconstruct the image corresponding to the high-frequency information in the image.

[0016] The beneficial effects of this invention are as follows: The multimodal large model-guided individualized rapid magnetic resonance imaging (MRI) sampling and reconstruction method of this invention provides pathological priors through a multimodal large model, guides the generation of image acquisition information, and further studies the downstream tasks of MRI reconstruction images for each individual. At the same time, it can utilize the prior information most suitable for the individual to perform the reconstruction task, effectively enhance MRI image reconstruction, and has better reconstruction results in pathological sites with strong interpretability; the multimodal large model is used to locate the pathological parts of the image, effectively assisting clinicians in diagnosis. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a structural diagram of the nuclear magnetic resonance image reconstruction model guided by the multimodal large model in the method of the present invention; Figure 3 This is a diagram of the individualized sampling network structure in the method of this invention; Figure 4 This is a flowchart of the PA module in the model of the method of the present invention; Figure 5 Visualization of reconstructed images at high sampling rates and pathological localization results; Figure 6 Visualization of reconstructed images at low sampling rates and pathological localization results. Detailed Implementation

[0018] The present invention will now be described in detail with reference to specific embodiments.

[0019] Example 1 This invention relates to a multimodal large model-guided method for individualized rapid magnetic resonance sampling and reconstruction, such as... Figure 1 As shown, please follow these steps: Step 1: Construct a multimodal large model-guided nuclear magnetic resonance image reconstruction model; Step 2: Solve the NMR image reconstruction model guided by the multimodal large model constructed in Step 1 in two stages; Step 3: Train a multimodal large model as a guided nuclear magnetic resonance image reconstruction model; Step 4: Apply the trained multimodal large model as a guided MRI image reconstruction model to perform individualized MRI imaging and pathological localization.

[0020] Example 2 Based on Example 1, the specific process of step 1 is as follows: Given undersampled k-space data The goal is to recover high-fidelity MRI images from undersampled k-space data. The multimodal large model-guided NMR image reconstruction model is as follows: (1) In equation (1), This is a mapping operator used to map fully sampled MRI images. Mapped to an undersampled k-space, , For coil sensitivity information, For Fast Fourier Transform (FFT). It is a learnable sampling matrix; and It is a regularization parameter used to balance data items and multiple regularization terms; As a learnable parameter, it is set to be the sum of a global sampling parameter and an individual sampling parameter. The purpose of this setting is to prevent the reconstruction results from being insufficient with only individual sampling parameters. For noise reduction regularization; For local constraint regularization; in: (2) In equation (2), It is an individualized sampling network whose input is a low-frequency image. This is because for magnetic resonance imaging, low-frequency information determines anatomical structures and is easy to obtain, and is the same for most people. The output is a high-frequency image. These are global sampling parameters, which are then jointly trained with the reconstruction model to obtain global sampling parameters adapted to each individual. Low-frequency image; Learnable sampling matrix If we set it to the form of a learnable probability mask combined with a threshold operation, then we have: (3) In equation (3), It's a sigmoid function operation, where t is a large number used to ensure... The value is close to 0 or 1; It is a Bernoulli variable, ensuring that it is learned. The value is 0 or 1 to simulate the value of the sampling mask in a real environment, where 0 means that the information of the corresponding point is not acquired, and 1 means that the information of the corresponding point is acquired. A learnable probability mask; Then we have: (4) In equation (4), , for each individual learnable probability parameter; If we only consider a one-dimensional imaging scene, then The values ​​of each column of parameters in the table are the same, that is After simplification, you only need to learn Since Bernoulli variables are difficult to implement in actual code operations, a uniform distribution is introduced. By using it to realize a pseudo-Bernoulli variable, then It can be represented as: (5) In equation (5), It is a sigmoid function operation, where s is a large number; To standardize operations and ensure that the acceleration factor is set to the preset value; As an acceleration factor, ; In formula (1), the first term This is a data fidelity measure to ensure that the reconstructed NMR image remains consistent with the original k-space data; In formula (1), the second term and the third item The first term is a hidden regularization term that ensures the reconstructed image is confined to a solution space that meets the needs of downstream clinical diagnosis. The second term minimizes noise and aliasing artifacts in the reconstructed image by introducing an adaptive denoising regularization term. The third term is a lesion-guided term that introduces a multimodal large model to provide information about the pathological region, thereby enhancing the image quality of that region. Denoising Regularization for: (6) In equation (6), As the first denoising network, Let be the learnable parameters in this denoising network, and obtain their values ​​at the th . The Taylor expansion formula for the step is as follows: (7) In equation (7) For the first denoising network about Jacobian matrix, let To obtain information about denoising regularization Approximate estimate: (8) When the disturbance ∆ Enough hours Approaching 0, based on this, we can draw the following conclusions: (9) Local constraint regularization for: (10) In equation (10), It is a multimodal large model used to provide information about pathological areas; This serves as the second denoising network, used to further refine and denoise the lesion area; Similarly, for the k-th iteration, This can be processed using a univariate Taylor expansion, resulting in: (11) In equation (11), For multimodal large models The corresponding Jacobian matrix, let ,So This can be expressed as:

[0021] (12) For the second denoising network about Jacobian matrix, when perturbation ∆ Enough hours If the value approaches 0, then based on the above formula, the following approximate estimate can be obtained: (13) like Figure 2 As shown, the NMR image reconstruction model guided by the multimodal large model is represented as follows: .

[0022] Example 3 Based on Example 2, the first denoising network With the second denoising network Both have the same structure, using a ResNet network composed of 7 cascaded skip connections. Specifically, the first 6 modules each consist of a single layer. The convolutional layer consists of one BN layer and one ReLU activation layer. The 7th module consists of one layer. It consists of a convolutional layer and a batch normalization (BN) layer; Multimodal large model An improved CLIP (Contrastive Language-Image Pre-training) network is adopted. Specifically, the original CLIP network includes an image feature extraction module and a language feature extraction module. The image feature extraction module has 16 layers. A fully connected layer is added after every 4 layers, for a total of 4 fully connected layers. The language feature extraction module remains unchanged. like Figure 3 As shown, individualized sampling network Coordinate information is introduced because high-frequency information from different coordinates can provide different individual information, allowing for better acquisition of sampling information tailored to the specific individual. Specifically: coordinate information is cascaded after being encoded by sine and cosine respectively, and then passed through a fully connected layer to obtain coordinate features; low-frequency images are processed through a three-layer network A to obtain image features, where each layer of network A consists of a single layer... The convolutional layer consists of one GN layer and one LeakyReLU activation layer. Image features and coordinate features are then cascaded and passed through a three-layer network B. The first two layers of network B each consist of a single layer... A convolutional layer, a GN layer, and a LeakyReLU activation layer; the last layer consists of a single layer. It consists of a convolutional layer and a GN layer; the obtained features are input into a global average pooling and fully connected layer to obtain the final high-frequency sampling information (high-frequency image).

[0023] Example 4 Based on Example 3, the specific process of step 2 is as follows: Solving the first part: Solving equation (14) yields the parameters of the multimodal large-scale model-guided nuclear magnetic resonance image reconstruction model and... : (14) For convenience, the mapping operator is still described as follows: Instead ; nuclear magnetic resonance images As a noise reduction network Given an input image and an output image with noise and aliasing artifacts removed, we have: (15) Substituting formula (15) into the first equation of formula (14), the first equation expands and solves to obtain: (16) We obtain a globally denoised solution, which already shows good results overall. Formula (16) is the global denoising module DN, where the second equation in formula (16) is the data consistency module DC1; The second equation in formula (14) is solved using the ADMM optimization algorithm (alternating multiplier method). The number of iterations of the ADMM algorithm is set to n. The iterative formulas are: (17) in, , and These are all intermediate quantities generated during the ADMM iterative solution process. Set to 1, and in the first iteration ; As can be seen from formula (17), ,and ;like Figure 4 As shown, formula (17) is the pathological perception module PA; For the first equation in formula (17), its analytical solution is: (18) In equation (18), I is the identity matrix, which is obtained by using... Approximate iterative update, formula (18) is the data consistency module DC2; For the second equation in formula (17), expand Then, its analytical solution is: (19) The second term in formula (19) is the data consistency module DC3; Second stage solution: Will Decomposed into low-frequency sampling parameters With high frequency sampling parameters The sum (i.e., Equation 20) will include the low-frequency sampling parameters. Low-frequency images acquired The coordinate information of high-frequency information is used as an individualized sampling mask network. The input is high-frequency sampling information (high-frequency image) and the output is global sampling parameters. Substituting into formula (2) yields the learnable parameters. The final undersampled image is obtained using formula (22). Then As input to a multimodal large-scale model-guided MRI image reconstruction model, the final reconstruction result can be obtained (i.e., ... As a noise reduction network The input is solved according to formulas (15) to (19) to obtain the final reconstruction result. Formula (20) is as follows: (20) In equation (20), These are low-frequency sampling parameters. For high-frequency sampling parameters, due to the physical mechanism of magnetic resonance imaging (MRI), low-frequency sampling often covers the middle portion, determining the anatomical structure of the image; high-frequency sampling often covers the edge portion, determining the details of the image. In general settings, the low-frequency sampling portion is consistent for any individual's scan, all aimed at acquiring anatomical information of the image. It can be regarded as prior knowledge, which can be determined before scanning, and scanning only the low-frequency part of the information is extremely fast; These are the globally sampled parameters obtained jointly. With low-frequency sampling information The difference represents the detailed information possessed by each individual; Low-frequency images It can be expressed as: (twenty one) In equation (21), and These are the Fast Fourier Transform and the Inverse Fast Fourier Transform, respectively. This refers to the low-frequency sampling mask trajectory generated based on the low-frequency sampling information parameters; It is a real image; The final undersampled image The expression is: (twenty two).

[0024] Example 5 Based on Example 4, the specific process of step 3 is as follows: Step 3.1, construct the dataset; The dataset consists of multiple data sets, each of which includes fully sampled k-space data reconstructed MRI images and lesion pixel-level labels for the MRI images; Step 3.2: Train a multimodal large model guided by the dataset to reconstruct the nuclear magnetic resonance image. The specific process is as follows: Step 3.2.1: Using the reconstructed MRI image from the fully sampled k-space data and the lesion pixel-level labels from the MRI image as input, the Dice, DEC, and Focal losses are applied to... Fine-tuning is performed on the pre-set fully connected layers until their localization maps (segmentation and classification results) achieve the pre-set performance, at which point fine-tuning stops, resulting in a trained multimodal large model. The trained multimodal large model The parameters are frozen and integrated into the multimodal large model-guided MRI image reconstruction model to provide pathological site information and guide the training of the multimodal large model-guided MRI image reconstruction model. Step 3.2.2, will The undersampled information collected from the corresponding sampling trajectory is used as the initial input. Then, the gradient of the parameters of the multimodal large-scale model-guided MRI image reconstruction model is updated using the loss function. Finally, the gradient is used as the input of the Adam algorithm to update the parameters of the reconstruction model, thereby obtaining the optimal parameters of the multimodal large-scale model-guided MRI image reconstruction model. Fix the optimal parameters; The loss function is: (twenty three) In equation (23), Num is the number of samples; Real images; To reconstruct the model in The final reconstructed image of the stage (full-sample MRI image); for The location map output at stage K. ; Step 3.2.3, extract the low-frequency image. Input to individualized sampling network In this process, individualized sampling parameters are obtained, and these individualized sampling parameters are then compared with the global sampling parameters. The learnable parameters are obtained through formula (2). Then, the undersampled image is obtained using formula (22). undersampled image The multimodal large model with fixed optimal parameters is used for training in the MRI image reconstruction model to obtain the trained multimodal large model with MRI image reconstruction model. Training individualized sampling network The loss function is: (twenty four) in, (25) In equation (25), This refers to the high-frequency sampling mask portion for sampling. To reconstruct the image corresponding to the high-frequency information in the image; Step 4: Apply the trained multimodal large model as a guided MRI image reconstruction model to perform individualized MRI imaging and pathological localization.

[0025] Through the above training process, this invention can determine all the optimal parameters of the trained multimodal large-scale model-guided MRI image reconstruction model, including the individualized sampling network, optimal global sampling parameters, and reconstruction model parameters. Based on actual scanning conditions, the MRI image reconstruction model guided by the trained multimodal large-scale model first performs a rapid scan of the patient's low-frequency information and directly obtains the corresponding image prior based on Fourier transform. Since no other operations are involved, this process is often very fast. Based on this information, the individualized sampling parameters are obtained by inputting into the individualized sampling network, and an undersampled image is obtained through scanning. Finally, a high-quality reconstructed image is obtained through the reconstruction network, and the multimodal large-scale model is used to further refine the image. Obtain its pathological location map.

[0026] Example 6 In numerical experiments, this invention uses two contrast datasets from the public MRI dataset fastMRI: PD (non-fat suppression) for the knee and FLAIR (fluid attenuated inversion recovery sequence) for the brain. The k-space sampling mode sampling rates are set to 1 / 4 and 1 / 8, respectively. During training and testing, this invention uses datasets of size [missing data]. A multimodal large-scale model was trained using 60% of the anatomically detailed continuous 2D slices selected from the 3D volume of the fastMRI dataset to guide MRI image reconstruction and test reconstruction accuracy. To objectively evaluate different methods, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) were used to measure the average reconstruction accuracy on the test set. In addition to evaluating the overall image reconstruction quality, this invention also specifically evaluated the reconstruction quality of specific pathological sites. Specifically, the PSNR and SSIM values ​​of the pathological sites within the bounding boxes were measured, denoted as PA-PSNR and PA-SSIM, respectively.

[0027] As shown in Tables 1 and 2, the multimodal large model-guided MRI image reconstruction model (PASS) of this invention is compared with traditional reconstruction methods, purely data-driven reconstruction methods, and model-driven reconstruction methods at different sampling rates. Traditional reconstruction methods include GRAPPA; purely data-driven reconstruction methods include SwinMR, Reflow, and Nail; model-driven reconstruction methods include ISTA-Net, MoDL, and MoDL set as a learnable sampling model, named LS-MoDL. The multimodal large model-guided MRI image reconstruction model of this invention achieves the best reconstruction accuracy at different sampling rates. Figure 5 and Figure 6 The images are visualizations of the reconstructed images at high and low sampling rates, respectively. It can be seen that the reconstructed MRI images of the present invention have clear structural details and no obvious artifacts; and can locate the pathological parts (the parts within the marked boxes) better.

[0028] Table 1. Comparison of different methods on the test data set at a 1 / 4 sampling rate.

[0029] Table 2. Comparison results of different methods on the test data set at a sampling rate of 1 / 8.

Claims

1. A method for rapid individualized magnetic resonance sampling and reconstruction guided by a multimodal large model, characterized in that, The specific steps are as follows: Step 1: Construct a multimodal large model-guided nuclear magnetic resonance image reconstruction model; Multimodal large model An improved CLIP network is adopted, specifically: the original CLIP network includes an image feature extraction module and a language feature extraction module. The image feature extraction module has 16 layers. A fully connected layer is added after every 4 layers, for a total of 4 fully connected layers. The language feature extraction module remains unchanged. Step 2: Solve the NMR image reconstruction model guided by the multimodal large model constructed in Step 1 in two stages; The specific process of step 2 is as follows: Solving the first part: Solving equation (14) yields the parameters of the multimodal large model-guided nuclear magnetic resonance image reconstruction model and : （14） In equation (14), This is a mapping operator used to map fully sampled MRI images. Mapped to an undersampled k-space, , For coil sensitivity information, For Fast Fourier Transform, It is a learnable sampling matrix; These are learnable parameters; and It is a regularization parameter used to balance data items and multiple regularization terms; For multimodal large models; As the first denoising network, These are the learnable parameters in this denoising network; For local constraint regularization; Given undersampled k-space data; For convenience, the mapping operator is still described as follows: Instead , These are global sampling parameters; nuclear magnetic resonance images As a noise reduction network Given an input image and an output image with noise and aliasing artifacts removed, we have: （15） Substituting formula (15) into the first equation of formula (14), the first equation expands and solves to obtain: （16） We obtain a globally denoised solution; The second equation in formula (14) is solved using the ADMM optimization algorithm. The number of iterations of the ADMM algorithm is set to n. The iterative formulas are: （17） in, , and These are all intermediate quantities generated during the ADMM iterative solution process. Set to 1, and in the first iteration ; As can be seen from formula (17), ,and ; For the first equation in formula (17), its analytical solution is: （18） In equation (18), I is the identity matrix, which is obtained by using... Approximate iterative update; For the second equation in formula (17), expand Then, its analytical solution is: （19）； Second stage solution: Will Decomposed into low-frequency sampling parameters With high frequency sampling parameters The sum, i.e., formula (20), is the low-frequency sampling parameters. Low-frequency images acquired Coordinate information as an individualized sampling mask network The input and output are high-frequency sampling information and global sampling parameters. Substituting into formula (2) yields the learnable parameters. The final undersampled image is obtained using formula (22). Then As input to a multimodal large model-guided nuclear magnetic resonance image reconstruction model, the final reconstruction result can be obtained. Formula (20) is as follows: （20） In equation (20), For low-frequency sampling parameters, These are high-frequency sampling parameters; Low-frequency images It can be expressed as: （21） In equation (21), and These are the Fast Fourier Transform and the Inverse Fast Fourier Transform, respectively. This refers to the low-frequency sampling mask trajectory generated based on the low-frequency sampling information parameters; It is a real image; The final undersampled image The expression is: （22）； Step 3: Train a multimodal large model as a guided nuclear magnetic resonance image reconstruction model; Step 4: Apply the trained multimodal large model as a guided MRI image reconstruction model to perform individualized MRI imaging and pathological localization.

2. The multimodal large model-guided individualized rapid magnetic resonance sampling and reconstruction method according to claim 1, characterized in that, In step 1, the multimodal large model-guided MRI image reconstruction model is as follows: （1） In equation (1), This is a mapping operator used to map fully sampled MRI images. Mapped to an undersampled k-space, , For coil sensitivity information, For Fast Fourier Transform, It is a learnable sampling matrix; and It is a regularization parameter used to balance data items and multiple regularization terms; These are learnable parameters; For noise reduction regularization; For local constraint regularization; in: （2） In equation (2), It is a personalized sampling network; These are global sampling parameters; Low-frequency image; Learnable sampling matrix If we set it to the form of a learnable probability mask combined with a threshold operation, then we have: （3） In equation (3), It is a sigmoid function operation; A learnable probability mask; Then we have: （4） In equation (4), , for each individual learnable probability parameter; If we only consider a one-dimensional imaging scene, then The values ​​of each column of parameters in the table are the same, that is After simplification, you only need to learn Introduce a uniform distribution To implement a pseudo-Bernoulli variable, then Represented as: （5） In equation (5), It's a sigmoid function operation. To standardize operations; As an acceleration factor, .

3. The multimodal large model-guided individualized rapid magnetic resonance sampling and reconstruction method according to claim 2, characterized in that, Denoising Regularization for: （6） In equation (6), As the first denoising network, Let be the learnable parameters in this denoising network, and obtain their values ​​at the th . The Taylor expansion formula for the step is as follows: （7） In equation (7) For the first denoising network about Jacobian matrix, let To obtain information about denoising regularization Approximate estimate: （8） When disturbance Enough hours Approaching 0, based on this, we can draw the following conclusions: （9）。 4. The multimodal large model-guided individualized rapid magnetic resonance sampling and reconstruction method according to claim 3, characterized in that, Local constraint regularization for: （10） In equation (10), For multimodal large models; This is the second denoising network; Similarly, for the k-th iteration, This can be processed using a univariate Taylor expansion, resulting in: （11） In equation (11), For multimodal large models The corresponding Jacobian matrix, let ,So This can be expressed as: （12） For the second denoising network about Jacobian matrix, when perturbed Enough hours As the value approaches 0, based on the above formula, we can obtain the following approximate estimate: （13）。 5. The multimodal large model-guided individualized rapid magnetic resonance sampling and reconstruction method according to claim 4, characterized in that, First noise reduction network With the second denoising network Both have the same structure, using a ResNet network, consisting of 7 cascaded skip-connected modules, with the first 6 modules each consisting of a single layer. The convolutional layer consists of one BN layer and one ReLU activation layer. The 7th module consists of one layer. It consists of a convolutional layer and a BN layer.

6. The multimodal large model-guided individualized rapid magnetic resonance sampling and reconstruction method according to claim 4, characterized in that, Personalized sampling network Specifically, the coordinate information is cascaded after being encoded by sine and cosine respectively, and then passed through a fully connected layer to obtain coordinate features; the low-frequency image is input into a three-layer network A to obtain image features; the image features and coordinate features are cascaded and passed through a three-layer network B, and the obtained features are input into a global average pooling and fully connected layer to obtain the final high-frequency sampling information; Each layer of the three-layer network A consists of one layer. A convolutional layer consists of one GN layer and one LeakyReLU activation layer; In the three-layer network B, the first two layers each consist of a single layer. A convolutional layer, a GN layer, and a LeakyReLU activation layer; the last layer consists of a single layer. It consists of a convolutional layer and a GN layer.

7. The multimodal large model-guided individualized rapid magnetic resonance sampling and reconstruction method according to claim 4, characterized in that, The specific process of step 3 is as follows: Step 3.1, construct the dataset; The dataset consists of multiple data sets, each of which includes fully sampled k-space data reconstructed MRI images and lesion pixel-level labels for the MRI images; Step 3.2: Train a multimodal large model guided by the dataset to reconstruct the nuclear magnetic resonance image. The specific process is as follows: Step 3.2.1: Using the fully sampled k-space data to reconstruct the MRI image and the lesion pixel-level labels of the MRI image as input, the Dice, DEC, and Focal losses are applied to... Fine-tuning is performed on the pre-defined fully connected layers until their localization maps achieve the pre-defined performance, at which point fine-tuning stops, resulting in a well-trained multimodal large model. The trained multimodal large model The parameters are frozen and integrated into the multimodal large model-guided MRI image reconstruction model to provide pathological site information and guide the training of the multimodal large model-guided MRI image reconstruction model. Step 3.2.2, will The undersampled information collected from the corresponding sampling trajectory is used as the initial input. Then, the gradient of the parameters of the multimodal large-scale model-guided MRI image reconstruction model is updated using the loss function. Finally, the gradient is used as the input of the Adam algorithm to update the parameters of the reconstruction model, thereby obtaining the optimal parameters of the multimodal large-scale model-guided MRI image reconstruction model. Fix the optimal parameters; The loss function is: （23） In equation (23), Num is the number of samples; Real images; To reconstruct the model in The final reconstructed image of the stage, namely the full-sample MRI image; for The location map output at stage K. ; Step 3.2.3, extract the low-frequency image. Input to individualized sampling network In this process, individualized sampling parameters are obtained, and these individualized sampling parameters are then compared with the global sampling parameters. The learnable parameters are obtained through formula (2). Then, the undersampled image is obtained using formula (22). undersampled image The multimodal large model with fixed optimal parameters is used for training in the MRI image reconstruction model to obtain the trained multimodal large model with MRI image reconstruction model. Training individualized sampling network The loss function is: （24） in, （25） In equation (25), This refers to the high-frequency sampling mask portion for sampling; To reconstruct the image corresponding to the high-frequency information in the image.

Images