MRI Imaging Prediction Methods and Systems for Neoadjuvant Therapy of Rectal Cancer

By integrating baseline MRI and individualized clinical information through a dual-branch memory enhancement module and hierarchical modulation U-Net, predictive images for future time points are generated, solving the problem of predicting the degree of tumor regression before neoadjuvant therapy and realizing the visual evaluation of individualized treatment plans.

CN122714481BActive Publication Date: 2026-11-10SUZHOU INST OF BIOMEDICAL ENG & TECH CHINESE ACADEMY OF SCI
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611213045.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-08-11
Publication Date
2026-11-10
Estimated Expiration
2046-08-11

AI Technical Summary

Technical Problem

Existing technologies cannot predict the degree of tumor regression before neoadjuvant therapy, lack objective quantitative assessment methods, and cannot make individualized treatment decisions using baseline MRI and individualized clinical information. Furthermore, traditional models do not fully utilize the three-dimensional spatial information of MRI images and individualized clinical information.

Method used

By employing a dual-branch memory enhancement module and hierarchical modulation U-Net, baseline MRI images and individualized clinical information are integrated through a shared memory slot to generate predictive images for future time points, enabling a visual assessment of treatment efficacy.

Benefits of technology

It enables visualized assessment of efficacy before neoadjuvant therapy, improves the anatomical accuracy of predictive images and the rationality of tumor regression changes, and supports individualized treatment decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122714481B_ABST
    Figure CN122714481B_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for predicting the efficacy of neoadjuvant therapy for rectal cancer using MRI images, belonging to the field of medical image processing. The method includes: acquiring the patient's baseline MRI images, individualized clinical information, and target prediction time points; extracting initial feature maps from T2WI and ADC sequences respectively; embedding the target prediction time points and concatenating them with clinical information to generate conditional vectors; inputting the initial feature maps and conditional vectors into a dual-branch memory enhancement module to obtain enhanced T2WI and ADC features; fusing the two enhanced features using cross-attention; inputting the fused features and conditional vectors into a hierarchical modulation U-Net, employing a differentiated modulation strategy based on feature depth, and outputting predicted feature maps; and generating predicted T2WI and ADC sequences. This invention enables visualized prediction of future follow-up MRI images before treatment and can be used to assist physicians in developing individualized neoadjuvant therapy plans.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a method and system for predicting the efficacy of neoadjuvant therapy for rectal cancer using MRI images. Background Technology

[0002] Currently, the evaluation of neoadjuvant therapy efficacy mainly relies on post-treatment magnetic resonance imaging (MRI). Clinicians use T2-weighted imaging to observe changes in tumor morphology, volume, and depth of invasion, and combine this with MRI tumor regression grading or solid tumor efficacy evaluation criteria for subjective scoring. Diffusion-weighted imaging and apparent diffusion coefficients are also used to compare changes in tumor cell density before and after treatment. However, existing evaluation methods have the following technical limitations: Evaluation can only be performed after neoadjuvant therapy is completed, making it impossible to predict the degree of tumor regression before treatment and hindering timely dynamic adjustments to the treatment plan. If imaging findings at future time points could be predicted based on individual patient information before treatment, it would help screen for treatment-sensitive individuals and avoid ineffective treatments.

[0003] The MRI tumor regression grading or solid tumor treatment efficacy evaluation criteria rely on the subjective experience of radiologists, have limited inter-observer consistency, and lack objective quantitative assessment methods.

[0004] Current technologies can only describe or classify existing follow-up MRI images, and cannot generate predicted images for a specific future time point based on the patient's baseline images, clinical information, and that time point. This makes it difficult for clinicians to visually observe the evolution of tumors as treatment progresses, limiting the visualization basis for individualized treatment decisions.

[0005] Furthermore, the radiographic evolution of locally advanced rectal cancer patients after neoadjuvant therapy exhibits significant individual differences, and different pathological types show varying sensitivities to neoadjuvant therapy. Current research on longitudinal MRI data analysis remains insufficient: on the one hand, it fails to fully utilize the three-dimensional spatial information of the tumor in MRI images, relying solely on two-dimensional slices of the largest tumor layer for model construction, making it difficult to capture the heterogeneity of the tumor in three-dimensional space; on the other hand, it only uses patient imaging data to predict outcomes at a specific time point, losing individualized clinical information and failing to provide targeted modeling for different individuals.

[0006] Therefore, how to effectively utilize patients' baseline MRI, individualized clinical information, and follow-up time points to construct a predictive model that can generate future follow-up images and realize a visual assessment of efficacy before neoadjuvant therapy is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] To achieve the above-mentioned objectives and other advantages of the present invention, a first objective of the present invention is to provide a method for predicting the pre-treatment efficacy of neoadjuvant therapy for rectal cancer using MRI images, comprising the following steps: Obtain the patient's baseline MRI images, individualized clinical information, and target prediction time points; Feature extraction was performed on the T2WI and ADC sequences of the baseline MRI images to obtain an initial feature map; The target predicted time point is processed by embedding vectors and concatenated with individualized clinical information to generate a conditional vector. The initial feature map and conditional vector are input into the dual-branch memory enhancement module, which includes a decoupled spatial attention branch, a conditional temporal evolution branch, and a shared memory slot. The two branches interact bidirectionally with the shared memory slot to obtain the enhanced T2WI features and ADC features. The enhanced T2WI features and ADC features are fused by cross-attention to obtain the fused features; The fused features and conditional vectors are input into a hierarchical modulation U-Net, which employs a differentiated modulation strategy based on the depth of the feature map and outputs a predicted feature map. Predicted T2WI and ADC sequences are generated based on the predicted feature map.

[0008] Furthermore, the calculation method for the read operation of the shared memory slot is as follows: in, For the enhanced feature map after reading, For the input feature map, As a learnable scalar of memory strength, To recover the projection matrix, To query the projection matrix, For shared memory slot matrix.

[0009] Furthermore, the write operation update method for the shared memory slot is as follows: in, For the k-th memory slot, For write gating, To write the weights, This is for writing value vectors.

[0010] Furthermore, the number of shared memory slots ranges from 8 to 32, and the prototype feature dimension ranges from 64 to 256.

[0011] Furthermore, the hierarchical modulation U-Net includes a three-layer downsampling encoder and a three-layer upsampling decoder, for a total of six feature layers. Each feature layer dynamically generates modulation parameters for the corresponding feature layer based on the condition vector through an independent multilayer perceptron.

[0012] Furthermore, the differentiated modulation strategy includes: For shallow features, a combination of channel scaling, channel offsetting, and spatial modulation is used for modulation. For mid-level features, modulation is performed using a combination of channel scaling and channel offset. For deep features, modulation is performed using only channel scaling.

[0013] Furthermore, the shallow features include the output features of the encoder first layer, encoder second layer, and decoder third layer; the middle features include the output features of the encoder third layer and decoder second layer; and the deep features include the output features of the decoder first layer.

[0014] Furthermore, the spatial modulation is achieved through a spatial graph generation network, which includes: a fully connected layer that maps the conditional vector to a vector, reshapes it into a three-dimensional tensor, smooths it through 3D convolution, and outputs the spatial modulation graph through the Sigmoid function.

[0015] Furthermore, the specific method for generating the predicted T2WI sequence and ADC sequence is as follows: the predicted feature map is used to generate a gated map through 3D convolution and the Sigmoid function; the gated map is multiplied element-wise with the normalized original input image to obtain a gated weighted map; the gated weighted map is then added element-wise with the predicted feature map and output through the Sigmoid function.

[0016] Furthermore, the personalized clinical information includes one or more of the following: demographic information, medical history information, molecular immune and mismatch repair indicators, laboratory blood and inflammatory indicators, and tumor markers.

[0017] Furthermore, the method also includes the step of generating a spatial attention heatmap: fusing the attention weights in the shared memory slot read operation, the spatial attention map of the cross-attention module, and the spatial modulation map of the hierarchical modulation U-Net shallow decoder to generate an overall spatial attention heatmap and visualize it in the prediction sequence.

[0018] A second objective of this invention is to provide an MRI imaging prediction system for the pre-treatment efficacy of neoadjuvant therapy for rectal cancer, comprising: The data acquisition module is configured to acquire the patient's baseline MRI images, individualized clinical information, and target prediction time points; The condition generation and preliminary feature extraction module is configured to extract features from T2WI and ADC sequences to obtain initial feature maps, and to perform embedding vector processing on the target prediction time points, which are then concatenated with individualized clinical information to generate condition vectors. The dual-branch memory enhancement module includes a decoupled spatial attention branch, a conditional temporal evolution branch, and a shared memory slot. The two branches interact bidirectionally with the shared memory slot to output enhanced T2WI features and ADC features. The cross-attention fusion module is configured to perform cross-attention fusion on the enhanced T2WI features and ADC features to obtain fused features. The hierarchical modulation module is configured to input the fused features and conditional vectors into the hierarchical modulation U-Net and output a predicted feature map. The image prediction module is configured to generate predicted T2WI sequences and ADC sequences based on the predicted feature maps.

[0019] A third objective of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0020] A fourth objective of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0021] Compared with the prior art, the beneficial effects of the present invention are: This invention connects DSAM and CTEM via a shared memory slot, achieving knowledge synergy and feature enhancement across modal information through bidirectional read / write operations in a dual-branch manner. Individualized clinical information and target time points together constitute a conditional vector, used to guide the reading and writing of the memory slot. Through bidirectional read / write, the tumor evolution prototype stored in the memory slot can not only be read by the current sample, but the enhanced features of the current sample can also be written back to the memory slot for use by subsequent samples. This method achieves the sharing of cross-modal features and asynchronous knowledge, improving the problem of insufficient information interaction in the feature fusion and utilization process in traditional cross-modal modeling, thereby making the generated predicted images more reasonable in terms of anatomical structure and temporal changes.

[0022] This invention also designs a Hierarchical Conditional Modulation U-Net (HCM-U-Net) structure. Modulation parameters for each layer are generated through individualized conditional vectors, and a hierarchically differentiated injection strategy is employed to use differentiated modulation methods for features at different depths. Based on the high resolution and rich spatial details of shallow feature maps, spatial modulation is used for pixel-level modulation control to preserve tumor boundaries and texture information. Based on the moderate semantics and balanced information of mid-layer features, channel multiply-accumulate modulation is used to balance global and local feature information. Based on the low resolution and large receptive field of deep features, channel scaling is used to regulate the overall feature response mode. This method avoids the problem of excessive constraints or information loss on multi-scale features caused by uniform modulation, enabling individualized conditions to be efficiently and appropriately injected into corresponding layer features, thereby improving the anatomical realism of the predicted image and the rationality of tumor regression changes.

[0023] In this invention, the shared memory slot can effectively integrate data features from different patients, combined with individualized clinical information, and dynamically update and store tumor evolution prototypes during training through bidirectional read-write operations. Simultaneously, by combining HCM-U-Net for differential modulation of multi-scale features, and integrating baseline image features and individualized clinical features, it can predict images of patients at target time points before treatment, achieving pre-treatment visualization and prediction of NAT efficacy.

[0024] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 Flowchart of MRI imaging method for predicting the efficacy of neoadjuvant therapy for rectal cancer; Figure 2 This is a schematic diagram of the hierarchical modulation structure in HCM-U-Net; Figure 3 A schematic diagram of a model for predicting the efficacy of neoadjuvant therapy for rectal cancer. Figure 4 A diagram of a shared memory slot structure in a method for predicting the efficacy of neoadjuvant therapy for rectal cancer. Figure 5 Block diagram of an MRI imaging prediction system for neoadjuvant therapy of rectal cancer; Figure 6 A schematic diagram of computer equipment; Figure 7 This is a schematic diagram of a computer-readable storage medium. Detailed Implementation

[0026] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0027] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0028] The drawing numbers in this application are only used to distinguish the steps in the scheme and are not used to limit the execution order of the steps. The specific execution order is as described in the specification.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0030] This invention provides a method and system for predicting the efficacy of neoadjuvant therapy (NAT) for rectal cancer based on longitudinal MRI data. A shared memory slot is constructed based on the patient's baseline MRI images, individual clinical information, and follow-up time points. Anatomical features of T2WI sequences and functional features of ADC sequences are extracted by decoupling spatial attention and conditional temporal evolution. A hierarchical modulation strategy is used to generate predicted T2WI and ADC images for future time points, enabling a visual assessment of the efficacy of NAT treatment before treatment. The specific scheme is as follows: Example 1

[0031] A method for predicting the efficacy of neoadjuvant therapy for rectal cancer using MRI imaging, such as Figure 1 As shown, it includes the following steps: S100: Acquire the patient's baseline MRI images, individualized clinical information, and target prediction time points; In some embodiments, 1500 patients with locally advanced rectal cancer who received neoadjuvant therapy were enrolled from seven hospitals. Baseline MRI, T2WI and ADC sequences of follow-up MRI after neoadjuvant therapy, follow-up time points (e.g., weeks 6, 12, and 24 after treatment), and individualized clinical information were retrospectively collected. Individualized clinical information included: basic demographic information (age, sex, height, weight, body mass index); baseline medical and personal history (comorbidities, Charlson's comorbidity index, family history of cancer, hypertension, diabetes); molecular immune and mismatch repair indicators (MMR, hMLH1, hMSH2, hMSH6, PMS2); laboratory blood and inflammatory markers (hemoglobin, albumin, albumin / globulin ratio, white blood cells, neutrophils, high-sensitivity C-reactive protein, inflammatory burden index); tumor markers (carcinoembryonic antigen, carbohydrate antigen 199, carbohydrate antigen 125, alpha-fetoprotein, carbohydrate antigen 724, carbohydrate antigen 242); and prognostic follow-up outcomes, including overall survival and disease-free survival.

[0032] In this embodiment, the inclusion criteria for patients were: age ≥18 years; pathologically confirmed locally advanced rectal cancer, clinically staged as cT3 N0-cT4 N0, any T, and corresponding N1-N2; receiving neoadjuvant therapy, including long-course chemoradiotherapy, short-course radiotherapy combined with chemotherapy, and full-course neoadjuvant therapy; having complete baseline clinical data before neoadjuvant therapy; undergoing pelvic MRI within one month before neoadjuvant therapy, with both T2WI and ADC sequences; and having at least one follow-up pelvic MRI examination after neoadjuvant therapy (before surgery), with both T2WI and ADC sequences, and a clearly defined follow-up time point. The exclusion criteria were: history of other malignant tumors (excluding cured basal cell carcinoma of the skin or cervical carcinoma in situ); history of pelvic radiotherapy or colorectal surgery; inability to assess treatment response, including lack of follow-up MRI data or inability to measure the primary lesion; poor baseline or follow-up MRI image quality, including severe artifacts, signal loss, or inability to register ADC sequences; having received non-standard neoadjuvant therapy or other experimental treatments concurrently.

[0033] Disease-free survival is defined as the time from the start of neoadjuvant therapy to tumor recurrence or death from any cause, using magnetic resonance imaging (MRI) tumor regression grading to determine tumor progression. For patients who have neither recurred nor died at the follow-up cutoff date, disease-free survival is calculated from the follow-up cutoff date. Overall survival is defined as the time from the start of neoadjuvant therapy to death from any cause. For patients still alive at the follow-up cutoff date, overall survival is calculated from the follow-up cutoff date.

[0034] The patient's baseline MRI is paired with the actual MRI at each follow-up time point to form an independent training sample, while the corresponding target prediction time point t is recorded. If a patient has N follow-up MRIs at different time points, then N independent training samples are formed. These samples differ only in that the target prediction time point t is different from the corresponding actual MRI image, while the input baseline MRI and individualized clinical information are the same.

[0035] In some embodiments, the data is partitioned as follows: data from the three largest centers are divided into training, validation, and test sets in a 7:2:1 ratio; data from the remaining four hospitals are used as an external validation set. The training set is used for model parameter optimization, the validation set for hyperparameter tuning and model selection, the test set for internal performance evaluation, and the external validation set for independently evaluating the model's generalization ability. To ensure data independence, the baseline and all follow-up images of the same patient appear in only one set, ensuring no overlap between patients in different sets.

[0036] S200. Extract features from the T2WI and ADC sequences of the baseline MRI images to obtain an initial feature map; Specifically, a 3D convolutional neural network is used to perform preliminary feature extraction on the T2WI sequence to obtain spatial structure features; the same feature extraction is performed on the ADC sequence to obtain functional features.

[0037] S300: The target predicted time point is processed by embedding vectors and concatenated with individualized clinical information to generate a conditional vector through a multilayer perceptron. Specifically, the target time point t is mapped to a time embedding vector through a learnable embedding table. This vector is then concatenated with standardized individualized clinical features and input into a multilayer perceptron, which outputs a conditional vector. .

[0038] S400. The initial feature map and conditional vector are input to a dual-branch memory enhancement module, which includes a decoupled spatial attention branch, a conditional temporal evolution branch, and a shared memory slot. The conditional vector is input into the multilayer perceptrons of the decoupled spatial attention branch and the conditional temporal evolution branch respectively to generate corresponding modulation spatial decoupling parameters and offset field generation parameters, which are used to guide the feature enhancement process of the two branches. The two branches perform bidirectional read-write interaction with the shared memory slot to obtain the enhanced T2WI features and ADC features. like Figure 2 As shown, the shared memory slot matrix is ​​denoted as Where K is the number of memory slots. This represents the prototype feature dimension. In some preferred instances, K ranges from 8 to 32. The value range is from 64 to 256.

[0039] The read operation retrieves relevant prototypes from the shared memory slots using an attention mechanism. Let the input feature map of the current branch be denoted as... First, use the query projection matrix. A linear transformation is performed on the feature vector at each spatial location in the feature map to obtain the query feature vector. The query feature vectors at all locations constitute the query map. .

[0040] Then, each query vector and memory slot are processed using a dot product. The score is calculated and transformed into a probability distribution using Softmax normalization to obtain the attention weights. The read vector map is then obtained by weighted summation of the attention weights across the memory slots. Then by restoring the projection matrix Mapping the read vector at each position back to the original number of channels yields the recovered feature map P.

[0041] Introducing a learnable scalar γ to control memory strength, for P and Perform residual fusion to obtain enhanced feature maps for the read operation. : in, For the enhanced feature map after reading, For the input feature map, As a learnable scalar of memory strength, To recover the projection matrix, To query the projection matrix, This is a shared memory slot matrix. The working logic of the read operation is as follows: ... The feature map is projected onto the query space, and its dimensions are transformed to [ ], followed by transposition with shared memory slot Multiplying them yields the attention score, whose dimensions are [ The attention weights are calculated using the Softmax function and then multiplied by the shared memory slot M to obtain the read vector, which has the dimension []. ], by restoring the projection matrix Mapping this back to the number of input channels C, and then summing the residuals, yields the final enhanced feature map. .

[0042] The write operation updates the contents of the shared memory slot by using learnable write keys and write values, combined with attention weights. First, the output feature map... Global feature vectors are extracted in the spatial dimension, and the specific calculation method is as follows: in, The features are global, and then two learnable independent linear layers are used. and The global features are mapped to the memory slot space to obtain the write key vector. and write value vector .

[0043] Calculate write key Shared memory slots The dot product of each memory slot is normalized to the write weights using the Softmax function. : Write weight satisfy This indicates the proportion of the current sample's feature information allocated to different memory slots, representing the global features. Through a multilayer perceptron The gate value is obtained from the Sigmoid function. Control write strength, combined with and write value The memory slots are updated using momentum updates, and the updated memory slots are output. The specific calculation method is as follows: in, This is the momentum coefficient, which is not updated during training and has a value of 0.99. The logic of the write operation is as follows: encode the information of the current branch into the write value. Then, based on the writing weight Distributed to each memory slot, by gating By controlling the magnitude of the update required, it is ultimately superimposed onto the original memory slot, completing the gradual update of the memory slot.

[0044] Shared memory slots can dynamically update stored feature prototypes during training without changing feature dimensions through bidirectional read and write operations. This can effectively integrate tumor evolution features from different patients and unify the feature evolution patterns of different patients and at different follow-up time points. Through cross-sample knowledge accumulation and collaboration, it facilitates subsequent modeling and prediction.

[0045] like Figure 4 As shown, S500 performs cross-attention fusion on the enhanced T2WI features and ADC features to obtain fused features; Specifically, the enhanced T2WI features output by the decoupled spatial attention branch are used as the query matrix, and the enhanced ADC features output by the conditional temporal evolution branch are used as the key matrix and value matrix. The attention weights of the query and the key are calculated, and then the values ​​are weighted and summed to achieve cross-modal fusion of anatomical features and functional features.

[0046] S600. The fused features and conditional vectors are input into the hierarchical modulation U-Net, which adopts a differentiated modulation strategy according to the depth of the feature map and outputs a predicted feature map. like Figure 3 As shown, the hierarchical modulation U-Net uses a 3D U-Net as its basic architecture, comprising three downsampling operations and three upsampling operations, for a total of six main feature layers. Each feature layer uses an independent multilayer perceptron to dynamically generate modulation parameters based on a conditional vector. After upsampling at each layer, the decoder concatenates the resulting feature map with the output feature map of the corresponding encoder layer along the channel dimension. The multilayer perceptrons corresponding to each feature layer have the same structure but independent parameters, and the output dimension is determined by the number of modulation parameters required for that layer.

[0047] A hierarchical differential modulation strategy is adopted, using different modulation methods according to the depth of the feature map: Shallow features include the output features of the encoder's first layer, encoder's second layer, and decoder's third layer. Shallow feature maps have high resolution and contain rich spatial details, employing a combination of channel scaling, channel biasing, and spatial modulation. For shallow features, channel scaling weights are generated using independent multilayer perceptrons. Channel offset and intermediate vector .Will Spatial modulation maps are obtained through a spatial graph generation network. The spatial graph generation network includes, in sequence: a fully connected layer... Mapped to The vector is reconstructed into a 3D tensor through a reshaping operation, spatially smoothed through a 3D convolution of 3×3×3, and output via the Sigmoid function.

[0048] The intermediate features include the output features of the encoder's third layer and the decoder's second layer. The mid-level features have moderate semantics and balanced information, and adopt a combination of channel scaling and channel biasing without spatial modulation.

[0049] Deep features only include the first layer of the decoder ( Deep features have low resolution and large receptive fields, so they are modulated using only channel scaling.

[0050] Modulation functions of each layer Defined according to different layer depths: The modulated output feature map is then: .

[0051] The HCM-U-Net mechanism uses individualized clinical condition vectors to differentiate and regulate feature maps of different depths, enabling precise injection of conditional information into multi-scale features. This avoids information loss and over-constraint caused by uniform modulation in traditional encoder-decoder structures, thereby improving the anatomical realism of predicted sequences and the rationality of tumor regression changes.

[0052] The third layer output feature map of the decoder The feature map is then processed by two independent 1×1×1 3D convolutions with a stride of 1 and padding of 0 to obtain the predicted feature map of the T2WI sequence. Predicted feature map of ADC sequence .

[0053] Will Gated graphs are generated using 3D convolutions with a stride of 1 and padding of 1, and the sigmoid function. ,Will Gated graphs are generated using the same operation. The gated graph is then element-wise multiplied with the original input T2WI and ADC sequences after Min-Max normalization to obtain the gated weighted sequence. and The two gated weighted maps are added element-wise to their corresponding predicted feature maps to obtain fused features. Finally, the predicted feature maps are output through the Sigmoid function. The predicted feature maps include T2WI sequence predicted feature maps and ADC sequence predicted feature maps.

[0054] S700. Generate predicted T2WI and ADC sequences based on the predicted feature map.

[0055] This embodiment employs an end-to-end training strategy, optimizing all trainable parameters by minimizing the joint loss function between the predicted and real follow-up sequences. The total loss function includes pixel-level L1 loss, adversarial loss, memory slot orthogonal regularization loss, and offset field smoothing loss. in, For pixel-level L1 loss, calculate the absolute pixel error between the predicted sequence and the true sequence; To combat the loss, two independent discriminators are used to distinguish between true and false T2WI and ADC sequences, respectively. The memory slot orthogonal regularization loss is calculated as follows: This ensures that different memory slots are orthogonal to each other, avoiding the storage of redundant information; To compensate for the offset field smoothing loss, calculate the average of the sum of the absolute differences between each voxel and its adjacent voxels.

[0056] In a preferred example, the loss weights are as follows: =1、 =0.5、 =0.01、 =0.01.

[0057] To generate a spatial attention heatmap, attention information from three modules is integrated: First, during the read operations of the decoupled spatial attention branch and conditional temporal evolution branch, the attention weight of each spatial location to the memory slot is obtained, and then averaged to obtain a single-channel attention map. Secondly, the spatial attention map of the query position to the ADC sequence features is extracted from the cross-attention module. Finally, the last spatial modulation map is extracted from the shallow decoder of the hierarchical modulation U-Net. The three factors are weighted and fused to obtain an overall spatial attention heatmap, which is then visualized in the prediction sequence using color mapping.

[0058] The efficacy of neoadjuvant therapy in patients with locally advanced rectal cancer is highly complex and varies significantly from patient to patient, with differences in treatment regimens, follow-up frequency, and follow-up duration. Traditional methods, based on clinical indicators, radiomics, or deep learning features, construct classification models that output only binary categories (effective / ineffective) or risk scores. Physicians only obtain an abstract probability value, which cannot directly reflect the details of tumor evolution during treatment. This method, by utilizing baseline imaging, individualized clinical information, and future follow-up time points before treatment, enables visualized pre-treatment prediction of efficacy, thus solving the aforementioned technical problems.

[0059] Example 2 Based on the same concept, this embodiment also provides an MRI image prediction system for the pre-treatment efficacy of neoadjuvant therapy for rectal cancer, which applies the MRI image prediction method for the pre-treatment efficacy of neoadjuvant therapy for rectal cancer provided in Embodiment 1. For a detailed description of the MRI image prediction method for the pre-treatment efficacy of neoadjuvant therapy for rectal cancer provided in Embodiment 1, please refer to the corresponding description in the above method embodiments, and it will not be repeated here.

[0060] An MRI imaging system for predicting the pre-treatment efficacy of neoadjuvant therapy for rectal cancer, such as Figure 5 As shown, the system 800 includes: The data acquisition module 810 is configured to acquire the patient's baseline MRI images, individualized clinical information, and target prediction time points; The condition generation and preliminary feature extraction module 820 is configured to extract features from T2WI and ADC sequences to obtain an initial feature map, and to perform embedding vector processing on the target prediction time point, and concatenate it with individualized clinical information to generate a condition vector. The dual-branch memory enhancement module 830 includes a decoupled spatial attention branch, a conditional temporal evolution branch, and a shared memory slot. The two branches interact bidirectionally with the shared memory slot to output enhanced T2WI features and ADC features. The cross-attention fusion module 840 is configured to perform cross-attention fusion on the enhanced T2WI features and ADC features to obtain fused features; The hierarchical modulation module 850 is configured to input the fused features and conditional vectors into the hierarchical modulation U-Net and output a predicted feature map. Image prediction module 860 is configured to generate predicted T2WI sequences and ADC sequences based on the predicted feature map.

[0061] Based on the technical solution of the above embodiments, optionally, the calculation method for the read operation of the shared memory slot is as follows: in, For the enhanced feature map after reading, For the input feature map, As a learnable scalar of memory strength, To recover the projection matrix, To query the projection matrix, For shared memory slot matrix.

[0062] Based on the technical solutions of the above embodiments, optionally, the write operation update method of the shared memory slot is as follows: in, For the k-th memory slot, For write gating, To write the weights, This is for writing value vectors.

[0063] Based on the technical solution of the above embodiments, optionally, the number of shared memory slots ranges from 8 to 32, and the prototype feature dimension ranges from 64 to 256.

[0064] Based on the technical solution of the above embodiments, optionally, the hierarchical modulation U-Net includes a three-layer downsampling encoder and a three-layer upsampling decoder, for a total of six feature layers. Each feature layer dynamically generates the modulation parameters of the corresponding feature layer according to the condition vector through an independent multilayer perceptron.

[0065] Based on the technical solutions of the above embodiments, optionally, the differentiated modulation strategy includes: For shallow features, a combination of channel scaling, channel offsetting, and spatial modulation is used for modulation. For mid-level features, modulation is performed using a combination of channel scaling and channel offset. For deep features, modulation is performed using only channel scaling.

[0066] Based on the technical solutions of the above embodiments, optionally, the shallow features include the output features of the encoder first layer, the encoder second layer, and the decoder third layer; the middle features include the output features of the encoder third layer and the decoder second layer; and the deep features include the output features of the decoder first layer.

[0067] Based on the technical solution of the above embodiments, optionally, the spatial modulation is implemented through a spatial graph generation network, which includes: a fully connected layer that maps the conditional vector to a vector, reshapes it into a three-dimensional tensor, smooths it through 3D convolution, and outputs the spatial modulation graph through the Sigmoid function.

[0068] Based on the technical solution of the above embodiments, optionally, the specific method for generating the predicted T2WI sequence and ADC sequence is as follows: the predicted feature map is used to generate a gated map through 3D convolution and the Sigmoid function; the gated map is multiplied element-wise with the normalized original input image to obtain a gated weighted map; and the gated weighted map is added element-wise with the predicted feature map and then output through the Sigmoid function.

[0069] Based on the technical solutions of the above embodiments, optionally, the personalized clinical information includes one or more of the following: demographic information, medical history information, molecular immune and mismatch repair indicators, laboratory blood and inflammatory indicators, and tumor markers.

[0070] Based on the technical solution of the above embodiments, the method may optionally further include the step of generating a spatial attention heatmap: fusing the attention weights in the shared memory slot read operation, the spatial attention map of the cross-attention module, and the spatial modulation map of the layered modulation U-Net shallow decoder to generate an overall spatial attention heatmap and visualize it in the prediction sequence.

[0071] Example 3 A computer device 900, such as Figure 6As shown, the system includes a memory 910, a processor 920, and a computer program 930 stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a method for predicting the pre-treatment efficacy of neoadjuvant therapy for rectal cancer using MRI images. For a detailed description of the method, please refer to the corresponding description in the above method embodiments; it will not be repeated here.

[0072] Example 4 A computer-readable storage medium, such as Figure 7 As shown, a computer program is stored thereon. When executed by a processor, the computer program implements the steps of a method for predicting the pre-treatment efficacy of neoadjuvant therapy for rectal cancer using MRI images. For a detailed description of the method, please refer to the corresponding description in the above method embodiments, which will not be repeated here.

[0073] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention. Applications, modifications, and variations of the invention will be readily apparent to those skilled in the art.

[0074] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

[0075] The apparatus, computer device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, computer device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device, and non-volatile computer storage medium will not be repeated here.

[0076] Those skilled in the art will also know that, besides implementing the controller in the form of purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller take the form of logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included within it for implementing various functions can also be considered structures within that hardware component. Alternatively, the devices for implementing various functions can be considered as both software units implementing the method and structures within a hardware component.

[0077] The systems, apparatuses, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above apparatuses are described separately as various units based on their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0078] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0083] This specification may be described in the general context of computer-executable instructions, such as program units, that are executed by a computer. Generally, program units include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program units may reside in local and remote computer storage media, including storage devices.

[0084] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0085] The above description is merely an embodiment of this specification and is not intended to limit the scope of one or more embodiments of this specification. Various modifications and variations can be made to one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A method for predicting the efficacy of neoadjuvant therapy for rectal cancer using MRI images, characterized in that, Includes the following steps: Obtain the patient's baseline MRI images, individualized clinical information, and target prediction time points; Feature extraction was performed on the T2WI and ADC sequences of the baseline MRI images to obtain an initial feature map; The target predicted time point is processed by embedding vectors and concatenated with individualized clinical information to generate a conditional vector. The initial feature map and conditional vector are input into the dual-branch memory enhancement module, which includes a decoupled spatial attention branch, a conditional temporal evolution branch, and a shared memory slot. The two branches interact bidirectionally with the shared memory slot to obtain the enhanced T2WI features and ADC features. The enhanced T2WI features and ADC features are fused by cross-attention to obtain the fused features; The fused features and conditional vectors are input into a hierarchical modulation U-Net, which employs a differentiated modulation strategy based on the depth of the feature map and outputs a predicted feature map. Based on the predicted feature map, a predicted T2WI sequence and an ADC sequence are generated; The calculation method for the read operation of the shared memory slot is as follows: in, For the enhanced feature map after reading, For the input feature map, As a learnable scalar of memory strength, To recover the projection matrix, To query the projection matrix, For a shared memory slot matrix; The write operation update method for the shared memory slot is as follows: in, For the k-th memory slot, For write gating, To write the weights, This is for writing value vectors.

2. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer using MRI images as described in claim 1, characterized in that, The number of shared memory slots ranges from 8 to 32, and the prototype feature dimension ranges from 64 to 256.

3. The MRI image prediction method for neoadjuvant therapy of rectal cancer as described in claim 1, characterized in that, The hierarchical modulation U-Net includes a three-layer downsampling encoder and a three-layer upsampling decoder, for a total of six feature layers. Each feature layer dynamically generates modulation parameters for the corresponding feature layer based on the condition vector through an independent multilayer perceptron.

4. The MRI image prediction method for neoadjuvant therapy of rectal cancer as described in claim 3, characterized in that, The differentiated modulation strategies include: For shallow features, a combination of channel scaling, channel offsetting, and spatial modulation is used for modulation. For mid-level features, modulation is performed using a combination of channel scaling and channel offset. For deep features, modulation is performed using only channel scaling.

5. The MRI image prediction method for neoadjuvant therapy of rectal cancer as described in claim 4, characterized in that, The shallow features include the output features of the encoder's first layer, encoder's second layer, and decoder's third layer; the middle features include the output features of the encoder's third layer and decoder's second layer; and the deep features include the output features of the decoder's first layer.

6. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer using MRI images as described in claim 4, characterized in that, The spatial modulation is achieved through a spatial graph generation network, which includes: a fully connected layer that maps conditional vectors to vectors, reshapes them into three-dimensional tensors, smooths them through 3D convolution, and outputs the spatial modulation graph through a Sigmoid function.

7. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer using MRI images as described in claim 1, characterized in that, The specific method for generating the predicted T2WI sequence and ADC sequence is as follows: the predicted feature map is used to generate a gated map through 3D convolution and the Sigmoid function; the gated map is multiplied element-wise with the normalized original input image to obtain a gated weighted map; the gated weighted map is then added element-wise with the predicted feature map and output through the Sigmoid function.

8. The method for predicting the efficacy of neoadjuvant therapy for rectal cancer using MRI images as described in claim 1, characterized in that, The personalized clinical information includes one or more of the following: demographic information, medical history information, molecular immune and mismatch repair indicators, laboratory blood and inflammatory indicators, and tumor markers.

9. The MRI image prediction method for neoadjuvant therapy of rectal cancer as described in claim 1, characterized in that, The method further includes the step of generating a spatial attention heatmap: fusing the attention weights in the shared memory slot read operation, the spatial attention map of the cross-attention module, and the spatial modulation map of the hierarchical modulation U-Net shallow decoder to generate an overall spatial attention heatmap and visualize it in the prediction sequence.

10. An MRI image prediction system for neoadjuvant therapy of rectal cancer, using the method described in claim 1, characterized in that, include: The data acquisition module is configured to acquire the patient's baseline MRI images, individualized clinical information, and target prediction time points; The condition generation and preliminary feature extraction module is configured to extract features from T2WI and ADC sequences to obtain initial feature maps, and to perform embedding vector processing on the target prediction time points, which are then concatenated with individualized clinical information to generate condition vectors. The dual-branch memory enhancement module includes a decoupled spatial attention branch, a conditional temporal evolution branch, and a shared memory slot. The two branches interact bidirectionally with the shared memory slot to output enhanced T2WI features and ADC features. The cross-attention fusion module is configured to perform cross-attention fusion on the enhanced T2WI features and ADC features to obtain fused features. The hierarchical modulation module is configured to input the fused features and conditional vectors into the hierarchical modulation U-Net and output a predicted feature map. The image prediction module is configured to generate predicted T2WI sequences and ADC sequences based on the predicted feature maps.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • PSMA PET / CT and enhanced MRI multi-modal image registration and fusion method and system

    CN120339343A

  • Dynamic prediction method for rheumatoid arthritis patient based on multi-modal data fusion and deep learning

    CN121054259A