Method and system for predicting time sequence evolution of stereotactic radiotherapy curative effect of brain metastases

Abstract

The invention provides a method and system for predicting time sequence evolution of a brain metastasis tumor stereotactic radiotherapy curative effect, and the method comprises the steps: a time sequence image generation step: inputting the data information of a target brain tumor region into a generative adversarial network, and obtaining a time sequence image corresponding to the data information; a fusion prediction step: extracting image features of the time sequence image to form an image feature sequence, and fusing the image feature sequence with text features extracted by medical text description information in the data information to obtain a fused feature sequence; and inputting the fusion feature sequence into a learning model, and outputting a probability sequence with image content change characteristics at each time point in a future time sequence. According to the method, high-reality and coherent multi-modal MRI prediction images at a plurality of time points in the future can be generated, and a dynamic visual basis is provided for evaluating treatment response.

Description

Technical Field

[0001] This invention belongs to the field of time series data modeling and analysis technology, specifically, it relates to a method and system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases. Background Technology

[0002] In recent years, AI technology has been widely used in the field of medical image-assisted diagnosis and treatment. Deep learning methods such as Convolutional Neural Networks (CNN) have made significant progress in tasks such as brain tumor detection, segmentation, auxiliary judgment, and data augmentation.

[0003] Traditional prognostic assessment of brain tumors relies heavily on the experience and judgment of clinicians, making it highly subjective and difficult to achieve accurate prediction. With the development of AI technology, more and more studies are exploring AI-based methods for predicting the efficacy of brain tumor treatments to improve the scientific rigor and accuracy of treatment decisions. Currently, several studies have applied AI technology to predict and analyze the prognostic effects of brain tumor treatment, covering aspects such as postoperative survival prediction, radiotherapy or chemotherapy sensitivity assessment, and recurrence risk prediction. Among these, preliminary research has been conducted on efficacy prediction models for brain metastases based on multimodal MRI, but these models still face challenges such as limited data scale, insufficient model generalization ability, and challenges in multimodal data fusion. In particular, current research does not integrate multidimensional information such as imaging, pathological immunohistochemical indicators, and clinical medical records to comprehensively explore the evolution of SRS (Survival Restraint) efficacy in brain metastases and the SRS itself.

[0004] Therefore, further exploration of imaging biomarkers related to the evolution of SRS efficacy, combined with new information such as pathological immunohistochemical indicators and clinical medical records, to construct a more efficient, stable, and interpretable predictive method is of great significance for promoting personalized and precise treatment of brain metastases.

[0005] Furthermore, data augmentation is a core research direction in brain tumor image generation algorithms, primarily addressing issues such as limited sample size and imbalanced data distribution in brain tumor image datasets. Among these, brain tumor image generation methods based on Generative Adversarial Networks (GANs) can synthesize high-quality pseudo-image data to enhance the model's generalization ability and optimize algorithm performance. GANs, with their powerful capabilities in image generation tasks, have become the most important data augmentation technique in this field. In natural image generation tasks, several models have already been able to generate detailed, high-quality images, such as DM-GAN.

[39] and DF-GAN

[40] In the field of medical image generation, researchers have also begun to explore methods for joint text-image generation.

[0006] For example, Khwaja et al.

[41] A text-to-image conversion architecture, CELL-E, is proposed, capable of generating 2D probability density maps of intracellular protein distribution based on protein sequences and cell morphology reference images. (Che et al.)

[42] A unified framework is proposed that can simultaneously handle text-to-image, image-to-text, and generation tasks between different medical modalities, namely CT, MRI, and X-ray. This framework achieves efficient alignment and generation of multimodal data and demonstrates superior performance in MRI synthesis and MRI-CT conversion tasks.

[0007] However, current research has failed to fully integrate patients' multidimensional medical information, and studies on time-series image generation for the evolution of SRS treatment efficacy in brain metastases are still lacking. These shortcomings limit the potential application of image generation methods in precision medicine. Therefore, establishing an image generation model that can accurately simulate the growth and evolution of brain metastases after SRS treatment by comprehensively utilizing patients' multidimensional medical information will provide important prior knowledge for treatment plan decisions and radiotherapy planning, and has significant implications and broad prospects in medical applications.

[0008] The references are as follows:

[39] Zhu M, Pan P, Chen W, et al. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 5802-5810.

[40] Tao M, Tang H, Wu S, et al. Deep fusion generative adversarial networks for text-to-image synthesis[J]. arXiv preprint arXiv:2008.05865,2020.

[41] Khwaja E, Song YS, Huang B. CELL-E: Biological Zero-Shot Text-to-Image Synthesis for Protein Localization Prediction[J]. bioRxiv, 2022:2022.05. 27.493774.

[42] Zhan C, Lin Y, Wang G, et al. Medm2g: Unifying medical multi-modal generation via cross-guided diffusion with visual invariant[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. 2024: 11502-11512. Patent document CN120635122A discloses a multi-task automatic glioma segmentation and IDH genotyping method based on SAM: It involves acquiring a multimodal MRI image dataset of gliomas; preprocessing the dataset to obtain a standardized dataset, which is then divided into training and test sets according to a preset ratio; establishing a SAM-based multi-task automatic glioma segmentation and IDH genotyping model, which includes a four-branch image encoder, a feature fusion module, a cue encoder, a mask decoder, and an IDH classifier; inputting the training set into the model for training, setting a joint loss function, and optimizing the model parameters to obtain an optimized model; inputting the test set into the optimized model for segmentation, IDH genotyping, and prediction, obtaining the tumor segmentation accuracy and IDH genotyping accuracy of the optimized model. However, this approach cannot achieve quantitative prediction of patient treatment sensitivity or survival probability estimation.

[0009] This problem urgently needs to be solved. Summary of the Invention

[0010] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases.

[0011] A method for generating time-series images of the evolution of the efficacy of stereotactic radiotherapy for brain metastases according to the present invention includes: a time-series image generation step: inputting data information of the target brain tumor region into a generative adversarial network to obtain a multimodal MRI image corresponding to the data information; Fusion prediction steps: Extract the temporal image features of the multimodal MRI images to form an image feature sequence, and then fuse the image feature sequence with the text features extracted from the medical text description information in the data information to obtain a fused feature sequence; input the fused feature sequence into the learning model to output a probability sequence of image content change characteristics at each time point in the future time series.

[0012] Preferably, in the time-series image generation step, the data information of the target brain tumor region includes graphic input, text input, time series and SRS treatment dosage; the text input includes medical text description information.

[0013] Preferably, the temporal image generator of the generative adversarial network includes: an image encoder, a text encoder, and a binary gate encoder; Based on the graphic input, the output of the image encoder is expressed as: (6) (7) in, This represents the output of the image encoder. , Representing the feature space, , and These represent the length, width, and number of channels of a multimodal MRI image, respectively. This indicates a ResBlock encoding operation. Indicates graphical input. This indicates 5 repetitions of the operation. For average downsampling operation, express Module operations; According to the time series, that is The global text features are output by the text encoder, and the expression is: (8) in, Represents global text features. Indicates a resampling operation; Indicates multilayer sensor operation; This represents an image-to-text encoder; Indicates text input; Based on the SRS treatment dosage and time series, a time-series feature is generated by the binary gated encoder; the time-series feature is then passed through a time-series feature attention module to process the time nodes. Image features superimposed on time nodes The image is then decoded to obtain a time-series generated image. The expression for the time-series generated image is:

[0014] in, This indicates that the image was generated in a time sequence. , , and Representing time nodes Time nodes Time nodes With time nodes Image 2; Represent real numbers; in, The expression is: (9) (10) in, express Convolution operation; express Convolution operation; Indicates feature fusion; express Convolution operation; Represents the convolutional features of the image output; This represents the Transformer features output from the image; This represents the image output features of the ResBlock encoding operation; This indicates a repeated operation.

[0015] Preferably, in the fusion prediction step, the expression for the image feature sequence is: (19) The text features extracted from the medical text description information in the data information are expressed as follows: (18) In the above formula, Represents an image feature sequence; This refers to the text features extracted from medical text description information in the data. and For mutual attention mechanism; The balance parameter representing internal-mutual attention; and For internal attention calculation; where, Indicates The vector obtained by the operation and transformation; subscript Indicates features derived from text; subscript This indicates that the information originates from image features.

[0016] Preferably, in the fusion prediction step, the expression for the fusion feature sequence is: (twenty two) in, Indicates the fused feature sequence; Indicates time node The fusion characteristics; The learning model is the AdaBoost model; (twenty three) in, This represents a probability sequence of image content changes at each point in time within a future time series. Indicates the number of base classifiers. Indicates the ordinal number of the base classifier; Indicates the first One basic classifier; This represents the classifier weight coefficient.

[0017] A system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases, according to the present invention, comprises: Temporal image generation module: Inputs data information of the target brain tumor region into a generative adversarial network to obtain a multimodal MRI image corresponding to the data information; Fusion prediction module: Extracts temporal image features from the multimodal MRI images to form an image feature sequence, and then fuses the image feature sequence with text features extracted from medical text description information in the data information to obtain a fused feature sequence; inputs the fused feature sequence into the learning model, and outputs a probability sequence of image content change characteristics at each time point in the future time series.

[0018] Preferably, in the time-series image generation module, the data information of the target brain tumor region includes graphic input, text input, time series, and SRS treatment dosage; the text input includes medical text description information.

[0019] Preferably, the temporal image generator of the generative adversarial network includes: an image encoder, a text encoder, and a binary gate encoder; Based on the graphic input, the output of the image encoder is expressed as: (6) (7) in, This represents the output of the image encoder. , Representing the feature space, , and These represent the length, width, and number of channels of a multimodal MRI image, respectively. This indicates a ResBlock encoding operation. Indicates graphical input. This indicates 5 repetitions of the operation. For average downsampling operation, express Module operations; According to the time series, that is The global text features are output by the text encoder, and the expression is: (8) in, Represents global text features. Indicates a resampling operation; Indicates multilayer sensor operation; This represents an image-to-text encoder; Indicates text input; Based on the SRS treatment dosage and time series, a time-series feature is generated by the binary gated encoder; the time-series feature is then passed through a time-series feature attention module to process the time nodes. Image features superimposed on time nodes The image is then decoded to obtain a time-series generated image. The expression for the time-series generated image is:

[0020] in, This indicates that the image was generated in a time sequence. , , and Representing time nodes Time nodes Time nodes With time nodes Image 2; Represent real numbers; in, The expression is: (9) (10) in, express Convolution operation; express Convolution operation; Indicates feature fusion; express Convolution operation; Represents the convolutional features of the image output; This represents the Transformer features output from the image; This represents the image output features of the ResBlock encoding operation; This indicates a repeated operation.

[0021] Preferably, in the fusion prediction module, the expression for the image feature sequence is: (19) The text features extracted from the medical text description information in the data information are expressed as follows: (18) In the above formula, Represents an image feature sequence; This refers to the text features extracted from medical text description information in the data. and For mutual attention mechanism; The balance parameter representing internal-mutual attention; and For internal attention calculation; where, Indicates The vector obtained by the operation and transformation; subscript Indicates features derived from text; subscript This indicates that the information originates from image features.

[0022] Preferably, in the fusion prediction module, the expression for the fusion feature sequence is: (twenty two) in, Indicates the fused feature sequence; Indicates time node The fusion characteristics; The learning model is the AdaBoost model; (twenty three) in, This represents a probability sequence of image content changes at each point in time within a future time series. Indicates the number of base classifiers. Indicates the ordinal number of the base classifier; Indicates the first One basic classifier; This represents the classifier weight coefficient.

[0023] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention provides a time-series generation model that integrates multimodal information, which can generate highly realistic and coherent multimodal MRI prediction images at multiple future time points, providing dynamic visual evidence for assessing treatment response.

[0024] 2. This invention provides an intelligent decision-making model based on time-series predictive images. By fusing radiomics features and clinical text features, and using hidden Markov chains to model the evolution process, it achieves quantitative prediction of patient treatment sensitivity and survival probability estimation.

[0025] 3. This invention provides a method for generating time-series images of the therapeutic evolution of brain metastases based on generative adversarial networks, which solves the problem that existing technologies cannot prospectively predict and visualize the continuous growth and evolution of tumors after stereotactic radiotherapy. Attached Figure Description

[0026] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the SRS intelligent decision-making model for brain metastases provided by the present invention. Detailed Implementation

[0027] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0028] A method for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases, provided by the present invention, includes: First, we collected and established a multidimensional medical information standard database for brain metastases SRS, which includes multimodal MRI imaging data, pathological immunohistochemical indicators of the primary lesion, patient clinical information and clinical treatment plans, to provide high-quality data support for subsequent research. Next, by quantitatively analyzing the differences in sensitivity to SRS treatment among different individuals, we explored imaging biomarkers that are closely related to the evolution of SRS efficacy in brain metastases.

[0029] Then, based on multidimensional medical information, a time-series image generation model for the evolution of SRS efficacy was constructed, and corresponding time-series MRI images were generated.

[0030] Finally, by combining time-series MRI images, imaging biomarkers, and multidimensional medical information, an intelligent decision-making model for SRS in brain metastases was constructed to assist clinicians in screening patients who are not sensitive to SRS treatment and optimizing radiotherapy strategies.

[0031] To address the challenge of predicting and visualizing tumor growth and evolution in SRS treatment of brain metastases, this invention proposes to introduce temporal description and generative adversarial methods to construct a temporal image generation model for SRS efficacy evolution, namely EE-TGAN. Figure 1 As shown.

[0032] 1. Network Input: EE-TGAN input consists of three parts: graphical input... Text input and SRS treatment dosage and time series .

[0033] The graphical input This includes segmentation masks obtained from multimodal MRI and SVD-Net.

[0034] The text input Includes: pathological immunohistochemical indicators and clinical case text information.

[0035] This invention trains a time-series image generator. Prospective multimodal MRI images of brain metastases generated 3–12 months after SRS treatment, i.e., sequentially generated images. . ,Right now This indicates graphical input.

[0036] Then, to and paired sample real images from the training database By using a discriminator To compare the realism of the images.

[0037] Finally, time series image generator and discriminator During training, the game is continuously played until a Nash equilibrium is reached, thus completing the image generation task.

[0038] 2. EE-TGAN generator, specifically, a time-series image generator. The encoding section consists of an image encoder, a text encoder, and a binary gate encoder. The input to the image encoder is... It consists of five residual coding modules, namely ResBlock and one Encoding module composition. Image encoder output. It can be obtained from the following equations (6) and (7): (6) (7) in, This indicates 5 repetitions of the operation. For average downsampling operation, express Module operations.

[0039] The input to the text encoder is , Text features are obtained through feature extraction using a pre-trained CLIP. Features are extracted from the penultimate layer of the frozen CLIP feature extractor, and an MLP layer is applied to obtain global text features. It is obtained by the following formula (8): (8) in, Represents global text features. Indicates a resampling operation; Indicates multilayer sensor operation; This represents an image-to-text encoder; Indicates text input; SRS therapeutic dose, denoted as and time series, denoted as Then, through a binary gated encoder and the decoding part of the generator, intermediate features can be converted into temporal features, and the temporal feature attention module can be used to process the time nodes. The image Features overlaid on time nodes Image This ensures that the generated time series images remain consistent.

[0040] Finally, a time-generated image is obtained through a decoder consisting of a simple 5-layer convolutional neural network. , It can be obtained from the following equations (9) and (10): (9) (10) 3. EE-TGAN Discriminator. This invention uses ResNet50 as the discriminator for the image authenticity task. The discriminator's role is to generate more realistic fused images. The input to ResNet50... Defined as real images of samples in the training database And generate images, i.e., generate images in sequence. Image pairs. pass After feature extraction from each Sage module, image discrimination features are obtained. It can be expressed by the following formula (11): (11) in, , hour, ; in the formula middle, Indicates the ordinal number of the Sage module. Indicates the number of Sage modules. express Image identification features obtained by each Sage module; express Sage module operations.

[0041] then, Using global average pooling and a single layer consisting of 1024 neurons After the first layer, the result is output through a binary classification linear layer. This process can be represented by the following equation (12): (12) in, Indicates the judgment result. Represents a linear classifier; Indicates fully connected layer operations; This indicates a global average pooling operation.

[0042] Finally, this invention determines the final generator weights and temporally generated images by training ResNet50 until the network can no longer distinguish between real and fake images.

[0043] 4. EE-TGAN Loss Function. EE-TGAN performs adversarial generation by minimizing the generator and maximizing the discriminator. During this process, a significant loss function is optimized to target the desired outcome. Includes texture loss and similarity loss It can be expressed by the following equations (13) and (14): (13) (14) in, These are deep convolutional features of the generator encoder. These are the convolutional features at the bottom layer of the discriminator. and It is a pixel. In the The grayscale value of each modality and the pixel grayscale value of that point in the generated image. and It involves calculating the mean and variance. express function, Represents the 1-norm operation; This indicates the operation of computational structure measurement indicators; and It is a small constant added to prevent the denominator from approaching zero.

[0044] Due to the discriminator There are only two classifications, true and false, i.e., [0,1]. Therefore, this invention uses the binary cross-entropy loss function as its loss function, expressed by the following equation (15): (15) in, Represents the loss function. Indicates the number of samples; Indicates sample The label is 1 for positive classes and 0 for negative classes; Indicates sample Category The predicted probability.

[0045] A robust intelligent decision-making model for SRS (Surgery-Related Syndrome) in brain metastases. This invention aims to develop a robust intelligent decision-making model for SRS by utilizing multi-dimensional medical information, including pre- and post-SRS treatment time-series MRI images, segmented tumor masks, pathological and immunohistochemical markers of the patient's primary lesion, and clinical case-treatment protocol text. Figure 1 As shown, this provides physicians with prospective SRS efficacy assessment and AI decision-making, thereby improving patients' quality of life and prognosis.

[0046] Specifically, this model can analyze the correlation between SRS and treatment sensitivity, and analyze treatment outcomes at different time points within 3 to 12 months using the Hidden Markov Chain method. Based on survival probability, it ultimately determines whether a patient is suitable for SRS, providing doctors with prior knowledge and scientific basis for developing the best treatment plan. The model construction process includes: Firstly, multidimensional medical feature extraction and feature fusion under SRS treatment. Based on previous research, this invention aims to generate a time-series image of the evolution of SRS treatment efficacy in brain metastases using EE-TGAN. The time-series images are generated and segmented into lesions using NMU-NAS to obtain a segmentation mask, denoted as . .

[0047] Then, for each time-series multimodal MRI image, pydiomics is used in... We obtained radiomics features and used a biomarker to screen for time-series image features of biological markers associated with the efficacy of SRS. CLIP text feature extraction was performed on the pathological immunohistochemical indicators of the primary lesion and clinical medical record information to obtain text features. .

[0048] Image features Text features Feature fusion enables better encoding of mutual information between images of different modalities, and allows for advanced diagnosis using other non-image information about tumors. The feature fusion process is performed through a bidirectional multidimensional feature fusion module, namely BMFM, to obtain a high-level fused representation of multi-domain features.

[0049] The input to the bidirectional multidimensional feature fusion module is and ,pass Operation converted The vector can be represented by the following equations (16) and (17): (16) (17) in, and These represent linear projection and layer normalization, respectively.

[0050] Output of the bidirectional multidimensional feature fusion module The following equations (18) and (19) are used to calculate: (18) (19) ( , (20) in, and Calculations for internal attention; and For mutual attention mechanism; The parameter representing the balance between internal and mutual attention. 0.5; It can be obtained from the following formula (21): (twenty one) in, Indicates the dimension of the Key vector; =8.

[0051] Secondly, SRS therapy sensitivity prediction. For brain tumor patients, corresponding fusion features can be obtained under SRS treatment. It is expressed as the following formula (22): (twenty two) This invention uses the AdaBoost ensemble learning method to train these classifiers as "additive models" to predict the sensitivity of SRS treatment at different time periods.

[0052] In the AdaBoost ensemble learning algorithm, the input is a fused feature vector matrix. The output is then defined as a binary classification problem; SRS treatment sensitivity is defined as 0, and SRS treatment insensitivity is defined as 1, i.e. It also outputs the probability of sensitivity to SRS treatment. ; Each basic classifier is defined as... The linear combination of the output probabilities of the base classifiers is As shown in equation (23): (twenty three) in, This represents a probability sequence of image content changes at each point in time within a future time series. Indicates the number of base classifiers. Indicates the ordinal number of the base classifier; Indicates the first One basic classifier; This represents the classifier weight coefficient.

[0053] Third, an SRS intelligent decision-making model is constructed based on Hidden Markov Chains. This invention defines SRS intelligent decision-making as a Hidden Markov Model (HMM) problem. Patients with brain metastases have two states: survival and death. The time-series SRS treatment sensitivity prediction result, i.e., sensitive or insensitive, is abstracted as a discrete Markov chain.

[0054] Therefore, the parameters of this hidden Markov model can be defined as follows (24): (twenty four) This invention sets the model and observation sequence Based on the preceding research, it is known that calculations in the model... The set of a certain state probability of occurrence That is, the parameters of the AI ​​decision-making model. , H , Known to calculate the probability of final patient survival. .

[0055] Development and Evaluation of an Intelligent Decision-Making Platform for Brain Metastases via SRS. This invention aims to develop an intelligent decision-making software platform for brain metastases based on prior software development. The software employs a client / server architecture deployment method. It constructs and integrates a precise segmentation model for brain metastases, a time-series image generation model for SRS efficacy evolution, and an intelligent decision-making model for brain metastases. A user-friendly, interactive graphical interface is developed, and test data is collected for validation.

[0056] This software can provide new perspectives and practical tools for monitoring brain metastases, evaluating treatment efficacy, and assisting in clinical diagnosis and treatment, thereby further improving the prognosis of patients with brain metastases.

[0057] The evaluation and validation process includes: assuming the input sample is not clinical MRI multimodal image data, the model input is obtained after data standardization. .

[0058] first, The metastatic tumors were segmented using a precise segmentation model, namely NMU-NAS. and according to Select the tumor from this case area . This indicates an element-wise product operation.

[0059] Next, the patient's pathological immunohistochemical indicators, clinical case information, and treatment plan text information were also standardized into structured data. and will As input to the time-series image generation model for SRS efficacy evolution, it generates time-series SRS efficacy prediction images, i.e., time-series generated images. , =1, 2, 3, 4.

[0060] Transform the time series image into a time series generated image. Pathological immunohistochemical indicators, clinical case texts, and treatment plans are fed into the brain metastasis SRS intelligent decision-making model to complete the segmentation and localization of lesions in patients with brain metastases and to obtain prospective SRS intelligent decision-making results.

[0061] Finally, the performance of the model of this invention was evaluated by statistically analyzing and calculating evaluation metrics such as Dice, accuracy, specificity, sensitivity, and F1-Score.

[0062] The present invention also provides a system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases. The system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases can be implemented by executing the process steps of the method for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases. That is, those skilled in the art can understand the method for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases as a preferred embodiment of the system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases.

[0063] A system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases, according to the present invention, comprises: Temporal image generation module: Inputs data information of the target brain tumor region into a generative adversarial network to obtain a multimodal MRI image corresponding to the data information; Fusion prediction module: Extracts temporal image features from the multimodal MRI images to form an image feature sequence, and then fuses the image feature sequence with text features extracted from medical text description information in the data information to obtain a fused feature sequence; inputs the fused feature sequence into the learning model, and outputs a probability sequence of image content change characteristics at each time point in the future time series.

[0064] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0065] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases, characterized in that, include: The time-series image generation steps are as follows: the data information of the target brain tumor region is input into a generative adversarial network to obtain a multimodal MRI image corresponding to the data information; Fusion prediction steps: Extract the temporal image features of the multimodal MRI images to form an image feature sequence, and then fuse the image feature sequence with the text features extracted from the medical text description information in the data information to obtain a fused feature sequence; input the fused feature sequence into the learning model to output a probability sequence of image content change characteristics at each time point in the future time series.

2. The method for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases according to claim 1, characterized in that, In the time-series image generation step, the data information of the target brain tumor region includes graphic input, text input, time series, and SRS treatment dosage; the text input includes medical text description information.

3. The method for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases according to claim 2, characterized in that, The temporal image generator of the generative adversarial network includes: an image encoder, a text encoder, and a binary gate encoder; Based on the graphic input, the output of the image encoder is expressed as: (6) (7) in, This represents the output of the image encoder. , Representing the feature space, , and These represent the length, width, and number of channels of a multimodal MRI image, respectively. This indicates a ResBlock encoding operation. Indicates graphical input. This indicates 5 repetitions of the operation. For average downsampling operation, express Module operations; According to the time series, that is The global text features are output by the text encoder, and the expression is: （8） in, Represents global text features. Indicates a resampling operation; Indicates multilayer sensor operation; This represents an image-to-text encoder; Indicates text input; Based on the SRS treatment dosage and time series, a time-series feature is generated by the binary gated encoder; the time-series feature is then passed through a time-series feature attention module to process the time nodes. Image features superimposed on time nodes The image is then decoded to obtain a time-series generated image. The expression for the time-series generated image is: in, This indicates that the image was generated in a time sequence. , , and Representing time nodes Time nodes Time nodes With time nodes Image 2; Represent real numbers; in, The expression is: (9) (10) in, express Convolution operation; express Convolution operation; Indicates feature fusion; express Convolution operation; Represents the convolutional features of the image output; This represents the Transformer features output from the image; This represents the image output features of the ResBlock encoding operation; This indicates a repeated operation.

4. The method for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases according to claim 1, characterized in that, In the fusion prediction step, the expression for the image feature sequence is: (19) The text features extracted from the medical text description information in the data information are expressed as follows: (18) In the above formula, Represents an image feature sequence; This refers to the text features extracted from medical text description information in the data. and For mutual attention mechanism; The balance parameter representing internal-mutual attention; and For internal attention calculation; where, Indicates The vector obtained by the operation and transformation; subscript Indicates features derived from text; subscript This indicates that the information originates from image features.

5. The method for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases according to claim 4, characterized in that, In the fusion prediction step, the expression for the fusion feature sequence is: (22) in, Indicates the fused feature sequence; Indicates time node The fusion characteristics; The learning model is the AdaBoost model; (23) in, This represents a probability sequence of image content changes at each point in time within a future time series. Indicates the number of base classifiers. Indicates the ordinal number of the base classifier; Indicates the first One basic classifier; This represents the classifier weight coefficient.

6. A system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases, characterized in that, include: Temporal image generation module: Inputs data information of the target brain tumor region into a generative adversarial network to obtain a multimodal MRI image corresponding to the data information; Fusion prediction module: Extracts temporal image features from the multimodal MRI images to form an image feature sequence, and then fuses the image feature sequence with text features extracted from medical text description information in the data information to obtain a fused feature sequence; inputs the fused feature sequence into the learning model, and outputs a probability sequence of image content change characteristics at each time point in the future time series.

7. The system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases according to claim 6, characterized in that, In the time-series image generation module, the data information of the target brain tumor region includes graphic input, text input, time series, and SRS treatment dosage; the text input includes medical text description information.

8. The system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases according to claim 7, characterized in that, The temporal image generator of the generative adversarial network includes: an image encoder, a text encoder, and a binary gate encoder; Based on the graphic input, the output of the image encoder is expressed as: (6) (7) in, This represents the output of the image encoder. , Representing the feature space, , and These represent the length, width, and number of channels of a multimodal MRI image, respectively. This indicates a ResBlock encoding operation. Indicates graphical input. This indicates 5 repetitions of the operation. For average downsampling operation, express Module operations; According to the time series, that is The global text features are output by the text encoder, and the expression is: （8） in, Represents global text features. Indicates a resampling operation; Indicates multilayer sensor operation; This represents an image-to-text encoder; Indicates text input; Based on the SRS treatment dosage and time series, a time-series feature is generated by the binary gated encoder; the time-series feature is then passed through a time-series feature attention module to process the time nodes. Image features superimposed on time nodes The image is then decoded to obtain a time-series generated image. The expression for the time-series generated image is: in, This indicates that the image was generated in a time sequence. , , and Representing time nodes Time nodes Time nodes With time nodes Image 2; Represent real numbers; in, The expression is: (9) (10) in, express Convolution operation; express Convolution operation; Indicates feature fusion; express Convolution operation; Represents the convolutional features of the image output; This represents the Transformer features output from the image; This represents the image output features of the ResBlock encoding operation; This indicates a repeated operation.

9. The system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases according to claim 6, characterized in that, In the fusion prediction module, the expression for the image feature sequence is: (19) The text features extracted from the medical text description information in the data information are expressed as follows: (18) In the above formula, Represents an image feature sequence; This refers to the text features extracted from medical text description information in the data. and For mutual attention mechanism; The balance parameter representing internal-mutual attention; and For internal attention calculation; where, Indicates The vector obtained by the operation and transformation; subscript Indicates features derived from text; subscript This indicates that the information originates from image features.

10. The system for predicting the temporal evolution of the efficacy of stereotactic radiotherapy for brain metastases according to claim 9, characterized in that, In the fusion prediction module, the expression for the fusion feature sequence is: (22) in, Indicates the fused feature sequence; Indicates time node The fusion characteristics; The learning model is the AdaBoost model; (23) in, This represents a probability sequence of image content changes at each point in time within a future time series. Indicates the number of base classifiers. Indicates the ordinal number of the base classifier; Indicates the first One basic classifier; This represents the classifier weight coefficient.

Images