Liver cancer longitudinal recurrence prediction and treatment effect evaluation system based on multi-modal fusion

Abstract

The invention belongs to the technical field of medical data processing, and provides a multimodal fusion-based liver cancer longitudinal relapse prediction and treatment effect evaluation system, which comprises a data set construction module, which is used for forming a longitudinal queue data set by using longitudinal queue data samples of a plurality of patients, and setting a relapse time label for each longitudinal queue data sample; the training module is used for training a recurrence network by utilizing the longitudinal queue data set to obtain a recurrence model; the recurrence prediction module inputs the to-be-predicted longitudinal queue data of the patient into the recurrence model to obtain a prediction result of each treatment time point, and the prediction result comprises the recurrence probability of more than one future time period; the curative effect evaluation module is used for acquiring simulation longitudinal queue data corresponding to different treatment modes selected by the to-be-evaluated patient at the current relapse time point; and inputting the simulated longitudinal queue data into the recurrence model to obtain a prediction result of each treatment time point. According to the method, the accuracy and generalization of the recurrence model are improved, and doctors are accurately and efficiently assisted in selecting treatment modes.

Description

Technical Field

[0001] This invention relates to the field of medical data processing technology, and in particular to a system for predicting longitudinal recurrence of liver cancer and evaluating treatment efficacy based on multimodal fusion. Background Technology

[0002] Liver cancer recurrence after treatment is a challenging problem for clinicians, and deep learning technology has made some breakthroughs in predicting liver cancer recurrence. Multimodal data can provide more comprehensive information for predicting liver cancer recurrence. Related technologies for predicting liver cancer recurrence (such as Chinese patent CN117612711A) integrate multiple modalities of data, including clinical text, imaging, and pathology, and construct multimodal prediction models based on these data and fusion strategies to improve the accuracy of liver cancer recurrence prediction. However, these technologies do not consider the relationship between the natural progression of the disease and the overall duration of the disease, as well as the patient's current sensitivity to treatment methods. Therefore, there is still room for improvement in the accuracy of liver cancer recurrence prediction. Furthermore, existing multimodal datasets and multimodal model construction have stricter requirements for data inclusion and exclusion criteria, resulting in a smaller usable dataset size, which contradicts the requirement of deep learning for large datasets. In addition, cases with missing modalities cannot be included in the dataset, leading to poor generalization of multimodal models.

[0003] In technologies related to evaluating the efficacy of liver cancer treatment, to more accurately assess the effectiveness of liver cancer treatment, medical data from patients who have not undergone other treatments (i.e., patients who have only received one type of liver cancer treatment) are often included in the dataset for model training and evaluation. However, in reality, liver cancer patients experience multiple relapses and treatment changes throughout their disease course, and their sensitivity to different treatment methods varies. Existing liver cancer treatment efficacy evaluation models based on a single treatment method are difficult to meet the requirements of assisting doctors in evaluating treatment efficacy and selecting treatment plans. Summary of the Invention

[0004] This application aims to at least address the technical problems existing in the prior art and provide a system for predicting longitudinal recurrence of liver cancer and evaluating treatment efficacy based on multimodal fusion.

[0005] The multimodal fusion-based system for predicting longitudinal recurrence and evaluating treatment efficacy in liver cancer provided in this application includes: The dataset construction module uses longitudinal cohort data samples from multiple patients to form a longitudinal cohort dataset, and sets a relapse time label for each longitudinal cohort data sample. The longitudinal cohort data samples of patients are generated in chronological order based on multimodal data and time covariates of patients at multiple treatment time points. The time covariates of each treatment time point include the time since the first treatment and the time since the last treatment. The multimodal data of each treatment time point includes the treatment method and the preceding treatment method at that treatment time point. The training module uses a longitudinal queue dataset to train the relapse network and obtain a relapse model. The relapse prediction module inputs the patient's longitudinal cohort data to be predicted into the relapse model to obtain the prediction results for each treatment time point in the longitudinal cohort data to be predicted. The prediction results include the relapse probability for more than one future time period. The efficacy assessment module obtains simulated longitudinal cohort data corresponding to different treatment methods selected by the patient to be assessed at the current relapse time point; the simulated longitudinal cohort data corresponding to each treatment method is input into the relapse model to obtain the prediction results of each treatment time point in the simulated longitudinal cohort data corresponding to that treatment method.

[0006] The beneficial technical effects of this application are as follows: 1. This application introduces a time covariate for each treatment time point into the input of the relapse model, i.e., the multimodal model. The time since the first treatment reflects the natural progression of the disease and the overall duration of the illness; the time since the last treatment reflects the "freshness" of recent treatment interventions, which affects the patient's sensitivity to treatment methods. Using the time covariate for each treatment time point along with the multimodal data as input to the relapse model improves the accuracy of relapse prediction. Furthermore, the multimodal data for each treatment time point includes not only the treatment method at that time point but also preceding treatment methods, fully considering the impact of preceding treatment methods on relapse prediction. This allows the relapse model to more comprehensively represent the patient's actual condition, thereby further improving the accuracy of the relapse model's output prediction results. 2. The recurrence model of this application can predict multiple treatment methods without having to build a separate recurrence model for each treatment method. This simplifies the model structure, enables more efficient recurrence prediction, and is applicable to patients undergoing initial treatment and those who have undergone more than one treatment. 3. This application can simultaneously predict all treatment time points in the longitudinal cohort data to be predicted and the simulated longitudinal cohort data, and obtain the recurrence probability of more than one future time period for each treatment time point. Compared with the prediction of a single time point before treatment in related technologies, this application considers the information of historical treatment time points, and can make more comprehensive recurrence prediction. 4. Compared with the strict inclusion and exclusion requirements of previous prediction models for single treatment options, this application includes all historical relapse and treatment data of the same patient in the construction of the longitudinal cohort dataset, which can greatly expand the dataset size and is more in line with the actual treatment in clinical practice, thus improving the generalization of the relapse model. 5. This application utilizes a recurrence model to predict the recurrence probability of each treatment method when a patient with liver cancer chooses a different treatment, so as to accurately and efficiently assist doctors in making multiple treatment choices. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the structure of a liver cancer longitudinal recurrence prediction and treatment efficacy evaluation system based on multimodal fusion in a preferred embodiment of the present invention; Figure 2 This is a schematic diagram of the recurrence network in a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a long short-term memory network in a preferred embodiment of the present invention; Figure 4 This is a schematic diagram of the decision module in a preferred embodiment of the present invention. Detailed Implementation

[0008] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0009] This invention discloses a multimodal fusion-based system for predicting longitudinal recurrence of liver cancer and evaluating treatment efficacy. The execution carrier of this system includes, but is not limited to, a server and electronic devices such as terminals and computers.

[0010] In a preferred embodiment, please see Figure 1 A multimodal fusion-based system for predicting longitudinal recurrence and evaluating treatment efficacy in liver cancer includes: The dataset construction module uses longitudinal cohort data samples from multiple patients to form a longitudinal cohort dataset, and sets a relapse time label for each longitudinal cohort data sample. The longitudinal cohort data samples of patients are generated in chronological order based on multimodal data and time covariates of patients at multiple treatment time points. The time covariates of each treatment time point include the time since the first treatment and the time since the last treatment. The multimodal data of each treatment time point includes the treatment method and the preceding treatment method at that treatment time point. The training module uses a longitudinal queue dataset to train the relapse network and obtain a relapse model. The relapse prediction module inputs the patient's longitudinal cohort data to be predicted into the relapse model to obtain the prediction results for each treatment time point in the longitudinal cohort data. The prediction results include the relapse probability for more than one future time period. The efficacy assessment module obtains simulated longitudinal cohort data corresponding to different treatment methods selected by the patient to be assessed at the current relapse time point; the simulated longitudinal cohort data corresponding to each treatment method is input into the relapse model to obtain the prediction results of each treatment time point in the simulated longitudinal cohort data corresponding to that treatment method.

[0011] The following describes the working principle of the dataset construction module, which involves assembling a longitudinal cohort dataset using longitudinal cohort data samples from multiple patients: Step S11: Record multimodal data and temporal covariates, as well as recurrence time, for multiple patients at different treatment time points. This is not limited to recording the above information through regular follow-up. The multimodal data for each treatment time point includes imaging information and text data; the text data includes the treatment method and prior treatment methods at each treatment time point, as well as the patient's clinical records and / or laboratory test data.

[0012] For example, the imaging information is not limited to CT images (computed tomography images) or MRI images (magnetic resonance imaging images) of the patient's upper abdomen, chest, abdomen or other parts.

[0013] For example, the treatment methods are not limited to the following five: liver resection, ablation therapy, interventional therapy, radiotherapy, and systemic anti-tumor therapy. For ease of text model processing, different treatment methods are represented by different numerical values, such as: liver resection: 1; ablation therapy: 2; interventional therapy: 3; radiotherapy: 4; systemic anti-tumor therapy: 5. Preferably, for more accurate recurrence prediction and treatment efficacy evaluation, each of the above treatment methods can be further subdivided, such as dividing ablation therapy into radiofrequency ablation, microwave ablation, laser ablation, etc., and subdividing systemic anti-tumor therapy into targeted therapy and immunotherapy, etc. The multimodal data for each treatment time point includes the treatment method and prior treatment methods at that time point. The prior treatment method represents the treatment method of the patient's previous treatment. When the patient has no prior treatment and this is the first treatment, the prior treatment method can be represented as 0. When there is prior treatment at a certain treatment time point, the prior treatment method is represented according to the above numerical representation rules.

[0014] For example, clinical records include the patient's age, sex, weight, body mass index (BMI), cirrhosis, and hepatitis B virus infection status.

[0015] For example, laboratory test data are not limited to the concentration of alpha-fetoprotein, total bilirubin, albumin, alanine aminotransferase activity, aspartate aminotransferase activity, platelet count, gamma-glutamyl transferase, and thrombin time in the patient's blood.

[0016] For example, the time covariates for each treatment time point include the time since the first treatment and the time since the last treatment. The time since the first treatment for each treatment time point is the time difference between that treatment time point and the patient's first treatment time, and can be in months. It reflects the natural progression of the disease and the overall duration of the disease, and is an important covariate for relapse risk. The time since the last treatment for each treatment time point is the time difference between that treatment time point and the patient's last treatment time, and can be in days. It reflects the "freshness" of recent treatment interventions and affects sensitivity to treatment methods. If a treatment time point is the first treatment, then both the time since the first treatment and the time since the last treatment for that treatment time point are set to 0.

[0017] Step S12: For each patient, integrate the multimodal data and temporal covariates from more than one treatment time point according to the chronological order of their treatment time points to obtain the longitudinal cohort data sample for that patient. It should be noted that a patient may have more than one longitudinal cohort data sample.

[0018] For example, suppose a longitudinal queue data sample has Each treatment time point, If is a non-negative integer, then a sample of data in a vertical queue can be represented as: .

[0019] in, , , These represent the 0th and 1st treatment time points in the longitudinal cohort data sample, respectively. The treatment time point, the first Each treatment time point, ; , , These represent the 0th and 1st treatment time points in the longitudinal cohort data sample, respectively. The treatment time point, the first Multimodal data at each treatment time point , , These represent the 0th and 1st treatment time points in the longitudinal cohort data sample, respectively. The treatment time point, the first The time covariates at each treatment time point. Indicates the first Multimodal data at each treatment time point and the Temporal covariates at each treatment time point Related, .

[0020] In this embodiment, "vertical" refers to the direction of time development. A longitudinal cohort data sample refers to the entire longitudinal treatment process of a patient from the first diagnosis of liver cancer to the endpoint (cure / death / end of follow-up). Multimodal data and time covariates are collected at each treatment time point, and the data from multiple treatment time points constitute the longitudinal cohort dataset.

[0021] Step S13: Set recurrence time tags for the longitudinal cohort data samples obtained in step S12. In step S11, the patient's recurrence time was recorded synchronously. There can be more than one recurrence time, indicating that the patient has relapsed more than once.

[0022] In step S13, firstly, a relapse time needs to be assigned to each treatment time point in the longitudinal queue data sample. Specifically, following the chronological order, the relapse time that is closest to each treatment time point is taken as the relapse time corresponding to that treatment time point. Next, a relapse time sub-label is set for each treatment time point in the longitudinal queue data sample. Finally, the relapse time sub-labels of all treatment time points in the longitudinal queue data sample are combined according to the chronological order of the treatment time points to obtain the relapse time label of the longitudinal queue data sample.

[0023] For example, suppose the first data sample in the vertical queue is... The recurrence time sub-label of the treatment time is , This indicates the number of future time periods set in the prediction results, i.e., the number of classification categories. It is a positive integer; Indicates the first data sample in the vertical queue The treatment time point is at the The recurrence label value for the nth future time period, when the nth The relapse time corresponding to the treatment time point falls into the first... For a future time period, then When the first If the relapse time corresponding to each treatment time point does not fall into the first future time period, then ; Indicates the first data sample in the vertical queue The treatment time point is at the The recurrence label value for the nth future time period, when the nth The relapse time corresponding to the treatment time point falls into the first... For a future time period, then When the first The relapse time corresponding to the treatment time point did not fall into the [number missing] [section missing]. For a future time period, then ; Indicates the first data sample in the vertical queue The treatment time point is at the The recurrence label value for the nth future time period, when the nth The relapse time corresponding to the treatment time point falls into the first... For a future time period, then When the first The relapse time corresponding to the treatment time point did not fall into the [number missing] [section missing]. For a future time period, then .

[0024] In a preferred embodiment, the system further includes a missing data imputation module, which imputes missing data in patient clinical records and / or laboratory test data, enabling the system to effectively predict recurrence even if text data is missing. The missing data imputation module imputes missing data in patient clinical records and / or laboratory test data collected during step S11 of the dataset construction module. The missing data imputation module is not limited to imputation based on statistics (such as mean, median, etc.) or regression imputation methods. Preferably, the missing data imputation module uses Recursive Imputation for Time Series (RITS) or Bidirectional Recurrent Imputation for Time Series (BRITS) to complete the missing clinical records and / or laboratory test data, achieving missing data imputation based on longitudinal historical data (data from different treatment time points) and lateral data (data from the same treatment time point).

[0025] The following section introduces the network structure of the relapse model, namely the relapse network. Please see... Figure 2 The recurrence network includes: The multimodal feature extraction module obtains the multimodal joint representation vector for each treatment time point based on the multimodal data at that treatment time point. When processing longitudinal queue data samples, longitudinal queue data to be predicted, or simulated longitudinal queue data, the multimodal feature extraction module obtains the multimodal joint representation vector for each treatment time point based on the multimodal data at each treatment time point in the longitudinal queue data samples, longitudinal queue data to be predicted, or simulated longitudinal queue data. The temporal feature embedding module obtains a temporal context embedding vector based on the temporal covariates of each treatment time point. When processing longitudinal queue data samples, longitudinal queue data to be predicted, or simulated longitudinal queue data, the temporal feature embedding module obtains the temporal context embedding vector of each treatment time point based on the temporal covariates of each treatment time point in the longitudinal queue data samples, longitudinal queue data to be predicted, or simulated longitudinal queue data. The case regression module obtains the historical perceived state vector for each treatment time point based on the multimodal temporal context joint representation vector. The multimodal temporal context joint representation vector for each treatment time point is obtained by concatenating the multimodal joint representation vector and the temporal context embedding vector for that treatment time point. When processing longitudinal cohort data samples, longitudinal cohort data to be predicted, or simulated longitudinal cohort data, the case regression module obtains the historical perceived state vector for each treatment time point based on the multimodal temporal context joint representation vector of each treatment time point in the longitudinal cohort data samples, longitudinal cohort data to be predicted, or simulated longitudinal cohort data. The decision module obtains the prediction result for each treatment time point based on the historical perception state vector of each treatment time point. When processing longitudinal queue data samples, longitudinal queue data to be predicted, or simulated longitudinal queue data, the decision module obtains the prediction result for each treatment time point based on the historical perception state vector of each treatment time point in the longitudinal queue data samples, longitudinal queue data to be predicted, or simulated longitudinal queue data.

[0026] In this embodiment, multiple treatment time points can share one multimodal feature extraction module, one temporal feature embedding module, and one decision module. Alternatively, each treatment time point can use a separate multimodal feature extraction module, one temporal feature embedding module, and one decision module, which would require setting up multiple multimodal feature extraction modules, multiple temporal feature embedding modules, and multiple decision modules. Preferably, multiple multimodal feature extraction modules share network parameters (i.e., share weights). Multiple decision modules sharing network parameters (i.e., share weights) ensures consistent decision logic.

[0027] In this embodiment, please see Figure 2 The multimodal feature extraction module includes: An image encoder obtains an image vector for each treatment time point based on the image information at that time point. Specifically, the image information for each treatment time point is input into the image encoder, and the image encoder outputs the image vector for that treatment time point. For example, the image encoder is not limited to image encoders using existing visual transformers or contrastive language-image pre-trained models (CLIP). A visual transformer, also called a visual self-attention model, is a VisionTransformer (ViT).

[0028] For example, when an image encoder uses a vision transformer, the process by which the vision transformer processes CT or MRI images at a certain treatment time point follows a general procedure, consisting of the following steps: ① Preprocessing: Adjust each medical image (i.e., CT image or MRI image) to a uniform size (e.g., 224×224 pixels); perform normalization processing (e.g., scale pixel values ​​to the range of [0,1]).

[0029] ② Image segmentation: Each medical image is divided into image blocks of a fixed size. Each image block is convolved using a two-dimensional convolution with a kernel size that is the same as the image block size to obtain a one-dimensional vector. In addition, a learnable vector, denoted as cls, is added to each medical image.

[0030] ③Location coding: Encode the one-dimensional vector obtained from each image block of each medical image and the cls of each medical image using the existing sine-cosine location coding; ④ Transformer encoder processing: Input all vectors obtained from step ③ into the Transformer encoder. From the Transformer encoder output, only the feature vector corresponding to cls is selected as the global feature vector for each medical image. (CT or MRI images contain...) Zhang's medical images will be obtained A global feature vector, The image vector at this treatment time point is composed of several global feature vectors. . It is a positive integer. , , These respectively represent the first medical image, the second medical image, and the third medical image in a CT or MRI image. Global feature vectors of medical images.

[0031] The text encoder obtains the text vector for each treatment time point based on the text data at that time point; specifically, the text data for each treatment time point is input into the text encoder, and the text encoder outputs the text vector for that treatment time point.

[0032] For example, the text encoder is not limited to text encoders using existing BERT models or contrastive language-image pre-trained models (CLIP). The BERT model is a bidirectional encoder representation model. The text data for each treatment time point includes the treatment method, patient clinical records, or laboratory test data, and there are multiple texts. With a text encoder using a BERT or CLIP model, each text can be encoded into an independent feature vector. For example, the treatment method at that treatment time point yields a feature vector after being encoded by the text encoder, and the blood pressure at that treatment time point also yields a feature vector. The feature vectors of all texts together constitute the text vector for that treatment time point.

[0033] The multimodal feature fusion module fuses the image vector and text vector at each treatment time point to obtain the multimodal joint representation vector at that treatment time point.

[0034] In this embodiment, the multimodal feature fusion module is not limited to using a concatenation function or cross-attention mechanism to fuse the image vector and text vector at each treatment time point to obtain the multimodal joint representation vector at that treatment time point. Preferably, to achieve deep fusion of image vectors and text vectors, the multimodal feature fusion module uses a Transformer encoder to fuse the image vector and text vector at each treatment time point to obtain the multimodal joint representation vector at that treatment time point. This achieves fine-grained, global, and bidirectional interactive capabilities, enabling the establishment of extremely accurate and complex cross-modal associations.

[0035] When the multimodal feature fusion module uses a Transformer encoder for fusion, it combines the image vector, text vector, and a learnable initialization vector at each treatment time point. Each input is a transformer encoder, and the output vector is the multimodal joint representation vector at that treatment time point. For example, the image vector at a certain treatment time point... Text vector And introduce a learnable initialization vector As a dynamic prior for this treatment time point This indicates the number of text entries in the text data at that treatment time point. , , These represent the first, second, and third text entries in the text data at that treatment time point, respectively. The feature vector of the text. , , The three feature components are concatenated into the input sequence. After adding position encoding, input it into the Transformer encoder. Extract the final output from the Transformer encoder. The corresponding feature vector, as a single multimodal joint representation vector at that treatment time point, fully integrates visual details, clinical semantics, and temporal dynamic information.

[0036] In this embodiment, the time feature embedding module obtains a time context embedding vector based on the time covariance of each treatment time point. Preferably, the time feature embedding module performs: Step A1: Encode the time from the first treatment time for each treatment time point into a first time vector using the time feature embedding method, and encode the time from the last treatment time for each treatment time point into a second time vector. Step A2: Concatenate the first time vector and the second time vector of each treatment time point to obtain the time context embedding vector of that treatment time point.

[0037] In this embodiment, the temporal feature embedding method can employ existing absolute position encoding or relative position encoding. Preferably, the temporal feature embedding method employs existing sine-cosine positional encoding, which can effectively model the treatment time sequence information of liver cancer patients during dynamic follow-up, and is used to convert the time since the first treatment and the time since the last treatment into low-dimensional dense vectors.

[0038] For example, the temporal feature embedding method employs existing sine-cosine positional encoding, the first... The time elapsed between the first treatment time point and the treatment time is the scalar time. The numerical value of each treatment time point from the time of the first treatment is The first time vector is , include One dimension, , Let the first time vector be... The dimension index is , Then the first time vector The Middle Location code values ​​in each dimension for: ; Then the first time vector The Middle Location code values ​​in each dimension for: Represents the set of real numbers; Represents the sine function; Represents the cosine function; No. The time elapsed between the first treatment time point and the last treatment time is the scalar time. The numerical value of each treatment time point from the last treatment time is The second time vector is , include One dimension, , Let the second time vector be... The dimension index is , Then the second time vector The Middle Location code values ​​in each dimension for: ; Then the first time vector The Middle Location code values ​​in each dimension for: splicing the first The first time vector of each treatment time point Second time vector The second time vector Connect to the first time vector Then, obtain the temporal context embedding vector for that treatment time point. , This represents the concatenation operator.

[0039] In this embodiment, the multimodal joint representation vector and the temporal context embedding vector at each treatment time point are concatenated to obtain the multimodal temporal context joint representation vector at that treatment time point. Specifically, the concat function in deep learning is used to concatenate the multimodal joint representation vector and the temporal context embedding vector at each treatment time point along the channel dimension. For example, concatenating a 128-dimensional vector (equivalent to 128 channels) and a 256-dimensional vector (equivalent to 256 channels) yields a vector with 128 + 256 dimensions.

[0040] Preferably, the case regression module uses a long short-term memory network, which can learn long-distance dependencies. The multimodal temporal context joint representation vectors of all treatment time points in the longitudinal cohort data samples, the longitudinal cohort data to be predicted, or the simulated longitudinal cohort data are respectively input into the long short-term memory network. The long short-term memory network sequentially outputs the historical perception state vectors of all treatment time points in the longitudinal cohort data samples, the longitudinal cohort data to be predicted, or the simulated longitudinal cohort data.

[0041] The network structure of Long Short-Term Memory (LSTM) networks is as follows: Figure 3 As shown, the Long Short-Term Memory (LSTM) network includes a forgetting gate network, an input gate network 1, an input gate network 2, an output gate network, and a hyperbolic tangent activation function (tanh1). The forgetting gate network includes a cascaded third fully connected layer and a first sigmoid activation function. The input gate network 1 includes a cascaded fourth fully connected layer and a second sigmoid activation function (tanh2). The input gate network 2 includes a cascaded fifth fully connected layer and a hyperbolic tangent activation function (tanh2).

[0042] Long Short-Term Memory (LSTM) networks use gating mechanisms to... Multimodal temporal context joint representation vector at each treatment time point With the Historical perception state vector at each treatment time point Perform fusion processing to generate the first Historical perception state vector at each treatment time point The specific steps are as follows: First, the input vector of the Long Short-Term Memory (LSTM) network is formed by concatenation. Within a Long Short-Term Memory (LSTM) network, the forget gate network processes the input vector... The process is performed to obtain the output vector of the forget gate. : The first input gate network for the input vector The input gate's first output vector is obtained through processing. : The second input gate network for the input vector The input gate's second output vector is obtained through processing. : Output gate network for input vector The output vector of the output gate is obtained through processing. : No. Cellular state of long and short-term memory networks at each treatment time point for: Generating the number of units using the hyperbolic tangent activation function of a Long Short-Term Memory (LSTM) network Historical perception state vector at each treatment time point : in, , These represent the weight parameters and bias parameters of the third fully connected layer, respectively. , These represent the weight parameters and bias parameters of the fourth fully connected layer, respectively. , These represent the weight parameters and bias parameters of the fifth fully connected layer, respectively. , These represent the weight parameters and bias parameters of the sixth fully connected layer, respectively. , , Let represent the first sigmoid activation function, the second sigmoid activation function, and the third sigmoid activation function, respectively. Represents the hyperbolic tangent activation function of the hyperbolic tangent activation function unit; This represents the hyperbolic tangent activation function of the second input gate network. Indicates the first The cellular states of the long short-term memory network at each treatment time point. In the calculation of the historical perceptual state vector at the next treatment time point. Automatically used as the cellular state of the long short-term memory network at the previous treatment time point (as shown by the dotted line in the figure) ), Automatically used as the historical perception state vector of the previous treatment time point (as shown by the dashed line in the figure) ). , , , , , , , These are all network parameters that need to be learned during the training of recurrent networks.

[0043] In a Long Short-Term Memory (LSTM) network, the forget gate network determines which historical information is retained, while the input gate 1 and input gate 2 networks control the inflow of new information and simultaneously calculate the candidate cell state (i.e., the input gate 2 output vector). The system updates the cell state, selectively preserving long-term dependency information while updating new features; the output gate network controls the information output, generating a historical perception state vector for the current treatment time point. This mechanism enables Long Short-Term Memory (LSTM) networks to effectively capture long-term dynamic changes from the start of treatment to the current treatment time point, while fusing multimodal data (images, text, etc.) to provide high-quality temporal contextual representations for relapse prediction.

[0044] In this embodiment, the decision-making module includes a cascaded multilayer perceptron and a normalized exponential function. For details, please see... Figure 4 The decision module comprises a first fully connected layer, a first activation function layer, a second fully connected layer, and an output layer connected in sequence. The first fully connected layer, the first activation function layer, and the second fully connected layer form a multilayer perceptron, and the output layer uses a normalized exponential function. The first activation function layer is not limited to using the ReLU activation function, i.e., a linear rectified function.

[0045] For example, having After processing by the multimodal feature extraction module, the multimodal joint representation vector set of longitudinal cohort data samples, longitudinal cohort data to be predicted, or simulated longitudinal cohort data at each treatment time point is obtained as follows: in, , , Represent the multimodal joint representation vectors at the 0th treatment time point in the longitudinal cohort data sample, the longitudinal cohort data to be predicted, or the simulated longitudinal cohort data, respectively. The multimodal joint representation vector at each treatment time point, the first... Multimodal joint representation vector at each treatment time point.

[0046] After processing the temporal covariates in the aforementioned longitudinal queue data samples, or the longitudinal queue data to be predicted, or the simulated longitudinal queue data using the temporal feature embedding module, the temporal context embedding vector group is obtained as follows: in, , , Represent the temporal context embedding vectors at the 0th treatment time point in the longitudinal cohort data sample, the longitudinal cohort data to be predicted, or the simulated longitudinal cohort data, respectively. The temporal context embedding vector of the treatment time point, the first treatment time point Temporal context embedding vectors for each treatment time point.

[0047] The multimodal time context joint representation vector for each treatment time point is obtained by concatenating the temporal context embedding vector and the multimodal joint representation vector. Correspondingly, a multimodal time context joint representation vector group is obtained. in, , , The first and second treatment time points represent the multimodal temporal context joint representation vectors of the longitudinal cohort data sample, the longitudinal cohort data to be predicted, or the simulated longitudinal cohort data, respectively. The multimodal temporal context joint representation vector at each treatment time point, the first... Multimodal temporal context joint representation vector for each treatment time point.

[0048] Input the multimodal temporal context joint representation vector set into the case regression module to obtain the historical perceived state vector set: in, , , Represent the historical perceived state vector at the 0th treatment time point in the longitudinal cohort data sample, the longitudinal cohort data to be predicted, or the simulated longitudinal cohort data, respectively. The historical perception state vector of the treatment time point, the first The historical perceived state vector at each treatment time point. , Encoded from arrive The complete clinical evolution path. Represents Long Short-Term Memory Network For the Multimodal temporal context joint representation vector at each treatment time point and the Historical perception state vector at each treatment time point Process to obtain the first Historical perception state vector at each treatment time point .

[0049] Finally, the historical perceived state vector of each treatment time point is input into the decision module to obtain the prediction result for that treatment time point.

[0050] In this embodiment, the first The prediction results for each treatment time point are expressed as follows: in, , , They represent the first The probability of recurrence in the first future time period after treatment at each treatment time point, The treatment time point after the first treatment The probability of recurrence in the next future time period, the first The treatment time point after the first treatment The probability of recurrence in a future time period. , , The range of values ​​is .

[0051] In one example The value is 4, and the prediction results include 4 future time periods, i.e., 4 classification categories. Each classification category represents one future time period, specifically: Category I: ≤6 months Category II: >6 months and ≤1 year Category III: >1 year and ≤2 years Category IV: >2 years.

[0052] The higher the recurrence probability of a particular category (future time period), the higher the likelihood of recurrence under the current treatment method. The category (future time period) with the highest recurrence probability can be selected as the predicted recurrence time.

[0053] In this embodiment, the recurrence probability prediction for the i-th treatment time point is combined with... Figure 4 The decision module obtains the first result through the following process. The probability of recurrence in the j-th future time period after treatment at each treatment time point: The first Historical perception state vector at each treatment time point Input to Figure 4 The decision module is shown. The specific calculation process is as follows: First, the input historical perception state vector... The first fully connected layer linearly transforms it into the first representation. : The hidden representation is generated using the ReLU activation function through the first activation function layer. : ; Subsequently, hidden representation The second fully connected layer linearly transforms to obtain the first... linear score of each treatment time point : ; It can be represented as: .

[0054] Finally, the linear score is obtained by using the normalized exponential function softmax of the output layer. Each term in the equation is converted into a probability distribution, such as the first term. The treatment time point after the first treatment Recurrence probability in a future time period for: in, , These represent the weight parameters and bias parameters of the first fully connected layer, respectively. , These represent the weight parameters and bias parameters of the second fully connected layer, respectively. , , , These are all network parameters that the decision-making module needs to learn during recurrent network training; Indicates the first The category linear score for the first future time period after treatment at each treatment time point; Indicates the first The first treatment time point after treatment Linear scores for each future time period; Indicates the first The first treatment time point after treatment Linear scores for each future time period; Indicates the first The first treatment time point after treatment The linear score of the category for each future time period. , It is a positive integer.

[0055] Recurrence probability The model directly quantified the occurrence of the first [condition] after treatment. The confidence score for recurrence in a future time period is calculated in a process that ensures the sum of the probabilities of all categories is 1. This reflects the distribution characteristics of clinical recurrence risk. For example, when When =0.72, it means the first... The probability of relapse in the third future time period after treatment at the first time point is 72%.

[0056] The following describes the working principle of the training module. In a preferred embodiment, the training module trains the recurrence network using a longitudinal queue dataset to obtain a recurrence model, specifically including: Step S21, please see Figure 2 , and build a recurrence network.

[0057] Step S22: Divide the vertical queue dataset into a training set, a test set, and a validation set according to a preset partitioning ratio. The preset partitioning ratio is not limited to 8:1:1.

[0058] Step S23: Train the relapse network using the training set. During training, calculate the loss function value based on the cross-entropy loss function and the number of treatment time points in the longitudinal cohort data samples. Update the network parameters of the relapse network using stochastic gradient descent (SGD) based on the loss function value. When the training stopping condition is met, stop training and save a set of network parameter values ​​of the relapse network when the loss function value is minimized. Load the saved network parameter values ​​into the relapse network to obtain the trained relapse model. The training stopping condition is not limited to reaching the maximum preset number of training iterations or the loss function value being less than a preset loss threshold.

[0059] Step S24: Test and validate the trained recurrent model using the test set and validation set respectively. If the test and validation pass, the recurrent model that has passed the test and validation is the final recurrent model. If the test or validation fails, use deep learning model tuning methods such as changing the learning rate and changing the training optimizer, and then return to execute steps S23 and S24.

[0060] When the image encoder uses a pre-trained image encoder and the text encoder uses a pre-trained text encoder, the network parameters of the recurrence network include: the network parameters of the multimodal feature fusion module, the network parameters of the long short-term memory network used in the case regression module, and the network parameters of the decision module. When the image encoder uses an untrained image encoder and the text encoder uses an untrained text encoder, the network parameters of the recurrence network include: the network parameters of the image encoder, the network parameters of the text encoder, the network parameters of the multimodal feature fusion module, the network parameters of the long short-term memory network used in the case regression module, and the network parameters of the decision module.

[0061] In step S23, during the num-th training iteration, the network parameters of the recurrent network are updated using stochastic gradient descent (SGD) based on the loss function value, including: Calculate the gradient of the loss function value with respect to each network parameter in the recurrent network; Multiply the learning rate by the gradient of each network parameter to obtain the update increment of that network parameter; Subtract the update increment of each network parameter from its value in the (num-1)th training iteration to obtain the value of that network parameter in the numth training iteration; num represents the training iteration index, which is a positive integer.

[0062] In a preferred embodiment, the loss function of the training module during the training of the recurrent network using a longitudinal queue dataset. for: in, This indicates the number of treatment time points in the longitudinal cohort data sample. This represents the index of the treatment time point in the longitudinal queue data sample. Indicates the first data sample in the vertical queue Cross-entropy loss at each treatment time point, is a non-negative integer. Among them, the in the vertical queue data sample is the _th Cross-entropy loss at each treatment time point for: in, This indicates the number of future time periods set in the prediction results, i.e., the number of classification categories; Indicates the first data sample in the vertical queue The treatment time point is at the A relapse time tag for a future time period; This represents the first longitudinal cohort data sample output by the relapse model. The treatment time point after the first treatment The probability of recurrence in a future time period; It is a positive integer. (Using cross-entropy loss) It can accelerate the convergence of recurrent networks. Loss function This is the average of the cross-entropy loss across all treatment time points.

[0063] It should be noted that the recurrence prediction module and the efficacy evaluation module in this system do not necessarily work simultaneously. Preferably, when a patient has already undergone treatment, the recurrence prediction module can be used to predict the probability of recurrence; when a patient has relapsed but has not yet been treated and needs to choose a treatment method, the efficacy evaluation module can be used to assess the recurrence probability of each available treatment method, and the treatment method with the lower recurrence probability or the longest recurrence time can be selected to effectively assist doctors in choosing multiple treatment methods.

[0064] The following explains the working principle of the recurrence prediction module: Let the current time be Equivalent to the current treatment time The patient has already undergone a certain treatment. Collect the patient's current treatment time. and before Treatment time point ( The multimodal data and time covariates are integrated into a longitudinal cohort data to be predicted, which can also be represented as: .

[0065] in, , , These represent the 0th and 1st treatment time points in the longitudinal cohort data of the patients to be predicted, respectively. The treatment time point, the first Multimodal data at each treatment time point , , These represent the 0th and 1st treatment time points in the longitudinal cohort data of the patients to be predicted, respectively. The treatment time point, the first Treatment time point (i.e., current treatment time) (Time covariates)

[0066] Input the longitudinal cohort data to be predicted into the recurrence model to obtain the data from the longitudinal cohort data to be predicted. The prediction results for each treatment time point are used to extract the patient's current treatment time. The predicted results, and then at the current treatment time The future time period with the highest probability of recurrence is extracted from the prediction results as the predicted recurrence time.

[0067] The working principle of the efficacy assessment module is explained below: Suppose the patient has relapsed but is currently untreated, and the current time is... This also indicates the current relapse time. (Before collecting patient information) Treatment time point ( Using multimodal data and time covariates, calculate the current time. The time covariate is used to obtain the current time when the patient receives each treatment method. Given multimodal data, assuming there are 5 treatment methods, we obtain 5 simulated longitudinal cohort data sets. Any simulated longitudinal cohort data set can be represented as: .

[0068] in, Indicates the patient's choice of treatment method The current time obtained Multimodal data. Treatment methods. It refers to a specific treatment method, such as one of five treatment methods. This indicates that the patient will choose a treatment method. The current time obtained Multimodal data The first in the longitudinal cohort data of patients to be predicted Temporal covariates at each treatment time point Related.

[0069] Input each simulated longitudinal cohort data point into the recurrence model to obtain the data from that simulated longitudinal cohort. The prediction results for each treatment time point are then used to extract the patient's current treatment time. The prediction results. When the current treatment time is obtained from all 5 simulated longitudinal cohort data. After analyzing the prediction results, compare the current treatment times corresponding to the five treatment methods. Based on the predicted results, the system selects the treatment method with the longest recurrence time or the lowest recurrence probability, providing doctors with a reference. It can perform parallel prognostic simulations of multiple treatment methods for the same patient and generate treatment path diagrams, providing doctors with precise and efficient treatment decision support.

[0070] In the description of this specification, the references to terms such as "an embodiment," "some embodiments," "example," "specific example," "a implementation," "a preferred implementation," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0071] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multimodal fusion-based system for predicting longitudinal recurrence and evaluating treatment efficacy in liver cancer, characterized in that, The system includes: The dataset construction module uses longitudinal cohort data samples from multiple patients to form a longitudinal cohort dataset, and sets a relapse time label for each longitudinal cohort data sample. The longitudinal cohort data samples of patients are generated in chronological order based on multimodal data and time covariates of patients at multiple treatment time points. The time covariates of each treatment time point include the time since the first treatment and the time since the last treatment. The multimodal data of each treatment time point includes the treatment method and the preceding treatment method at that treatment time point. The training module uses a longitudinal queue dataset to train the relapse network and obtain a relapse model. The relapse prediction module inputs the patient's longitudinal cohort data to be predicted into the relapse model to obtain the prediction results for each treatment time point in the longitudinal cohort data to be predicted. The prediction results include the relapse probability for more than one future time period. The efficacy assessment module obtains simulated longitudinal cohort data corresponding to different treatment methods selected by the patient to be assessed at the current relapse time point; the simulated longitudinal cohort data corresponding to each treatment method is input into the relapse model to obtain the prediction results of each treatment time point in the simulated longitudinal cohort data corresponding to that treatment method; The recurrence network includes: The multimodal feature extraction module obtains the joint multimodal representation vector for each treatment time point based on the multimodal data at that time point. The temporal feature embedding module obtains a temporal context embedding vector based on the temporal covariates of each treatment time point; The case regression module obtains the historical perceived state vector for each treatment time point based on the multimodal temporal context joint representation vector. The multimodal temporal context joint representation vector for each treatment time point is obtained by concatenating the multimodal joint representation vector and the temporal context embedding vector for that treatment time point. The decision-making module obtains the prediction result for each treatment time point based on the historical perceived state vector of each treatment time point.

2. The system according to claim 1, characterized in that, The multimodal data at each treatment time point includes image information and text data; the text data includes the treatment method and preceding treatment method at each treatment time point, as well as the patient's clinical records and / or laboratory test data.

3. The system according to claim 1, characterized in that, The multimodal feature extraction module includes: An image encoder obtains the image vector for each treatment time point based on the image information at that time point. The text encoder obtains the text vector for each treatment time point based on the text data at that time point. The multimodal feature fusion module fuses the image vector and text vector at each treatment time point to obtain the multimodal joint representation vector at that treatment time point.

4. The system according to claim 1, characterized in that, The time feature embedding module performs the following: The time feature embedding method is used to encode the time from the first treatment point of each treatment time point into a first time vector, and the time from the last treatment point of each treatment time point into a second time vector; The first and second time vectors of each treatment time point are concatenated to obtain the temporal context embedding vector of that treatment time point.

5. The system according to claim 1, characterized in that, The case regression module uses a long short-term memory network.

6. The system according to claim 1, characterized in that, The decision-making module includes a cascaded multilayer perceptron and a normalized exponential function.

7. The system according to any one of claims 1-6, characterized in that, The loss function of the training module during the training of the recurrent network using the longitudinal queue dataset. for: in, This indicates the number of treatment time points in the longitudinal cohort data sample. This represents the index of the treatment time point in the longitudinal queue data sample. Indicates the first data sample in the vertical queue Cross-entropy loss at each treatment time point, It is a non-negative integer.

8. The system according to claim 7, characterized in that, In the vertical queue data sample, the first Cross-entropy loss at each treatment time point for: in, This indicates the number of future time periods set in the prediction results, i.e., the number of classification categories; Indicates the first data sample in the vertical queue The treatment time point is at the The recurrence label value for a future time period; This represents the first longitudinal cohort data sample output by the relapse model. The treatment time point after the first treatment The probability of recurrence in a future time period; It is a positive integer.

9. The system according to any one of claims 2-6 and 8, characterized in that, The system also includes: The missing data imputation module imputes missing data in patient clinical records and / or laboratory test data.

Images