Liver cancer longitudinal recurrence prediction and treatment efficacy evaluation system based on multi-modal fusion

Abstract

The present application belongs to the technical field of medical data processing, and provides a liver cancer longitudinal recurrence prediction and treatment efficacy evaluation system based on multi-modal fusion, comprising: a data set construction module, which uses multiple patient longitudinal cohort data samples to form a longitudinal cohort data set, and sets a recurrence time label for each longitudinal cohort data sample; a training module, which trains a recurrence network using the longitudinal cohort data set to obtain a recurrence model; a recurrence prediction module, which inputs the patient's to-be-predicted longitudinal cohort data into the recurrence model to obtain prediction results at each treatment time point, the prediction results including the recurrence probability of more than one future time period; and an efficacy evaluation module, which obtains simulated longitudinal cohort data corresponding to different treatment methods at the current recurrence time point of the patient to be evaluated; and inputs the simulated longitudinal cohort data into the recurrence model to obtain prediction results at each treatment time point. The present application improves the accuracy and generalizability of the recurrence model, and accurately and efficiently assists doctors in selecting treatment methods.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical data processing, and in particular to a liver cancer longitudinal recurrence prediction and treatment efficacy evaluation system based on multi-modal fusion. BACKGROUND

[0002] Liver cancer recurrence after treatment is a difficult problem faced by clinicians, and deep learning technology has certain breakthroughs in liver cancer recurrence prediction. Multi-modal data can provide more comprehensive information for liver cancer recurrence prediction. Related technologies for liver cancer recurrence prediction (such as Chinese patent CN117612711A) integrate clinical text, image, and pathology multi-modal data, and construct a multi-modal prediction model for analyzing liver cancer recurrence data based on multi-modal data and fusion strategies to improve the accuracy of liver cancer recurrence prediction. However, the related technology does not consider the relationship between the natural progression of the disease and the overall length of the disease course, as well as the current sensitivity of the patient to the treatment method, so there is room for improvement in the accuracy of liver cancer recurrence prediction. In addition, existing multi-modal data sets and multi-modal model construction have more stringent requirements for data inclusion and exclusion criteria, which leads to a small size of available data sets, which is contrary to the requirements of deep learning for large data. In addition, cases with missing modalities cannot be included in the data set, resulting in poor generalizability of the multi-modal model.

[0003] In related technologies for liver cancer treatment efficacy evaluation, in order to more accurately evaluate the efficacy of liver cancer treatment, the medical data of patients who have not undergone other treatments (i.e., patients who have only undergone one liver cancer treatment) are selected to be included in the data set for model training and evaluation. However, in reality, liver cancer patients will experience multiple recurrences and treatment options during their course of treatment, and their sensitivity to different treatment options varies. Existing liver cancer treatment efficacy evaluation models based on a single treatment method are difficult to meet the requirements of assisting doctors in evaluating treatment efficacy to select treatment options. SUMMARY

[0004] The present application aims to at least solve the technical problems existing in the prior art, and provides a liver cancer longitudinal recurrence prediction and treatment efficacy evaluation system based on multi-modal fusion.

[0005] The liver cancer longitudinal recurrence prediction and treatment efficacy evaluation system based on multi-modal fusion provided by the present application comprises:

[0006] a data set construction module, which uses multiple patient longitudinal cohort data samples to form a longitudinal cohort data set, and sets a recurrence time label for each longitudinal cohort data sample; wherein the patient longitudinal cohort data samples are generated in chronological order based on the patient's multi-modal data and time covariates at multiple treatment time points, the time covariates at each treatment time point include time from first treatment and time from last treatment, and the multi-modal data at each treatment time point includes the treatment mode at the treatment time point and the previous treatment mode;

[0007] a training module, which trains the recurrence network using the longitudinal cohort data set to obtain a recurrence model;

[0008] a recurrence prediction module, which inputs the patient's to-be-predicted longitudinal cohort data into the recurrence model to obtain the prediction results of each treatment time point in the to-be-predicted longitudinal cohort data, the prediction results including the recurrence probability of one or more future time periods;

[0009] an efficacy evaluation module, which obtains the simulated longitudinal cohort data corresponding to different treatment modes at the current recurrence time point for the patient to be evaluated; and inputs the simulated longitudinal cohort data corresponding to each treatment mode into the recurrence model to obtain the prediction results of each treatment time point in the simulated longitudinal cohort data corresponding to the treatment mode.

[0010] The beneficial technical effects of the present application are as follows:

[0011] 1. The present application introduces the time covariates at each treatment time point into the input of the recurrence model, i.e., the multi-modal model, wherein the time from first treatment can reflect the natural progression of the disease and the overall length of the disease course; the time from last treatment can reflect the "freshness" of the recent treatment intervention, which affects the sensitivity of the patient to the treatment mode; the time covariates at each treatment time point and the multi-modal data together as the input of the recurrence model can improve the accuracy of recurrence prediction; in addition, the multi-modal data at each treatment time point not only includes the treatment mode at the treatment time point, but also includes the previous treatment mode, fully considering the influence of the previous treatment mode on the recurrence prediction, so that the recurrence model can more comprehensively represent the actual situation of the patient, thereby further improving the accuracy of the prediction results output by the recurrence model;

[0012] 2. The recurrence model of the present application can perform prediction processing on multiple treatment modes, without the need to establish a recurrence model for each treatment mode, simplifying the model structure, enabling more efficient recurrence prediction, and being applicable to patients who have not been treated before and patients who have experienced more than one treatment;

[0013] 3. The application can simultaneously predict all treatment time points in the longitudinal queue data to be predicted and the simulated longitudinal queue data, obtain the recurrence probability of more than one future time period at each treatment time point, compared with the related art of predicting a single time point before treatment, the application considers the information of historical treatment time points, and can more comprehensively predict recurrence;

[0014] 4. Compared with the strict inclusion and exclusion requirements of the data set in the previous prediction model for a single treatment scheme, in the construction of the longitudinal queue data set, all historical recurrence and treatment data of the same patient are included in the data set, which can greatly expand the size of the data set, and is more in line with the actual treatment in the clinic, and improves the generalization of the recurrence model;

[0015] 5. The application uses the recurrence model to predict the recurrence probability of each treatment method when predicting the recurrence of liver cancer in patients, so as to accurately and efficiently assist doctors to realize the selection of multiple treatment methods. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a structure schematic diagram of a liver cancer longitudinal recurrence prediction and treatment efficacy evaluation system based on multi-modal fusion in a preferred embodiment of the application;

[0017] Figure 2 is a structure schematic diagram of a recurrence network in a preferred embodiment of the application;

[0018] Figure 3 is a structure schematic diagram of a long short-term memory network in a preferred embodiment of the application;

[0019] Figure 4 is a structure schematic diagram of a decision module in a preferred embodiment of the application. DETAILED DESCRIPTION

[0020] The embodiments of the application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary, only for explaining the application, and cannot be understood as limiting the application.

[0021] The application discloses a liver cancer longitudinal recurrence prediction and treatment efficacy evaluation system based on multi-modal fusion, and an execution carrier of the system includes but is not limited to a server, and terminal, computer and other electronic devices.

[0022] In a preferred embodiment, see Figure 1 , the liver cancer longitudinal recurrence prediction and treatment efficacy evaluation system based on multi-modal fusion comprises:

[0023] a data set construction module, which uses longitudinal cohort data samples of a plurality of patients to form a longitudinal cohort data set, and sets a recurrence time label for each longitudinal cohort data sample; wherein the longitudinal cohort data samples of the patients are generated in chronological order based on multi-modal data and time covariates of the patients at a plurality of treatment time points, the time covariates at each treatment time point include a time from first treatment and a time from last treatment, and the multi-modal data at each treatment time point includes a treatment modality at the treatment time point and a previous treatment modality;

[0024] a training module, which trains the recurrence network using the longitudinal cohort data set to obtain a recurrence model;

[0025] a recurrence prediction module, which inputs longitudinal cohort data to be predicted of a patient into the recurrence model to obtain prediction results of each treatment time point in the longitudinal cohort data to be predicted, the prediction results including recurrence probabilities of one or more future time periods;

[0026] a therapeutic effect evaluation module, which obtains simulated longitudinal cohort data corresponding to different treatment modalities at a current recurrence time point of a patient to be evaluated, and inputs the simulated longitudinal cohort data corresponding to each treatment modality into the recurrence model to obtain prediction results of each treatment time point in the simulated longitudinal cohort data corresponding to the treatment modality.

[0027] The working principle of the data set construction module will be introduced below, and the process of using longitudinal cohort data samples of a plurality of patients to form a longitudinal cohort data set includes:

[0028] Step S11, records multi-modal data and time covariates of a plurality of patients at different treatment time points, and a recurrence time. The above information is not limited to being recorded by regular follow-up methods. The multi-modal data at each treatment time point includes image information and text data; the text data includes a treatment modality and a previous treatment modality at each treatment time point, and further includes patient clinical records and / or laboratory test data.

[0029] For example, the image information is not limited to CT images (computed tomography images) or MRI images (magnetic resonance imaging images) of the upper abdomen, chest and abdomen, or other parts of the patient.

[0030] For example, the treatment modes are not limited to the following five: liver resection, ablation therapy, interventional therapy, radiotherapy and systemic anti-tumor therapy. For the convenience of text model processing, different treatment modes are represented as different numerical values, such as: liver resection: 1; ablation therapy: 2; interventional therapy: 3; radiotherapy: 4; systemic anti-tumor therapy: 5. Preferably, in order to more accurately predict recurrence and evaluate treatment efficacy, each of the above treatment modes can be further subdivided, such as radiofrequency ablation, microwave ablation, laser ablation, etc. for ablation therapy, and targeted therapy and immunotherapy, etc. for systemic anti-tumor therapy. The multi-modal data at each treatment time point includes the treatment mode at the treatment time point and the previous treatment mode. The previous treatment mode represents the treatment mode of the last treatment of the patient. When the patient has no previous treatment, the previous treatment mode can be represented as 0, and when there is a previous treatment at a certain treatment time point, the previous treatment mode is represented according to the above numerical value representation rule.

[0031] For example, the clinical records include the patient's age, gender, weight, body mass index (BMI), cirrhosis, and hepatitis B virus infection, etc.

[0032] For example, the laboratory test data are not limited to the patient's alpha-fetoprotein concentration, total bilirubin concentration, albumin concentration, alanine aminotransferase activity, aspartate aminotransferase activity, platelet count, gamma-glutamyl transferase, and thrombin time, etc.

[0033] For example, the time covariates at each treatment time point include the time from the first treatment and the time from the last treatment. The time from the first treatment at each treatment time point is the time difference between the treatment time point and the patient's first treatment time, which can be in months, reflecting the natural progression of the disease and the overall length of the disease course, and is an important covariate for recurrence risk. The time from the last treatment at each treatment time point is the time difference between the treatment time point and the patient's last treatment time, which can be in days, reflecting the "freshness" of recent treatment intervention and affecting the sensitivity to treatment mode. If a certain treatment time point is the first treatment, the time from the first treatment and the time from the last treatment at the treatment time point are both set to 0.

[0034] Step S12, for each patient, integrating the multi-modal data and time covariates of the patient at more than one treatment time point in chronological order of the treatment time points of the patient, to obtain a longitudinal cohort data sample of the patient. It should be noted that a patient can have more than one longitudinal cohort data sample.

[0035] For example, a longitudinal cohort data sample has treatment time points, is a non-negative integer, then a longitudinal cohort data sample can be represented as:

[0036] .

[0037] in, , , These represent the 0th and 1st treatment time points in the longitudinal cohort data sample, respectively. The treatment time point, the first Each treatment time point, ; , , These represent the 0th and 1st treatment time points in the longitudinal cohort data sample, respectively. The treatment time point, the first Multimodal data at each treatment time point , , These represent the 0th and 1st treatment time points in the longitudinal cohort data sample, respectively. The treatment time point, the first The time covariates at each treatment time point. Indicates the first Multimodal data at each treatment time point and the Temporal covariates at each treatment time point Related, .

[0038] In this embodiment, "vertical" refers to the direction of time development. A longitudinal cohort data sample refers to the entire longitudinal treatment process of a patient from the first diagnosis of liver cancer to the endpoint (cure / death / end of follow-up). Multimodal data and time covariates are collected at each treatment time point, and the data from multiple treatment time points constitute the longitudinal cohort dataset.

[0039] Step S13: Set recurrence time tags for the longitudinal cohort data samples obtained in step S12. In step S11, the patient's recurrence time was recorded synchronously. There can be more than one recurrence time, indicating that the patient has relapsed more than once.

[0040] In step S13, firstly, a relapse time needs to be assigned to each treatment time point in the longitudinal queue data sample. Specifically, following the chronological order, the relapse time that is closest to each treatment time point is taken as the relapse time corresponding to that treatment time point. Next, a relapse time sub-label is set for each treatment time point in the longitudinal queue data sample. Finally, the relapse time sub-labels of all treatment time points in the longitudinal queue data sample are combined according to the chronological order of the treatment time points to obtain the relapse time label of the longitudinal queue data sample.

[0041] For example, suppose the first data sample in the vertical queue is... a recurrence time sub-label of the i-th treatment time is , represents the number of future time periods set in the prediction result, i.e., the number of classification categories, is a positive integer; represents the recurrence label value of the i-th treatment time point in the j-th future time period in the longitudinal cohort data sample, when the recurrence time corresponding to the i-th treatment time point falls into the j-th future time period, ; represents the recurrence label value of the i-th treatment time point in the j-th future time period in the longitudinal cohort data sample, when the recurrence time corresponding to the i-th treatment time point falls into the j-th future time period, ; represents the recurrence label value of the i-th treatment time point in the j-th future time period in the longitudinal cohort data sample, when the recurrence time corresponding to the i-th treatment time point falls into the j-th future time period, .

[0042] ​​​​​​​​​​​​​​​​​​​​In a preferred embodiment, the system further comprises a missing data imputation module for imputing missing data in the patient clinical records and / or laboratory test data, so that the system can effectively make recurrence prediction even if there is missing text data. The missing data imputation module imputes missing data in the patient clinical records and / or laboratory test data collected by the data set construction module when performing step S11. The missing data imputation module is not limited to imputing missing data according to statistical quantities (such as mean, median, etc.) or regression imputation method. Preferably, the missing data imputation module uses a time series recursive imputation method (RITS) or a bidirectional time series recursive imputation method (BRITS) to fill in the missing clinical records and / or laboratory test data, and imputes missing data according to longitudinal historical data (data at different treatment time points) and transverse data (data at the same treatment time point).

[0043] The network structure of the recurrence model, i.e., the recurrence network, will be described below, please see Figure 2 The recurrence network comprises:

[0044] A multi-modal feature extraction module obtains a multi-modal joint representation vector for each treatment time point based on multi-modal data at the treatment time point; when processing longitudinal cohort data samples or to-be-predicted longitudinal cohort data or simulated longitudinal cohort data, the multi-modal feature extraction module obtains a multi-modal joint representation vector for each treatment time point based on multi-modal data at the treatment time point in the longitudinal cohort data samples or to-be-predicted longitudinal cohort data or simulated longitudinal cohort data;

[0045] A time feature embedding module obtains a time context embedding vector based on time covariates at each treatment time point; when processing longitudinal cohort data samples or to-be-predicted longitudinal cohort data or simulated longitudinal cohort data, the time feature embedding module obtains a time context embedding vector for each treatment time point based on time covariates at the treatment time point in the longitudinal cohort data samples or to-be-predicted longitudinal cohort data or simulated longitudinal cohort data;

[0046] a case regression module configured to obtain a historical perception state vector of each treatment time point based on a multi-modal temporal context joint representation vector of the treatment time point; wherein the multi-modal temporal context joint representation vector of each treatment time point is obtained by concatenating a multi-modal joint representation vector of the treatment time point and a temporal context embedding vector; when processing the longitudinal cohort data sample or the to-be-predicted longitudinal cohort data or the simulated longitudinal cohort data, the case regression module obtains the historical perception state vector of each treatment time point based on the multi-modal temporal context joint representation vector of each treatment time point in the longitudinal cohort data sample or the to-be-predicted longitudinal cohort data or the simulated longitudinal cohort data;

[0047] a decision module configured to obtain a prediction result of each treatment time point based on the historical perception state vector of the treatment time point; when processing the longitudinal cohort data sample or the to-be-predicted longitudinal cohort data or the simulated longitudinal cohort data, the decision module obtains the prediction result of each treatment time point based on the historical perception state vector of each treatment time point in the longitudinal cohort data sample or the to-be-predicted longitudinal cohort data or the simulated longitudinal cohort data.

[0048] In the embodiment, the plurality of treatment time points can share one multi-modal feature extraction module, one temporal feature embedding module, and one decision module. Alternatively, one multi-modal feature extraction module, one temporal feature embedding module, and one decision module can be used for each treatment time point, which requires setting multiple multi-modal feature extraction modules, multiple temporal feature embedding modules, and multiple decision modules. Preferably, the multiple multi-modal feature extraction modules share network parameters (i.e., share weights). The multiple decision modules share network parameters (i.e., share weights) to ensure consistent decision logic.

[0049] In the embodiment, please see Figure 2 , the multi-modal feature extraction module comprises:

[0050] an image encoder configured to obtain an image vector of each treatment time point based on image information of the treatment time point; specifically, the image information of each treatment time point is input into the image encoder, and the image encoder outputs the image vector of the treatment time point. Exemplarily, the image encoder is not limited to an existing visual transformer or an image encoder of a contrastive language-image pre-training model (CLIP). The visual transformer, also known as a visual self-attention model, is referred to as Vision Transformer (ViT) in English.

[0051] Exemplarily, when the image encoder adopts the visual transformer, the visual transformer processes the CT or MRI image of a certain treatment time point in accordance with a general process, which includes the following steps:

[0052] ① Preprocessing: Adjust each medical image (i.e., CT image or MRI image) to a uniform size (e.g., 224×224 pixels); perform normalization processing (e.g., scale pixel values ​​to the range of [0,1]).

[0053] ② Image segmentation: Each medical image is divided into image blocks of a fixed size. Each image block is convolved using a two-dimensional convolution with a kernel size that is the same as the image block size to obtain a one-dimensional vector. In addition, a learnable vector, denoted as cls, is added to each medical image.

[0054] ③Location coding: Encode the one-dimensional vector obtained from each image block of each medical image and the cls of each medical image using the existing sine-cosine location coding;

[0055] ④ Transformer encoder processing: Input all vectors obtained from step ③ into the Transformer encoder. From the Transformer encoder output, only the feature vector corresponding to cls is selected as the global feature vector for each medical image. (CT or MRI images contain...) Zhang's medical images will be obtained A global feature vector, The image vector at this treatment time point is composed of several global feature vectors. . It is a positive integer. , , These respectively represent the first medical image, the second medical image, and the third medical image in a CT or MRI image. Global feature vectors of medical images.

[0056] The text encoder obtains the text vector for each treatment time point based on the text data at that time point; specifically, the text data for each treatment time point is input into the text encoder, and the text encoder outputs the text vector for that treatment time point.

[0057] For example, the text encoder is not limited to text encoders using existing BERT models or contrastive language-image pre-trained models (CLIP). The BERT model is a bidirectional encoder representation model. The text data for each treatment time point includes the treatment method, patient clinical records, or laboratory test data, and there are multiple texts. With a text encoder using a BERT or CLIP model, each text can be encoded into an independent feature vector. For example, the treatment method at that treatment time point yields a feature vector after being encoded by the text encoder, and the blood pressure at that treatment time point also yields a feature vector. The feature vectors of all texts together constitute the text vector for that treatment time point.

[0058] The multimodal feature fusion module fuses the image vector and text vector at each treatment time point to obtain the multimodal joint representation vector at that treatment time point.

[0059] In this embodiment, the multimodal feature fusion module is not limited to using a concatenation function or cross-attention mechanism to fuse the image vector and text vector at each treatment time point to obtain the multimodal joint representation vector at that treatment time point. Preferably, to achieve deep fusion of image vectors and text vectors, the multimodal feature fusion module uses a Transformer encoder to fuse the image vector and text vector at each treatment time point to obtain the multimodal joint representation vector at that treatment time point. This achieves fine-grained, global, and bidirectional interactive capabilities, enabling the establishment of extremely accurate and complex cross-modal associations.

[0060] When the multimodal feature fusion module uses a Transformer encoder for fusion, it combines the image vector, text vector, and a learnable initialization vector at each treatment time point. Each input is a transformer encoder, and the output vector is the multimodal joint representation vector at that treatment time point. For example, the image vector at a certain treatment time point... Text vector And introduce a learnable initialization vector As a dynamic prior for this treatment time point This indicates the number of text entries in the text data at that treatment time point. , , These represent the first, second, and third text entries in the text data at that treatment time point, respectively. The feature vector of the text. , , The three features are concatenated into the input sequence. After adding position encoding, input it into the Transformer encoder. Extract the final output from the Transformer encoder. The corresponding feature vector, as a single multimodal joint representation vector at that treatment time point, fully integrates visual details, clinical semantics, and temporal dynamic information.

[0061] In this embodiment, the time feature embedding module obtains a time context embedding vector based on the time covariance of each treatment time point. Preferably, the time feature embedding module performs:

[0062] Step A1: Encode the time from the first treatment time for each treatment time point into a first time vector using the time feature embedding method, and encode the time from the last treatment time for each treatment time point into a second time vector.

[0063] Step A2, concatenate the first time vector and the second time vector of each treatment time point to obtain the temporal context embedding vector of the treatment time point.

[0064] In the present embodiment, the time feature embedding method can adopt existing Absolute Position Encoding, Relative Position Encoding. Preferably, the time feature embedding method adopts existing Sinusoidal Positional Encoding, which can effectively model the treatment time sequence information of liver cancer patients in the dynamic follow-up process, and is used to convert the time from the first treatment time and the time from the last treatment time into a low-dimensional dense vector.

[0065] For example, the time feature embedding method adopts existing Sinusoidal Positional Encoding, the time from the first treatment time of the first treatment time point is a scalar time, the numerical value of the time from the first treatment time of the first treatment time point is , the first time vector is , including dimensions, , , let the dimension index of the first time vector be , , then the position encoding value of the first time vector dimension is: ;

[0066] ;

[0067] , then the position encoding value of the first time vector dimension is:

[0068]

[0069] represents the real number set; represents the sine function; represents the cosine function;

[0070] the time from the last treatment time of the first treatment time point is a scalar time, the numerical value of the time from the last treatment time of the first treatment time point is , the second time vector is ​​​The numerical size of the treatment time point from the last treatment time is , the second time vector is , includes dimensions, , , let the dimension index of the second time vector be , , then the position encoding value of the first dimension in the second time vector is:

[0071] ;

[0072] then the position encoding value of the first dimension in the first time vector is:

[0073]

[0074] Splice the first time vector and the second time vector of the first treatment time point, that is, connect the second time vector after the first time vector , to obtain the time context embedding vector of the treatment time point, representing the connection operator.

[0075] In this embodiment, the multi-modal joint representation vector and the time context embedding vector of each treatment time point are spliced to obtain the multi-modal time context joint representation vector of the treatment time point. Specifically, the splicing function (concat function) in the deep learning technology is used to splice the multi-modal joint representation vector and the time context embedding vector of each treatment time point in the channel dimension. For example, a 128-dimensional vector (equivalent to 128 channels) and a 256-dimensional vector (equivalent to 256 channels) are spliced to obtain a 128+256-dimensional vector.

[0076] Preferably, the case regression module adopts a long short-term memory network that can learn long-distance dependencies. The multi-modal time context joint representation vectors of all treatment time points in the longitudinal cohort data sample or the to-be-predicted longitudinal cohort data or the simulated longitudinal cohort data are respectively input into the long short-term memory network, and the long short-term memory network sequentially outputs the historical perception state vectors of all treatment time points in the longitudinal cohort data sample or the to-be-predicted longitudinal cohort data or the simulated longitudinal cohort data.

[0077] The network structure of Long Short-Term Memory (LSTM) networks is as follows: Figure 3 As shown, the Long Short-Term Memory (LSTM) network includes a forgetting gate network, an input gate network 1, an input gate network 2, an output gate network, and a hyperbolic tangent activation function (tanh1). The forgetting gate network includes a cascaded third fully connected layer and a first sigmoid activation function. The input gate network 1 includes a cascaded fourth fully connected layer and a second sigmoid activation function (tanh2). The input gate network 2 includes a cascaded fifth fully connected layer and a hyperbolic tangent activation function (tanh2).

[0078] Long Short-Term Memory (LSTM) networks use gating mechanisms to... Multimodal temporal context joint representation vector at each treatment time point With the Historical perception state vector at each treatment time point Perform fusion processing to generate the first Historical perception state vector at each treatment time point The specific steps are as follows:

[0079] First, the input vector of the Long Short-Term Memory (LSTM) network is formed by concatenation. Within a Long Short-Term Memory (LSTM) network, the forget gate network processes the input vector... The process is performed to obtain the output vector of the forget gate. :

[0080]

[0081] The first input gate network for the input vector The input gate's first output vector is obtained through processing. :

[0082]

[0083] The second input gate network for the input vector The input gate's second output vector is obtained through processing. :

[0084]

[0085] Output gate network for input vector The output vector of the output gate is obtained through processing. :

[0086]

[0087] No. Cellular state of long and short-term memory networks at each treatment time point for:

[0088]

[0089] The hyperbolic tangent activation function unit number of the long short-term memory network is used to generate the historical perception state vector of the :

[0090]

[0091] The weight parameters and bias parameters of the third full connection layer are represented by , , The weight parameters and bias parameters of the fourth full connection layer are represented by , The weight parameters and bias parameters of the fifth full connection layer are represented by , The weight parameters and bias parameters of the sixth full connection layer are represented by , , The first S-shaped activation function, the second S-shaped activation function, and the third S-shaped activation function are represented by The hyperbolic tangent activation function of the hyperbolic tangent activation function unit is represented by The hyperbolic tangent activation function of the input gate second network is represented by The cell state of the long short-term memory network at the The cell state of the long short-term memory network at the previous treatment time point is automatically converted to The historical perception state vector at the previous treatment time point is automatically converted to . , , , , , , , All are network parameters that need to be learned in the recurrent network training.

[0092] The forget gate network in the long short-term memory network decides which historical information to keep, and the input gate first network and the input gate second network control the inflow of new information while calculating the candidate cell state (i.e., the input gate second output vector ​​​​), update the cell state, so as to selectively retain long-term dependency information and update new features; and the output gate network controls information output to generate a historical perception state vector at the current treatment time point This mechanism enables the long short-term memory network to effectively capture long-term dynamic changes from the start of treatment to the current treatment time point, while fusing multi-modal data (images, texts, etc.), thereby providing high-quality temporal context representation for recurrence prediction.

[0093] In the present embodiment, the decision module includes a cascaded multi-layer perceptron and a normalized exponential function. Specifically, please refer to Figure 4 The decision module includes a first fully connected layer, a first activation function layer, a second fully connected layer and an output layer connected in sequence, wherein the first fully connected layer, the first activation function layer and the second fully connected layer connected in sequence form a multi-layer perceptron, and the output layer adopts a normalized exponential function. The first activation function layer is not limited to using a ReLU activation function, i.e., a linear rectifier function.

[0094] For example, the multi-modal data of the longitudinal cohort data sample or the longitudinal cohort data to be predicted or the simulated longitudinal cohort data with treatment time points is processed by the multi-modal feature extraction module to obtain a multi-modal joint representation vector group as follows:

[0095]

[0096] wherein, , , represent the multi-modal joint representation vector at the 0th treatment time point, the multi-modal joint representation vector at the th treatment time point, and the multi-modal joint representation vector at the th treatment time point in the longitudinal cohort data sample or the longitudinal cohort data to be predicted or the simulated longitudinal cohort data.

[0097] The time feature embedding module is used to process the time covariates in the above longitudinal cohort data sample or the longitudinal cohort data to be predicted or the simulated longitudinal cohort data to obtain a time context embedding vector group as follows:

[0098]

[0099] wherein, , , represent the time context embedding vector at the 0th treatment time point, the time context embedding vector at the th treatment time point, and the time context embedding vector at the th treatment time point in the longitudinal cohort data sample or the longitudinal cohort data to be predicted or the simulated longitudinal cohort data.

[0100] The time context embedding vector of each treatment time point and the multi-modal joint representation vector are spliced to obtain a multi-modal time context joint representation vector of the treatment time point. Correspondingly, a multi-modal time context joint representation vector group is obtained:

[0101]

[0102] wherein, , , respectively represent the multi-modal time context joint representation vector of the 0th treatment time point, the multi-modal time context joint representation vector of the th treatment time point, and the multi-modal time context joint representation vector of the th treatment time point in the longitudinal cohort data sample or the longitudinal cohort data to be predicted or the simulated longitudinal cohort data.

[0103] The multi-modal time context joint representation vector group is input into a case regression module to obtain a historical perception state vector group:

[0104]

[0105] wherein, , , respectively represent the historical perception state vector of the 0th treatment time point, the historical perception state vector of the th treatment time point, and the historical perception state vector of the th treatment time point in the longitudinal cohort data sample or the longitudinal cohort data to be predicted or the simulated longitudinal cohort data. Wherein, , encodes the complete clinical evolution path from to . represents a long short-term memory network processing the multi-modal time context joint representation vector of the th treatment time point and the historical perception state vector of the th treatment time point to obtain the historical perception state vector of the th treatment time point .

[0106] Finally, the historical perception state vector of each treatment time point is input into a decision module to obtain the prediction result of the treatment time point.

[0107] In the embodiment, the prediction result of the th treatment time point is represented as:

[0108]

[0109] wherein, , , respectively represent the recurrence probability after treatment at the i-th treatment time point in the 1st future time period, the recurrence probability after treatment at the i-th treatment time point in the j-th future time period, the recurrence probability after treatment at the i-th treatment time point in the k-th future time period, and the recurrence probability after treatment at the i-th treatment time point in the l-th future time period, , , The value interval is .

[0110] In one example, is 4, the prediction result includes 4 future time periods, i.e. four classification categories, each classification category represents a future time period, specifically:

[0111] Category I: ≤6 months

[0112] Category II: >6 months and ≤1 year

[0113] Category III: >1 year and ≤2 years

[0114] Category IV: >2 years.

[0115] The greater the recurrence probability of which classification category (future time period) is, the higher the recurrence possibility of the classification category (future time period) under the treatment mode of the current treatment time point, and the classification category (future time period) with the maximum recurrence probability can be selected as the predicted recurrence time.

[0116] In the present embodiment, for the recurrence probability prediction of the i-th treatment time point, in combination with Figure 4 , the decision module obtains the recurrence probability value of the j-th future time period after treatment at the i-th treatment time point through the following process: The historical perception state vector of the i-th treatment time point

[0117] is input into the decision module shown in FIG. 8. The specific calculation process is as follows: first, the input historical perception state vector is linearly transformed into the first representation through the first fully connected layer: Figure 4

[0118]

[0119] ​​​​​​​​The hidden representation is generated by the first activation function layer using a ReLU activation function :

[0120] ;

[0121] The hidden representation is then linearly transformed by the second fully connected layer to get the class linear scores for the th treatment time point :

[0122] ;

[0123] which can be represented as .

[0124] Finally, the linear scores are converted to probability distributions by the output layer's normalized exponential function softmax, such as the probability of recurrence in the th future time period after treatment at the th treatment time point :

[0125]

[0126] where , are the weight and bias parameters of the first fully connected layer, respectively; , are the weight and bias parameters of the second fully connected layer, respectively; , , , are the network parameters that need to be learned by the decision module in the recurrence network training; is the class linear score for the 1st future time period after treatment at the th treatment time point; is the class linear score for the th future time period after treatment at the th treatment time point; is the class linear score for the th future time period after treatment at the th treatment time point; is the class linear score for the th future time period after treatment at the th treatment time point, , are positive integers.

[0127] The probability of recurrence The model directly quantified the occurrence of the first wave after treatment. The confidence score for recurrence in a future time period is calculated in a process that ensures the sum of the probabilities of all categories is 1. This reflects the distribution characteristics of clinical recurrence risk. For example, when When =0.72, it means the first... The probability of relapse in the third future time period after treatment at the first time point is 72%.

[0128] The following describes the working principle of the training module. In a preferred embodiment, the training module trains the recurrence network using a longitudinal queue dataset to obtain a recurrence model, specifically including:

[0129] Step S21, please see Figure 2 , and build a recurrence network.

[0130] Step S22: Divide the vertical queue dataset into a training set, a test set, and a validation set according to a preset partitioning ratio. The preset partitioning ratio is not limited to 8:1:1.

[0131] Step S23: Train the relapse network using the training set. During training, calculate the loss function value based on the cross-entropy loss function and the number of treatment time points in the longitudinal cohort data samples. Update the network parameters of the relapse network using stochastic gradient descent (SGD) based on the loss function value. When the training stopping condition is met, stop training and save a set of network parameter values ​​of the relapse network when the loss function value is minimized. Load the saved network parameter values ​​into the relapse network to obtain the trained relapse model. The training stopping condition is not limited to reaching the maximum preset number of training iterations or the loss function value being less than a preset loss threshold.

[0132] Step S24: Test and validate the trained recurrent model using the test set and validation set respectively. If the test and validation pass, the recurrent model that has passed the test and validation is the final recurrent model. If the test or validation fails, use deep learning model tuning methods such as changing the learning rate and changing the training optimizer, and then return to execute steps S23 and S24.

[0133] When the image encoder uses a pre-trained image encoder and the text encoder uses a pre-trained text encoder, the network parameters of the recurrence network include: the network parameters of the multimodal feature fusion module, the network parameters of the long short-term memory network used in the case regression module, and the network parameters of the decision module. When the image encoder uses an untrained image encoder and the text encoder uses an untrained text encoder, the network parameters of the recurrence network include: the network parameters of the image encoder, the network parameters of the text encoder, the network parameters of the multimodal feature fusion module, the network parameters of the long short-term memory network used in the case regression module, and the network parameters of the decision module.

[0134] In step S23, during the num-th training iteration, the network parameters of the recurrent network are updated using stochastic gradient descent (SGD) based on the loss function value, including:

[0135] Calculate the gradient of the loss function value with respect to each network parameter in the recurrent network;

[0136] Multiply the learning rate by the gradient of each network parameter to obtain the update increment of that network parameter;

[0137] Subtract the update increment of each network parameter from its value in the (num-1)th training iteration to obtain the value of that network parameter in the numth training iteration; num represents the training iteration index, which is a positive integer.

[0138] In a preferred embodiment, the loss function of the training module during the training of the recurrent network using a longitudinal queue dataset. for:

[0139]

[0140] in, This indicates the number of treatment time points in the longitudinal cohort data sample. This represents the index of the treatment time point in the longitudinal queue data sample. Indicates the first data sample in the vertical queue Cross-entropy loss at each treatment time point, is a non-negative integer. Among them, the in the vertical queue data sample is the _th Cross-entropy loss at each treatment time point for:

[0141]

[0142] in, This indicates the number of future time periods set in the prediction results, i.e., the number of classification categories; Indicates the first data sample in the vertical queue The treatment time point is at the A relapse time tag for a future time period; This represents the first longitudinal cohort data sample output by the relapse model. The treatment time point after the first treatment The probability of recurrence in a future time period; It is a positive integer. (Using cross-entropy loss) It can accelerate the convergence of recurrent networks. Loss function This is the average of the cross-entropy loss across all treatment time points.

[0143] It should be noted that the recurrence prediction module and the efficacy evaluation module in this system do not necessarily work simultaneously. Preferably, when a patient has already undergone treatment, the recurrence prediction module can be used to predict the probability of recurrence; when a patient has relapsed but has not yet been treated and needs to choose a treatment method, the efficacy evaluation module can be used to assess the recurrence probability of each available treatment method, and the treatment method with the lower recurrence probability or the longest recurrence time can be selected to effectively assist doctors in choosing multiple treatment methods.

[0144] The following explains the working principle of the recurrence prediction module:

[0145] Let the current time be Equivalent to the current treatment time The patient has already undergone a certain treatment. Collect the patient's current treatment time. and before Treatment time point ( The multimodal data and time covariates are integrated into a longitudinal cohort data to be predicted, which can also be represented as:

[0146] .

[0147] in, , , These represent the 0th and 1st treatment time points in the longitudinal cohort data of the patients to be predicted, respectively. The treatment time point, the first Multimodal data at each treatment time point, , , These represent the 0th and 1st treatment time points in the longitudinal cohort data of the patients to be predicted, respectively. The treatment time point, the first Treatment time point (i.e., current treatment time) (Time covariates)

[0148] Input the longitudinal cohort data to be predicted into the recurrence model to obtain the data from the longitudinal cohort data to be predicted. The prediction results for each treatment time point are used to extract the patient's current treatment time. The predicted results, and then at the current treatment time The future time period with the highest probability of recurrence is extracted from the prediction results as the predicted recurrence time.

[0149] The working principle of the efficacy assessment module is explained below:

[0150] Suppose the patient has relapsed but is currently untreated, and the current time is... This also indicates the current relapse time. (Before collecting patient information) Treatment time point ( Using multimodal data and time covariates, calculate the current time. The time covariate is used to obtain the current time when the patient receives each treatment method. Given multimodal data, assuming there are 5 treatment methods, we obtain 5 simulated longitudinal cohort data sets. Any simulated longitudinal cohort data set can be represented as:

[0151] .

[0152] in, Indicates the patient's choice of treatment method The current time obtained Multimodal data. Treatment methods. It refers to a specific treatment method, such as one of five treatment methods. This indicates that the patient will choose a treatment method. The current time obtained Multimodal data The first in the longitudinal cohort data of patients to be predicted Temporal covariates at each treatment time point Related.

[0153] Input each simulated longitudinal cohort data point into the recurrence model to obtain the data from that simulated longitudinal cohort. The prediction results for each treatment time point are then used to extract the patient's current treatment time. The prediction results. When the current treatment time is obtained from all 5 simulated longitudinal cohort data. After analyzing the prediction results, compare the current treatment times corresponding to the five treatment methods. Based on the predicted results, the system selects the treatment method with the longest recurrence time or the lowest recurrence probability, providing doctors with a reference. It can perform parallel prognostic simulations of multiple treatment methods for the same patient and generate treatment path diagrams, providing doctors with precise and efficient treatment decision support.

[0154] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", "one implementation", "a preferred implementation" or "some examples" etc. means that the particular feature, structure, material or characteristic being described in connection with the embodiment or example is included in at least one embodiment or example of the application. The illustrative appearances of the above-mentioned terms in various places in the specification are not necessarily intended to refer to the same embodiment or example. Moreover, the particular features, structures, materials or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0155] Although embodiments of the present application have been shown and described, it would be appreciated by those skilled in the art that changes, modifications, alternatives and variations to these embodiments could be made without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.

Claims

1. A liver cancer longitudinal recurrence prediction and treatment efficacy evaluation system based on multi-modal fusion, characterized in that, The system comprises: a dataset construction module, which uses longitudinal cohort data samples of a plurality of patients to form a longitudinal cohort dataset, and sets a recurrence time label for each longitudinal cohort data sample; wherein the longitudinal cohort data samples of the patients are generated in chronological order based on multi-modal data and time covariates of the patients at a plurality of treatment time points, the time covariates at each treatment time point include a time to first treatment and a time to last treatment, and the multi-modal data at each treatment time point includes a treatment modality at the treatment time point and a previous treatment modality; a training module, which trains a recurrence network using the longitudinal cohort dataset to obtain a recurrence model; a recurrence prediction module, which inputs a to-be-predicted longitudinal cohort data of a patient into the recurrence model to obtain a prediction result of each treatment time point in the to-be-predicted longitudinal cohort data, the prediction result including recurrence probabilities of one or more future time periods; an efficacy evaluation module, which obtains simulated longitudinal cohort data corresponding to different treatment modalities at a current recurrence time point of a to-be-evaluated patient, and inputs the simulated longitudinal cohort data corresponding to each treatment modality into the recurrence model to obtain a prediction result of each treatment time point in the simulated longitudinal cohort data corresponding to the treatment modality; wherein the recurrence network comprises: a multi-modal feature extraction module, which obtains a multi-modal joint representation vector at each treatment time point based on multi-modal data at the treatment time point; a time feature embedding module, which obtains a time context embedding vector based on time covariates at each treatment time point; a case regression module, which obtains a historical awareness state vector at each treatment time point based on a multi-modal time context joint representation vector at the treatment time point; wherein the multi-modal time context joint representation vector at the treatment time point is obtained by concatenating a multi-modal joint representation vector at the treatment time point and a time context embedding vector; a decision module, which obtains a prediction result at each treatment time point based on a historical awareness state vector at the treatment time point; The case regression module employs a Long Short-Term Memory (LSTM) network, which uses a gating mechanism to... Multimodal temporal context joint representation vector at each treatment time point With the Historical perception state vector at each treatment time point Perform fusion processing to generate the first Historical perception state vector at each treatment time point , Encoded from arrive The complete clinical evolution path, , Indicates the first Each treatment time point, It is a non-negative integer.

2. The system of claim 1, wherein, the multi-modal data at each treatment time point includes image information and text data; and the text data includes a treatment modality at each treatment time point and a previous treatment modality, and patient clinical records and / or laboratory test data.

3. The system of claim 1, wherein, The multi-modal feature extraction module comprises: an image encoder, which obtains an image vector at each treatment time point based on image information at the treatment time point; a text encoder, which obtains a text vector at each treatment time point based on text data at the treatment time point; a multi-modal feature fusion module, which fuses the image vector and the text vector at each treatment time point to obtain a multi-modal joint representation vector at the treatment time point.

4. The system of claim 1, wherein, The time feature embedding module performs: encoding the time to first treatment at each treatment time point into a first time vector and encoding the time to last treatment at each treatment time point into a second time vector using a time feature embedding method; concatenating the first time vector and the second time vector at each treatment time point to obtain a time context embedding vector at the treatment time point.

5. The system of claim 1, wherein, The decision module comprises a cascaded multi-layer perceptron and a normalized exponential function.

6. The system of any one of claims 1-5, wherein, The loss function in the process of training the recurrence network by the training module using the longitudinal cohort dataset Is: wherein, represents the number of treatment time points in the longitudinal cohort data sample, represents an index of a treatment time point in the longitudinal cohort data sample, represents the cross-entropy loss for the th treatment time point in the longitudinal cohort data sample, is a non-negative integer.

7. The system of claim 6, wherein, The cross-entropy loss for the longitudinal cohort data sample at the treatment time point is: H(θ) = -∑i=1n y i log(p i) + (1 - y i) log(1 - p i) in, This indicates the number of future time periods set in the prediction results, i.e., the number of classification categories; Indicates the first data sample in the vertical queue The treatment time point is at the The recurrence label value for a future time period; This represents the first longitudinal cohort data sample output by the relapse model. The treatment time point after the first treatment The probability of recurrence in a future time period; It is a positive integer.

8. The system of any of claims 2-5, 7, wherein, The system further comprises: a missing data imputation module, which imputes missing data in the patient clinical records and / or laboratory test data.

Images