A multi-modal based drug dosage prediction method for liver transplantation

Abstract

The application belongs to the technical field of biological medicine and data processing, and particularly relates to a medicine dosage prediction method for liver transplantation based on multi-modalities, which comprises collecting clinical data of liver transplantation recipients and processing the same to obtain a sample set, which is divided into a training set and a test set; a multi-modal linear regression model is constructed; the samples in the training set are subjected to a feature extraction module to obtain an original feature sequence; the original feature sequence is input into a feature dimension reduction module to obtain a dimension-reduced feature sequence; the dimension-reduced feature sequence is input into an attention mechanism module to obtain an enhanced feature sequence; the enhanced feature sequence is subjected to a multi-modality fusion module to obtain a fusion feature sequence; the fusion feature sequence is input into a prediction module to obtain a prediction result, and the model parameters are trained and optimized; the clinical data to be processed is input into the trained multi-modality linear regression model to obtain a medicine dosage prediction result; and the application can accurately predict individualized dosage of tacrolimus.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of biological medicine and data processing, and particularly relates to a liver transplantation drug dosage prediction method based on multi-modal. BACKGROUND

[0002] Liver transplantation has made great progress worldwide, but the occurrence of postoperative rejection is still one of the most intractable problems in liver transplantation. In the development of rejection, due to the recognition of liver transplantation donor T cells to allogeneic antigens, the recipient immune response is initiated, causing portal vein inflammation and bile duct injury, and damaging liver function. The incidence of acute rejection is as high as 10%-40%, which seriously affects the prognosis of liver transplant recipients. Due to the participation of both innate immunity and adaptive immunity in liver transplantation rejection, involving multiple complex molecular events, the internal pathogenesis cannot be fully explained. In clinical practice, the main response to prevent and treat rejection is the rational use of immunosuppressive drugs.

[0003] The liver is an "immune exempt" organ, and the rational use of immunosuppressive drugs can effectively prevent and treat a part of liver transplantation rejection. In fact, liver transplant recipients currently need to routinely take immunosuppressive drugs to prevent and treat rejection. Among them, tacrolimus has stronger immunosuppressive effect and smaller side effects than other immunosuppressive agents, and has become the first choice of immunosuppressive drugs for liver transplant recipients. The advent of immunosuppressive drugs has improved the prognosis of liver transplant recipients to some extent. However, due to the lack of individualized medication concept and standardized medication process, the precise medication of liver transplant recipients is a global problem, one of which is the lack of precise medication dosage.

[0004] Currently, various liver transplantation centers widely use blood drug concentration detection as the main basis for adjusting the dosage of medication to improve the medication regimen of recipients. Blood drug concentration refers to the concentration of a drug in the blood, which reflects the distribution and metabolism of the drug in the body. For immunosuppressive drugs that need to maintain a certain concentration range to achieve the best therapeutic effect, the level of blood drug concentration directly affects its therapeutic effect. Through blood drug concentration detection, the concentration level of the drug in the body can be accurately understood, so as to judge whether the dosage of medication needs to be adjusted, ensure that the concentration of the drug in the patient's body is maintained within the optimal therapeutic range, so as to achieve the best therapeutic effect and reduce the occurrence of adverse reactions. Blood drug concentration detection has been widely recognized and supported in clinical application. Many studies have shown that the dosage adjustment of immunosuppressive drugs guided by blood drug concentration detection can significantly improve the therapeutic effect and survival rate of liver transplant recipients, and reduce the occurrence of rejection and adverse reactions. However, this scheme still has many problems:

[0005] (1)Blood concentration detection is a post-dose test, which is lagging and requires a certain time for the recipient to reach a stable blood concentration. The high incidence of acute rejection after liver transplantation is exactly one week to one month after surgery. The lagging adjustment of liver transplantation drugs will miss the golden time of drug use, that is, from the first dose of immunosuppressive drugs before surgery to the use of combined immunosuppressive drugs one week after surgery. At this time, a forward-looking drug guide is particularly important.

[0006] (2)Blood concentration results can be used as a basis for improving subsequent drug dosage and method, but before that, drug use is often mechanical and uniform, or similar drug dosage is used according to literature reports and past experience, or the initial dose is adjusted according to individual factors of the recipient. Due to insufficient data integration, preoperative and early postoperative drug use lacks systematicness and standardization, but it is crucial to reasonably select the type and initial dose of immunosuppressive agents and timely reach the desired concentration.

[0007] At present, many gene mutations highly related to liver transplantation immunosuppressive blood concentration have been reported, but there is a lack of systematic integration of exploration. On the other hand, there are great differences in the basic conditions of recipients, including age, weight, underlying diseases, etc. The existing liver transplantation drug dosage prediction method usually only considers single-mode data of blood concentration test results, without considering the relevance between different modal data; at the same time, there are many factors that may affect liver transplantation drugs, and there are individual differences between different individuals, and the existing method does not well distinguish the importance of different influencing factors. Therefore, how to effectively integrate different modal data and clarify the proportion of each influencing factor is crucial to predict the individualized dosage of each liver transplant recipient. SUMMARY

[0008] To solve the above problems, the present application provides a liver transplantation drug dosage prediction method based on multi-modal, which learns the nonlinear, high-dimensional and complex correlation between the physiological data of patients and blood concentration and drug dosage by using artificial intelligence method, predicts the dosage of tacrolimus that patients need to take after liver transplantation, and improves the accuracy of the prediction result.

[0009] The specific scheme includes:

[0010] S1. Collecting clinical data of liver transplant recipients, the clinical data including table type data and image type data; processing all the clinical data to obtain a sample set, dividing the sample set into a training set and a test set; at the same time, collecting the initial dosage of tacrolimus used by the liver transplant recipients after surgery; each sample in the sample set includes a table sample and an image sample;

[0011] S2. Constructing a multi-modal information linear regression model, which includes a feature extraction module, a feature dimension reduction module, an attention mechanism module, a multi-modal fusion module, and a prediction module;

[0012] S3. The samples in the training set are passed through a feature extraction module to obtain an original feature sequence, the original feature sequence including an original image feature sequence and an original table feature sequence;

[0013] S4. The original feature sequence is input into a feature dimension reduction module to obtain a dimension-reduced feature sequence, the dimension-reduced feature sequence including an image dimension-reduced feature sequence and a table dimension-reduced feature sequence;

[0014] S5. The dimension-reduced feature sequence is input into an attention mechanism module to obtain an enhanced feature sequence, the enhanced feature sequence including an image enhanced feature representation and a table enhanced feature representation;

[0015] S6. The enhanced feature sequence is input into a multi-modal fusion module, and a fusion feature sequence is output;

[0016] S7. The fusion feature sequence is passed through a prediction module to obtain a prediction result, a loss is calculated according to the prediction result, and model parameter optimization is performed through back propagation until the model parameters converge;

[0017] S8. The obtained to-be-processed clinical data is input into the trained multi-modal information linear regression model to obtain a dosage prediction result.

[0018] The dosage prediction method for liver transplantation provided by the application is based on real liver transplant recipient clinical data, and a statistical method is used to screen significant variables affecting individualized drug dosage. A multi-modal information linear regression model is further optimized based on this to construct a data set. ResNet50 is used to extract image features, a fully connected network layer containing a normalization layer and a ReLU activation function is used to extract table data features, a VAE module is used to realize feature dimension reduction, PLGA and PLGCroA are used to capture global features and local features within and between modalities, respectively, a gated attention mechanism with a bilinear product is used to fuse multi-modal data, and an MLP is used for final prediction. The prediction accuracy of the model is evaluated based on MAE, R2, MSE, RMSE and Real Rate, and the model parameters are optimized using a mean square error loss function and an Adam optimizer to improve the performance of the model. It is difficult to integrate different types of data using traditional machine learning methods to predict individualized doses of tacrolimus, and it is difficult to ensure accuracy and interpretability when processing complex data. The application effectively solves this problem. The attention module of the prediction model can capture complex inter-modal and intra-modal dependencies from the global and local perspectives, respectively. Visualization of attention weights provides a visual understanding of the model decision-making process. In summary, the multi-modal prediction model can accurately predict individualized doses of tacrolimus while improving its transparency and interpretability compared to traditional models. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flow chart of the method of the present application;

[0020] Figure 2 A schematic diagram of the model of the present application;

[0021] Figure 3 A structure diagram of the feature dimension reduction module VAE of the present application;

[0022] Figure 4 A structure diagram of the attention mechanism module PLGA of the present application;

[0023] Figure 5 A structure diagram of the attention mechanism module PLGCroA of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0025] The existing liver transplantation drug dosage prediction method mainly focuses on only considering the interaction between genes or the influence factor between drugs, ignoring the importance of clinical detection data and related medical image data for drug dosage prediction. For example, clinical detection data contains rich individual health information and can provide direct indicators of patient physiological and biochemical status. CT image data can directly reflect the patient's tissue anatomy and pathological changes. Moreover, single modal data often cannot fully understand complex problems, and without considering the complementarity and correlation between multi-modal data, the optimal prediction accuracy may not be achieved, or even the prediction result may be biased. In addition, the existing prediction method does not consider the influence factors of liver transplantation comprehensively, and does not distinguish the importance of different influence factors well. The constructed model lacks interpretability and is difficult to provide clear treatment guidance for doctors in clinical practice.

[0026] Therefore, the present application provides a multi-modal based liver transplantation drug dosage prediction method, as shown in Figure 1 , Figure 2 , which comprises the following steps:

[0027] S1. Collecting the clinical data of liver transplant recipients, the clinical data including table type data and image type data; processing all the clinical data to obtain a sample set, dividing the sample set into a training set and a test set; at the same time, collecting the initial tacrolimus dosage of the liver transplant recipients after surgery for comparison with the model predicted dosage.

[0028] Specifically, the table type data includes case data, routine laboratory examination data, tissue elasticity data, pharmacokinetics data and important immune checkpoint data, wherein:

[0029] The case data includes primary diseases of liver transplantation, past medical history, family history, the primary diseases include end-stage liver disease and acute liver failure, the past medical history includes disease history, operation history, treatment history, drug allergy history, and the family history includes genetic diseases and drug metabolism capacity;

[0030] The routine laboratory examination data includes 7 kinds of examination data of blood routine, coagulation function, kidney function, electrolyte, liver function, C-reactive protein and immunoglobulin level;

[0031] The tissue elasticity data includes liver tissue hardness;

[0032] The pharmacokinetics data includes tacrolimus blood concentration and metabolic enzyme genotype;

[0033] The important immune checkpoint data includes other immune suppression drug concentration detection, immune cell analysis and cytokine level; the other immune suppression drug concentration detection mainly includes cyclosporine A concentration detection, sirolimus concentration detection and everolimus concentration detection;

[0034] The image type data includes imaging result data, and the imaging result data includes liver ultrasound, liver CT and liver MRI.

[0035] Specifically, the process of step S1 for processing the clinical data of each liver transplant recipient includes:

[0036] S11. Preprocessing the table type data in the clinical data of the liver transplant recipient to obtain preprocessed data; the preprocessing includes missing value and abnormal value processing, normalization and standardization processing.

[0037] Specifically, the preprocessing process includes:

[0038] S111. Using a machine learning model to predict the missing values in the table type data, and filling fixed elements, generally 0 elements, in the missing positions. Predicting the missing values by machine learning can ensure the rationality of the data, improve the data quality, maintain the reasonable distribution and characteristics of the original data, automatically process and fill the missing values by machine learning, and also reduce manual intervention, improve the efficiency and consistency of data processing.

[0039] S112. Using statistical analysis and data visualization methods to identify abnormal values in the table type data processed in step S111, and filling fixed elements in the abnormal positions.

[0040] S113. Using a standardization method, the table type data processed in step S112 is converted into a distribution with zero mean and unit variance;

[0041] S114. The distribution in step S113 is normalized to obtain preprocessed data.

[0042] S12. A univariate analysis method is used to screen a plurality of significant variables from the preprocessed data, and all the significant variables are combined into a table sample.

[0043] Specifically, for each binary variable in the preprocessed data, a chi-square test method is used to determine whether the binary variable is significantly related to the tacrolimus dosage per unit time. If so, the binary variable is taken as a significant variable. Specifically, the basic principle of the chi-square test is to compare the difference between the observed frequency and the expected frequency under the null hypothesis. The chi-square statistic is the sum of the squares of the differences between the observed frequency and the expected frequency divided by the expected frequency. This value reflects the degree of deviation between the actual observation and the expected distribution. If the chi-square value is significantly greater than the critical value of the chi-square distribution under a given degree of freedom and a significance level, or the calculated chi-square value is less than the significance level, it is considered that there is a correlation between the two variables, i.e. the analyzed variables are not independent. In the present application, the two variables here refer to the binary variable and the tacrolimus dosage per unit time.

[0044] For each multi-class variable in the preprocessed data, an analysis of variance method is used to determine whether the multi-class variable is significantly related to the tacrolimus dosage per unit time. If so, the multi-class variable is taken as a significant variable. Specifically, the analysis of variance method assesses whether the independent variable has a significant effect on the dependent variable by calculating the between-group and within-group sum of squares to obtain the F statistic, thereby determining whether there is a statistically significant difference between different groups. The core lies in testing the difference between the mean of each group and the total mean and the significance of the difference between groups. In the present application, the independent variable here refers to the multi-class variable, and the dependent variable refers to the tacrolimus dosage per unit time.

[0045] For each continuous variable in the preprocessed data, a Spearman rank correlation coefficient analysis method is used to determine whether the continuous variable is significantly related to the tacrolimus dosage per unit time. If so, the continuous variable is taken as a significant variable. Specifically, the Spearman rank correlation coefficient analysis method measures the degree of correlation between two variables by evaluating the correlation between their ranks. The sum of squares of the rank differences of each data point is calculated, and then the sum of squares is adjusted according to the total number of observations. Finally, a coefficient between -1 and 1 is obtained, which reflects the correlation between the variables, where a positive value indicates a positive correlation, a negative value indicates a negative correlation, and a value close to 0 indicates no significant correlation. In the present application, the two variables here refer to the continuous variable and the tacrolimus dosage per unit time.

[0046] Specifically, the dichotomous variable includes: end-stage liver disease, acute liver failure, disease history, surgery history, treatment history, drug allergy history, genetic disease;

[0047] The polytomous variable includes: drug metabolism capacity, metabolic enzyme genotype, other immunosuppressive drug concentration detection, tacrolimus blood concentration;

[0048] The continuous variable includes: blood routine, coagulation function, renal function, electrolyte, liver function, C-reactive protein, immunoglobulin level, liver tissue stiffness, immune cell analysis, cytokine level.

[0049] S13. Image samples are obtained by performing enhancement processing on the image data in the clinical data of the same liver transplant recipient, and the enhancement processing includes scaling and normalization.

[0050] S14. The table sample and the image sample are combined into one sample.

[0051] S2. A multi-modal linear regression model is constructed, which includes a feature extraction module, a feature dimension reduction module, an attention mechanism module, a multi-modal fusion module, and a prediction module.

[0052] S3. The samples in the training set are passed through the feature extraction module to obtain the original feature sequence.

[0053] Specifically, step S3 includes:

[0054] S31. The image samples in the samples are processed by using a pre-trained ResNet50 to obtain the original image feature sequence.

[0055] Specifically, a convolutional layer is used at the input layer of the pre-trained ResNet50 to perform preliminary feature extraction on the image samples, and then a series of residual block operations are performed to further extract features. Each residual block contains multiple convolutional layers, batch normalization layers and ReLU activation functions. Residual connection helps to solve the problem of gradient disappearance. Finally, a global average pooling layer is used to reduce the spatial dimension and improve the stability of the features. The output of the last layer of the pre-trained ResNet50 is used as the original image feature sequence.

[0056] S32. The table samples in the samples are processed by using a fully connected network layer to obtain the original table feature sequence.

[0057] Specifically, the fully connected network layer includes a batch normalization layer and a ReLU activation function. The hidden layer of the fully connected network layer can learn the nonlinear features of the data, in addition to which the batch normalization layer helps to accelerate the training process, reduce the sensitivity of the initialized weights, and prevent overfitting. The ReLU activation function is added after the batch normalization layer, which can help to introduce nonlinearity, enabling the model to learn more complex data patterns.

[0058] S33. The original image feature sequence and the original table feature sequence are combined into an original feature sequence.

[0059] S4. The original feature sequence is input into a feature dimension reduction module VAE to obtain a dimension-reduced feature sequence.

[0060] Specifically, as shown in Figure 3 the feature dimension reduction module includes an encoder and a decoder; inputting the original feature sequence into the feature dimension reduction module to obtain the dimension-reduced feature sequence includes:

[0061] S41. The original image feature sequence is input into the encoder to obtain a first mean and a first standard deviation; a first latent variable is generated by using a reparameterization method;

[0062] S42. The decoder performs reconstruction processing on the first latent variable to obtain an image dimension-reduced feature sequence;

[0063] S43. The original table feature sequence is input into the encoder to obtain a second mean and a second standard deviation; a second latent variable is generated by using a reparameterization method;

[0064] S44. The decoder performs reconstruction processing on the second latent variable to obtain a table dimension-reduced feature sequence;

[0065] S45. The image dimension-reduced feature sequence and the table dimension-reduced feature sequence are combined into a dimension-reduced feature sequence.

[0066] Specifically, the VAE first maps the high-dimensional input data (the original image feature sequence or the original table feature sequence) to a parameterized representation of a low-dimensional latent space through the encoder, i.e., learns the latent variable distribution of the input data; in this process, the encoder outputs the latent variable mean and the latent variable standard deviation; the reparameterization method is used to add normal distribution noise with the latent variable standard deviation as the standard deviation around the latent variable mean to generate the latent variable, represented as:

[0067] c=m+n⊙e

[0068] where c denotes the first latent variable or the second latent variable, m denotes the first mean or the second mean, n denotes the first standard deviation or the second standard deviation; e denotes the first normal distribution noise with the first standard deviation as the standard deviation, or denotes the second normal distribution noise with the second standard deviation as the standard deviation; denotes element-level multiplication; the latent variable can be regarded as a compressed representation of the input data, which encodes the main features and structures of the input data.

[0069] The decoder attempts to reconstruct the original feature sequence from this low-dimensional latent variable c. The training objective of the VAE is to maximize the evidence lower bound (ELBO):

[0070]

[0071] ELBO measures the weight between the log-likelihood of the reconstructed data and the latent space regularization term, where the first term is the expected data reconstruction log-likelihood under the latent variable c, which measures the quality of using the latent variable c to reconstruct the input data x. The second term is the Kullback-Leibler divergence between the latent representation distribution q(z|x) and the prior distribution p(z), which measures the difference between the two probability distributions. By maximizing the evidence lower bound, a balance point is found so that the model can both reconstruct the data well and keep the distribution of the latent representation close to the prior distribution, thereby generating a good quality and generalization ability of the data representation.

[0072] S5. Input the reduced dimension feature sequence into the attention module to obtain an enhanced feature sequence.

[0073] Specifically, the attention module includes a PLGA module and a PLGCroA module; the PLGA module captures global and local features within the modal; the PLGCroA module jointly captures global and local features between the modal. Input the reduced dimension feature sequence into the attention module to obtain an enhanced feature sequence, including:

[0074] S51. Input the image reduced dimension feature sequence into the PLGA module to obtain an image splicing feature representation;

[0075] S52. Input the table reduced dimension feature sequence into the PLGA module to obtain a table splicing feature representation;

[0076] Specifically, as shown in Figure 4 the PLGA module includes a local attention unit and a global attention unit, and inputting the image reduced dimension feature sequence into the PLGA module to obtain an image splicing feature representation includes:

[0077] S511. Input the image reduced dimension feature sequence into four linear layers respectively to obtain key values, query values, global values and local values, which can be represented as:

[0078] Q = WQ X X

[0079] K = W K X X

[0080]

[0081] where Q denotes query value, W Q denotes query weight matrix, X denotes image reduced dimension feature sequence, K denotes key value, W K denotes key weight matrix, V l denotes local value, V g denotes global value, denotes local weight matrix, denotes global weight matrix. Wherein, the key value and the query value are shared in the local attention unit and the global attention unit.

[0082] S512. The local attention unit focuses on learning spatial information. First, the key value is transposed and multiplied by the query value to obtain the local attention map; then the local attention map is input into the softmax layer to apply the softmax operation to quantify the similarity between each feature and the remaining spatial features to obtain the local feature similarity; then the local feature similarity is multiplied by the local value to obtain the local feature, which can be represented as:

[0083]

[0084] where d is the feature dimension, denotes local feature.

[0085] S513. The global attention module can effectively capture the mutual dependence relationship between feature channels. First, the query value is transposed and multiplied by the key value to obtain the global attention map, which is used to quantify the similarity between each individual channel and other channels; then the global attention map is input into the softmax layer to apply the softmax operation to quantify the similarity between each feature and the remaining spatial features to obtain the global feature similarity; then the global feature similarity is multiplied by the global value to obtain the global feature, which can be represented as:

[0086]

[0087] wherein, denotes global feature.

[0088] S514. The local feature and the global feature are fused, and the fusion result is skip-connected with the image reduced dimension feature sequence to obtain the image splicing feature representation, which can be represented as:

[0089]

[0090] wherein, The image splicing feature representation is obtained. Similarly, the table splicing feature representation is obtained by inputting the table dimension reduction feature sequence into the PLGA module and performing the above operations.

[0091] S53. The image splicing feature representation and the table splicing feature representation are input into the PLGCroA module to obtain the image enhanced feature representation and the table enhanced feature representation.

[0092] Specifically, as shown in Figure 5 , the PLGCroA module also includes a local attention unit and a global attention unit, and step S53 includes:

[0093] S531. The table splicing feature representation is input into 4 linear layers to obtain key values K t , query values Q t , global values V gt , and local values V lt ; and the image splicing feature representation is input into another 4 linear layers to obtain key values K i , query values Q i , global values V gi , and local values V li .

[0094]

[0095]

[0096] wherein W i Q , W i K , W t Q , are weight matrices.

[0097] S532. The key values K t are transposed and multiplied by the query values Q i to obtain a first local attention map, the first local attention map is input into a softmax layer to obtain a first local similarity, and the first local similarity is multiplied by the local values V lt to obtain a first cross local feature

[0098]

[0099] S533. The query values Q i are projected by a fully connected layer to align with the spatial dimensions of the table splicing feature representation to obtain aligned query values Q proi , the aligned query values Q proi are transposed and multiplied by the key values Kt get the first global similarity by passing the first global attention map through a softmax layer, multiply the first global similarity with the global value V gi get the first cross-global feature

[0100]

[0101] S534. Fuse the first cross-local feature and the first cross-global feature, and perform skip concatenation on the fusion result and the image splicing feature representation to obtain an image enhanced feature representation

[0102]

[0103] S535. Perform transpose on the key value K i , and multiply the result by the query value Q t get the second local similarity by passing the second local attention map through a softmax layer, multiply the second local similarity with the local value V li get the second cross-local feature

[0104] S536. Project the query value Q t through a fully connected layer to obtain an aligned query value Q prot , perform transpose on the aligned query value Q prot , and multiply the result by the key value K i get the second global similarity by passing the second global attention map through a softmax layer, multiply the second global similarity with the global value V gt get the second cross-global feature

[0105] S537. Fuse the second cross-local feature and the second cross-global feature, and perform concatenation on the fusion result and the table splicing feature representation to obtain a table enhanced feature representation.

[0106] PLGCroA combines the features of two modalities to create a unified representation, thereby providing a more comprehensive and accurate understanding of the data; in addition, PLGCroA helps to extract features according to the context provided by one modality feature under the guidance of another modality feature, which will produce richer and more unique feature representations.

[0107] S6. Input the enhanced feature sequence into a multi-modal feature fusion module to obtain a fused feature sequence.

[0108] Specifically, the multi-modal fusion module is mainly implemented by a gating attention mechanism with bilinear product. Bilinear product is usually used to calculate the interaction between two embedding vectors, which can help the model understand the relationship between different modal features. Mainly includes:

[0109] S61. Set the first weight and the second weight through the gating mechanism; Specifically, for the enhanced feature representation of the two modalities, a gating mechanism is used to dynamically adjust the weight in the fusion process, which is realized through a parameterized function, here a fully connected layer with a sigmoid activation function is used, which outputs a weight of 0-1, representing the importance of the modality, and the corresponding weight is set as g1, g2;

[0110] S62. Multiply the image enhanced feature representation and the first weight to obtain the image feature embedding Multiply the table enhanced feature representation and the second weight to obtain the table feature embedding which can be represented as:

[0111]

[0112] S63. Calculate the interaction between the image feature embedding and the table feature embedding through bilinear product to obtain the fusion feature sequence, denoted as:

[0113]

[0114] where W ij is a learnable weight matrix used to transform and fuse information from the two modalities, and finally calculate the output fusion feature sequence

[0115] S7. The output feature passes through the prediction module to obtain the prediction result; calculate the loss according to the prediction result and the initial tacrolimus dose, and train and optimize the model parameters through back propagation until the model parameters converge.

[0116] Specifically, the prediction module is composed of MLP, which is composed of an input layer, multiple hidden layers and an output layer, each layer is composed of multiple neurons, and there is a weight connection between neurons; data passes through the hidden layer from the input layer, each layer uses weights and biases for linear combination, and then passes through a nonlinear activation function, and finally passes to the output layer; the hidden layer usually uses a nonlinear activation function sigmoid to introduce nonlinear characteristics and enhance the expression ability of the model; the mean square error loss function is used to calculate the difference between the predicted value and the true value, the gradient of the loss function with respect to the network parameters is calculated through the back propagation algorithm, and then the Adam optimizer is used to update the weights and biases.

[0117] Penalize large errors: when the prediction error is large, the value of the loss function will increase rapidly, so that the model pays more attention to the samples with large prediction errors during optimization, so as to better fit the extreme cases in the data.

[0118] Minimization strategy: minimizing the mean square error means that the predicted value is as close to the actual value as possible, which facilitates the optimization algorithm to update the parameters according to the loss function and gradually approaches the global or local minimum point.

[0119] Adam optimizer is an extension of the gradient descent optimization algorithm, which combines the advantages of adaptive gradient algorithm and root mean square propagation algorithm.

[0120] By adaptively adjusting the learning rate to optimize the parameters of the neural network model, the learning rate is adjusted according to the size and stability of the gradient during the training process, thereby improving the convergence speed and stability of the model.

[0121] S8. Obtain the to-be-processed clinical data, input the trained multi-modal information linear regression model, and obtain the initial tacrolimus use dose prediction result.

[0122] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "setting", "connecting", "fixing", "rotating" and the like should be understood broadly, for example, it can be fixed connection, or detachable connection, or integrated; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited, the above-mentioned terms in the present application can be understood according to the specific meaning according to the specific circumstances by those skilled in the art.

[0123] Although the embodiments of the present application have been shown and described, those skilled in the art can understand that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-modal based drug dosage prediction method for liver transplantation, characterized by, The method comprises the following steps: S1. Collecting clinical data of a liver transplant recipient, the clinical data comprising table type data and image type data; processing all the clinical data to obtain a sample set, dividing the sample set into a training set and a test set; each sample in the sample set comprising a table sample and an image sample; S2. Constructing a multi-modal information linear regression model, comprising a feature extraction module, a feature dimension reduction module, an attention mechanism module, a multi-modal fusion module, and a prediction module; S3. Obtaining an original feature sequence of a sample in the training set through the feature extraction module, the original feature sequence comprising an original image feature sequence and an original table feature sequence; S31. Processing the image sample in the sample by using a pre-trained ResNet50 to obtain the original image feature sequence; S32. Processing the table sample in the sample by using a fully connected network layer to obtain the original table feature sequence; S4. Inputting the original feature sequence into the feature dimension reduction module to obtain a dimension-reduced feature sequence, the dimension-reduced feature sequence comprising an image dimension-reduced feature sequence and a table dimension-reduced feature sequence; The feature dimension reduction module comprises an encoder and a decoder; inputting the original feature sequence into the feature dimension reduction module to obtain the dimension-reduced feature sequence comprises: S41. Inputting the original image feature sequence into the encoder to obtain a first mean and a first standard deviation, and generating a first latent variable by using a reparameterization method; S42. The decoder reconstructs the first latent variable to obtain the image dimension-reduced feature sequence; S43. Inputting the original table feature sequence into the encoder to obtain a second mean and a second standard deviation, and generating a second latent variable by using a reparameterization method; S44. The decoder reconstructs the second latent variable to obtain the table dimension-reduced feature sequence; S45. Combining the image dimension-reduced feature sequence and the table dimension-reduced feature sequence into the dimension-reduced feature sequence; S5. Inputting the dimension-reduced feature sequence into the attention mechanism module to obtain an enhanced feature sequence, the enhanced feature sequence comprising an image enhanced feature representation and a table enhanced feature representation; The attention module comprises a PLGA module and a PLGCroA module; inputting the dimension-reduced feature sequence into the attention module to obtain the enhanced feature sequence comprises: S51. Inputting the image dimension-reduced feature sequence into the PLGA module to obtain an image splicing feature representation; S52. Inputting the table dimension-reduced feature sequence into the PLGA module to obtain a table splicing feature representation; S53. Inputting the image splicing feature representation and the table splicing feature representation into the PLGCroA module to obtain the image enhanced feature representation and the table enhanced feature representation; S6. Inputting the enhanced feature sequence into the multi-modal fusion module to output a fusion feature sequence; The multi-modal fusion module is realized by a bilinear product-based gated attention mechanism, comprising: S61. Setting a first weight and a second weight through a gating mechanism; S62. Multiplying the image enhanced feature representation and the first weight to obtain an image feature embedding; multiplying the table enhanced feature representation and the second weight to obtain a table feature embedding; S63. Calculating the image feature embedding and the table feature embedding by a bilinear product to obtain the fusion feature sequence; ​ S7. The fusion feature sequence is input into the prediction module to obtain a prediction result, a loss is calculated according to the prediction result, and the model parameters are trained and optimized in a reverse propagation manner until the model parameters converge; S8. The obtained clinical data to be processed is input into the trained multi-modal information linear regression model to obtain a dosage prediction result.

2. The multi-modal based drug dosage prediction method for liver transplantation according to claim 1, wherein, The table type data includes case data, routine laboratory examination data, tissue elasticity data, pharmacokinetic data, and important immune checkpoint data, wherein: The case data includes primary diseases of liver transplantation, medical history, and family history; the primary diseases include end-stage liver disease and acute liver failure, the medical history includes disease history, operation history, treatment history, and drug allergy history, and the family history includes genetic diseases and drug metabolism capacity; The routine laboratory examination data includes 7 kinds of examination data including blood routine examination, blood coagulation function, kidney function, electrolyte, liver function, C-reactive protein, and immunoglobulin level; The tissue elasticity data includes liver tissue hardness; The pharmacokinetic data includes tacrolimus blood concentration and metabolic enzyme genotype; The important immune checkpoint data includes other immune suppressant drug concentration detection, immune cell analysis, and cytokine level; The image type data includes imaging result data, and the imaging result data includes liver ultrasound, liver CT, and liver MRI.

3. The multi-modal based drug dosage prediction method for liver transplantation according to claim 1, wherein, The process of processing the clinical data of each liver transplant recipient in step S1 includes: S11. Preprocessing the table type data in the clinical data of the liver transplant recipient to obtain preprocessed data; the preprocessing includes missing value and abnormal value processing, normalization and standardization processing; S12. Screening a plurality of significant variables from the preprocessed data by using a univariate analysis method, and combining all the significant variables into a table sample; S13. Enhancing the image type data in the clinical data of the same liver transplant recipient to obtain an image sample, and the enhancement processing includes scaling and normalization; S14. Combining the table sample and the image sample into a sample.

4. The multi-modal based drug dosage prediction method for liver transplantation according to claim 3, wherein, Step S12 includes: For each binary variable in the preprocessed data, a chi-square test method is used to determine whether the relationship between the binary variable and the tacrolimus dosage per unit time is significant, if so, the binary variable is taken as a significant variable; For each multi-class variable in the preprocessed data, a variance analysis method is used to determine whether the relationship between the multi-class variable and the tacrolimus dosage per unit time is significant, if so, the multi-class variable is taken as a significant variable; For each continuous variable in the preprocessed data, a Spearman rank correlation coefficient analysis method is used to determine whether the relationship between the continuous variable and the tacrolimus dosage per unit time is significant, if so, the continuous variable is taken as a significant variable.

5. The multi-modal based drug dose prediction method for liver transplantation according to claim 1, wherein, Inputting the image dimension reduction feature sequence into the PLGA module to obtain image splicing feature representation includes: S511. Inputting the image dimension reduction feature sequence into 4 linear layers to obtain key values, query values, global values, and local values; S512. Transposing the key values and multiplying them by the query values to obtain a local attention map, inputting the local attention map into a softmax layer to obtain a local feature similarity, and multiplying the local feature similarity by the local value to obtain a local feature; S513. After transposing the query value, multiply the key value to obtain the global attention map, input the global attention map into the softmax layer to obtain the global feature similarity, and multiply the global feature similarity with the global value to obtain the global feature; S514. Fuse the local feature and the global feature, and splice the fusion result with the image dimension reduction feature sequence to obtain the image splicing feature representation.

6. The multi-modal based drug dose prediction method for liver transplantation according to claim 1, wherein, Step S53 specifically includes: S531. Calculate the key value K corresponding to the image stitching feature representation through a linear layer. i Query value Q i Global value V gi and local value V li The key value K is calculated by concatenating the features of the table using a linear layer. t Query value Q t Global value V gt and local value V lt ; S532. Transpose the key value K t and multiply by the query value Q i to obtain a first local attention map. The first local attention map is passed through a softmax layer to obtain a first local similarity. The first local similarity is multiplied by the local value V lt to obtain a first cross-local feature. S533. Query value Q is processed through a fully connected layer. i Projection is performed to obtain the alignment query value Q proi For the aligned query value Q proi After transposing, multiply by the key value K t The first global attention map is obtained, and then passed through a softmax layer to obtain the first global similarity. The first global similarity is then compared with the global value V. gi Multiplication yields the first cross-regional global feature; S534. Fuse the first cross local feature and the first cross global feature, splice the fusion result with the image splicing feature representation to obtain the image enhanced feature representation; S535. transpose the key value K i and multiply by the query value Q t to obtain a second local attention map, pass the second local attention map through a softmax layer to obtain a second local similarity, multiply the second local similarity by the local value V li to obtain a second cross-local feature S536. projecting the query value Q through a fully connected layer to obtain an aligned query value Q t S538. transposing the aligned query value Q prot and multiplying it by the key value K to obtain a second global attention map prot S540. passing the second global attention map through a softmax layer to obtain a second global similarity i S542. multiplying the second global similarity with the global value V to obtain a second cross-global feature gt S544. obtaining a second global feature by adding the first cross-global feature and the second cross-global feature S537. Fuse the second cross local feature and the second cross global feature, and splice the fusion result with the table splicing feature representation to obtain the table enhanced feature representation.

Images