Liver transplantation medicine dosage prediction method based on multiple modes

Abstract

The invention belongs to the technical field of biological medicine and data processing, and particularly relates to a multi-modal-based liver transplantation drug dosage prediction method, which comprises the following steps: collecting clinical data of liver transplantation recipients and processing to obtain a sample set, and dividing the sample set into a training set and a test set; constructing a multi-modal linear regression model; samples in the training set pass through a feature extraction module to obtain an original feature sequence; inputting the original feature sequence into a feature dimension reduction module to obtain a dimension reduction feature sequence; inputting the dimension reduction feature sequence into an attention mechanism module to obtain an enhanced feature sequence; the enhanced feature sequence passes through a multi-modal fusion module to obtain a fused feature sequence; inputting the fused feature sequence into a prediction module to obtain a prediction result, and training optimization model parameters; obtaining to-be-processed clinical data, inputting the to-be-processed clinical data into the trained multi-modal linear regression model, and obtaining a medicament dosage prediction result; according to the method, the individualized dosage of tacrolimus can be accurately predicted.

Description

Technical Field

[0001] The present invention belongs to the field of biomedicine and data processing technology, and specifically relates to a method for predicting drug dosage for liver transplantation based on multimodality. Background Art

[0002] Liver transplantation has made great progress worldwide, but the occurrence of postoperative rejection is still one of the most difficult problems in liver transplantation. In the occurrence and development of rejection, the liver transplant recipient T cells recognize the allogeneic antigens of the donor liver, initiate the recipient immune response, cause portal inflammation and bile duct damage, and damage liver function. Among them, the incidence of acute rejection is as high as 10%-40%, which seriously affects the prognosis of liver transplant recipients. Due to the joint participation of innate immunity and adaptive immunity in liver transplant rejection, involving multiple complex molecular events, its intrinsic pathogenesis cannot be fully explained. In clinical practice, the main means of preventing and treating rejection is the rational use of immunosuppressive drugs.

[0003] The liver is an "immune-exempt" organ. The rational use of immunosuppressive drugs can effectively prevent and treat some liver transplant rejection reactions. In fact, liver transplant recipients currently need to take immunosuppressive drugs regularly to prevent and treat rejection reactions. Among them, tacrolimus has a stronger immunosuppressive effect and fewer side effects than other immunosuppressants, and has become the immunosuppressive drug of choice for liver transplant recipients. The advent of immunosuppressive drugs has improved the prognosis of liver transplant recipients to a certain extent. However, due to the lack of individualized medication concepts and standardized medication processes, the precise use of immunosuppressants for liver transplant recipients is a global problem, one of the specific manifestations of which is the inaccurate dosage of medication.

[0004] At present, various liver transplant centers widely use blood drug concentration testing as the main basis for adjusting drug dosage to improve the recipient's medication regimen. Blood drug concentration refers to the concentration of the drug in the blood, which reflects the distribution and metabolism of the drug in the body. For drugs such as immunosuppressants that need to maintain a certain concentration range to exert the best therapeutic effect, the level of blood drug concentration directly affects its therapeutic effect. Through blood drug concentration testing, we can accurately understand the concentration level of the drug in the body, so as to determine whether the dosage needs to be adjusted, and ensure that the concentration of the drug in the patient's body is maintained within the optimal therapeutic range, so as to achieve the best therapeutic effect and reduce the occurrence of adverse reactions. The clinical application of blood drug concentration testing has been widely recognized and supported. Many studies have shown that guiding the dosage adjustment of immunosuppressants through blood drug concentration testing can significantly improve the treatment effect and survival rate of liver transplant recipients, and reduce the occurrence of rejection and adverse reactions. However, there are still many problems with this program:

[0005] (1) Blood drug concentration testing is done after taking the drug, which has a lag, and it takes a certain amount of time for the recipient to reach a stable blood drug concentration. The peak period of acute rejection after liver transplantation is exactly one week to one month after surgery. Lagging liver transplant medication adjustments will miss the golden medication period, that is, from the first dose of immunosuppressive drugs before surgery to the combined immunosuppressive drugs one week after surgery. At this time, a forward-looking medication guide is particularly important.

[0006] (2) Blood drug concentration results can be used as a basis for improving subsequent medication dosage and method, but before this, medication is often mechanically unified, or similar medication dosages are used based on literature reports and previous experience, or the initial dose is adjusted based on individual factors of the recipient. Due to insufficient data integration, preoperative and early postoperative medication lacks systematicity and standardization, but at this stage, it is crucial to rationally select the type and initial dose of immunosuppressants and reach the ideal required concentration in a timely manner.

[0007] At present, many gene mutations that are highly correlated with the blood concentration of immunosuppressants in liver transplantation have been reported, but there is a lack of systematic integrated research. On the other hand, there are large differences in the basic conditions of recipients, including age, weight, underlying diseases, etc. Existing liver transplantation drug dosage prediction methods usually only consider single-modal data such as blood drug concentration test results, and do not consider the correlation between different modal data; at the same time, there are many factors that may affect liver transplantation medication, and there are personalized differences between different individuals. The existing methods do not distinguish the importance of different influencing factors. Therefore, how to effectively integrate different modal data and clarify the proportion of each influencing factor is crucial to predicting the individualized dosage of medication for each liver transplant recipient. Summary of the invention

[0008] To solve the above problems, the present invention provides a method for predicting drug dosage for liver transplantation based on multimodality, which uses artificial intelligence methods to learn the nonlinear, high-dimensional, and complex correlation between various physiological data of patients and blood drug concentrations and drug dosages, and predicts the dosage of tacrolimus that patients need to take after liver transplantation, thereby improving the accuracy of the prediction results.

[0009] Specific solutions include:

[0010] S1. Collect clinical data of liver transplant recipients, wherein the clinical data includes table data and image data; process all clinical data to obtain a sample set, and divide the sample set into a training set and a test set; and collect the initial tacrolimus dosage of liver transplant recipients after surgery; each sample in the sample set includes a table sample and an image sample;

[0011] S2. Construct a multimodal information linear regression model, which includes a feature extraction module, a feature dimension reduction module, an attention mechanism module, a multimodal fusion module, and a prediction module;

[0012] S3. The samples in the training set are extracted by a feature extraction module to obtain an original feature sequence, wherein the original feature sequence includes an original image feature sequence and an original table feature sequence;

[0013] S4. Inputting the original feature sequence into the feature dimension reduction module to obtain a reduced dimension feature sequence, wherein the reduced dimension feature sequence includes an image reduced dimension feature sequence and a table reduced dimension feature sequence;

[0014] S5. Inputting the dimension reduction feature sequence into the attention mechanism module to obtain an enhanced feature sequence, wherein the enhanced feature sequence includes an image enhanced feature representation and a table enhanced feature representation;

[0015] S6. inputting the enhanced feature sequence into the multimodal fusion module, and outputting a fused feature sequence;

[0016] S7. The fused feature sequence is passed through the prediction module to obtain the prediction result, and the loss is calculated according to the prediction result and the model parameters are optimized through back propagation training until the model parameters converge;

[0017] S8. Obtain the clinical data to be processed and input it into the trained multimodal information linear regression model to obtain the drug dosage prediction result. Beneficial effects of the present invention:

[0018] The method for predicting drug dosage for liver transplantation provided by the present invention is based on the clinical data of real liver transplant recipients, and uses statistical methods to screen significant variables that affect individualized drug dosage. Based on this, the data set is further optimized to construct a multimodal information linear regression model, and the pre-trained ResNet50 is used to extract image features. The fully connected network layer including a normalization layer and a ReLU activation function is used to extract tabular data features. The VAE module is used to achieve feature dimensionality reduction, and PLGA and PLGCroA are used to capture global features and local features within and between modalities, respectively. The gated attention mechanism with a bilinear product is used to fuse multimodal data, and MLP is used for the final prediction. The prediction accuracy of the model is evaluated based on MAE, R2, MSE, RMSE and Real Rate, and the mean square error loss function and Adam optimizer are used to tune the model parameters to improve the model performance. It is difficult to integrate different types of data when predicting personalized tacrolimus dosage based on traditional machine learning methods, and it is difficult for the model to simultaneously ensure accuracy and interpretability when processing complex data. The present invention effectively solves this problem. The attention module of the prediction model can capture the complex dependencies between and within modalities from a global and local perspective, respectively. The visualization of the attention weights provides a visual understanding of the model decision-making process. In summary, the multimodal-based prediction model can accurately predict the personalized tacrolimus dosage while its transparency and interpretability are improved compared to traditional models. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method of the present invention;

[0020] Figure 2 It is a schematic diagram of the model of the present invention;

[0021] Figure 3 This is the VAE structure diagram of the feature dimension reduction module of the present invention;

[0022] Figure 4 This is the structural diagram of the attention mechanism module PLGA of the present invention;

[0023] Figure 5 This is the structural diagram of the attention mechanism module PLGCroA of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Existing methods for predicting drug dosage for liver transplantation mainly focus on single-modality data, such as interactions between genes or influencing factors between drugs, while ignoring the importance of clinical test data and related medical imaging data for drug dosage prediction. For example, clinical test data contains rich individualized health information and can provide direct indicators of the patient's physiological and biochemical status. CT imaging data can intuitively reflect the patient's tissue anatomical structure and pathological changes. Moreover, single-modality data often cannot fully understand complex problems. Failure to consider the complementarity and correlation between multimodal data may not achieve optimal prediction accuracy, and may even lead to deviations in prediction results. In addition, existing prediction methods do not fully consider the influencing factors of liver transplantation, and do not distinguish the importance of different influencing factors. The constructed models lack interpretability, making it difficult to provide clear treatment guidance for doctors in clinical practice.

[0026] To this end, the present invention provides a method for predicting drug dosage for liver transplantation based on multimodality, such as Figure 1 , Figure 2 As shown, the following steps are included:

[0027] S1. Collect clinical data of liver transplant recipients, the clinical data including table data and image data; process all clinical data to obtain a sample set, and divide the sample set into a training set and a test set; and collect the initial tacrolimus dosage of liver transplant recipients after surgery for comparison with the dosage predicted by the model.

[0028] Specifically, the table data includes case data, routine laboratory test data, tissue elasticity data, pharmacokinetic data, and important immune checkpoint data, among which:

[0029] Case data included primary disease, past history, and family history of liver transplantation. Primary disease included end-stage liver disease and acute liver failure. Past history included disease history, surgical history, treatment history, and drug allergy history. Family history included hereditary diseases and drug metabolism ability.

[0030] Routine laboratory test data include blood routine, coagulation function, renal function, electrolytes, liver function, C-reactive protein, and immunoglobulin levels, a total of 7 test data;

[0031] Tissue elasticity data included liver tissue stiffness;

[0032] Pharmacokinetic data included tacrolimus plasma concentrations, metabolizing enzyme genotypes;

[0033] Important immune checkpoint data include other immunosuppressive drug concentration detection, immune cell analysis, and cytokine levels; other immunosuppressive drug concentration detection mainly includes cyclosporine A concentration detection, sirolimus concentration detection, and everolimus concentration detection;

[0034] The image data includes imaging result data, and the imaging result data includes liver ultrasound, liver CT, and liver MRI.

[0035] Specifically, the process of processing the clinical data of each liver transplant recipient in step S1 includes:

[0036] S11. Preprocessing the tabular data in the clinical data of the liver transplant recipients to obtain preprocessed data; the preprocessing includes missing value and abnormal value processing, normalization and standardization processing.

[0037] Specifically, the preprocessing process includes:

[0038] S111. Use machine learning models to predict missing values ​​in table data and fill in fixed elements at the missing positions, usually 0 elements. Predicting missing values ​​through machine learning can ensure the rationality of data, improve data quality, maintain the reasonable distribution and characteristics of original data, and automatically process and fill missing values ​​through machine learning, which can also reduce manual intervention and improve the efficiency and consistency of data processing.

[0039] S112. Use statistical analysis and data visualization methods to identify abnormal values ​​in the table data processed in step S111, and fill the abnormal positions with fixed elements.

[0040] S113. Using a standardization method, the tabular data processed in step S112 is converted into a distribution having a zero mean and a unit variance;

[0041] S114. Normalize the distribution of step S113 to obtain preprocessed data.

[0042] S12. Use univariate analysis to screen out multiple significant variables from the preprocessed data and combine all significant variables into a table sample.

[0043] Specifically, for each binary variable in the preprocessed data, the chi-square test method is used to determine whether its relationship with the tacrolimus dosage per unit time is significant. If so, the binary variable is used as a significant variable. Specifically, the basic principle of the chi-square test is to compare the difference between the observed frequency and the expected frequency when the null hypothesis is established. The statistic of the chi-square test is the chi-square value, which is the value obtained by dividing the sum of the squares of the difference between the observed frequency and the expected frequency by the expected frequency. This value reflects the degree of deviation between the actual observation and the expected distribution. If the chi-square value is significantly greater than the critical value of the chi-square distribution under a given degree of freedom and significance level, or the calculated chi-square value is less than the significance level, it is considered that there is an association between the two variables, that is, the analyzed variables are not independent of each other. In the present invention, the two variables here refer to the binary variable and the tacrolimus dosage per unit time.

[0044] For each multi-classification variable in the pre-processed data, the variance analysis method is used to determine whether the relationship between it and the tacrolimus dosage per unit time is significant. If so, the multi-classification variable is used as a significant variable. Specifically, the variance analysis method calculates the sum of squares between and within groups to obtain the F statistic to evaluate whether the influence of the independent variable on the dependent variable is significant, thereby determining whether there is a statistically significant difference between different groups. The core of the variance analysis method is to test the difference between the mean of each group and the total mean and the significance of the difference between the groups. In the present invention, the independent variable here refers to the multi-classification variable, and the dependent variable refers to the dosage of tacrolimus per unit time.

[0045] For each continuous variable in the preprocessed data, the Spearman rank correlation coefficient analysis method is used to determine whether its relationship with the tacrolimus dosage per unit time is significant. If so, the continuous variable is used as a significant variable. Specifically, the Spearman rank correlation coefficient analysis method measures the degree of association between the two variables by evaluating the correlation between the ranks of the two variables, and calculates the sum of squares of the rank differences of each data point, and then adjusts the sum of squares according to the total number of observations, and finally obtains a coefficient between -1 and 1, reflecting the correlation between the variables, wherein a positive value indicates a positive correlation, a negative value indicates a negative correlation, and a value close to 0 indicates no significant correlation. In the present invention, the two variables here refer to the continuous variable and the tacrolimus dosage per unit time.

[0046] Specifically, dichotomous variables included: end-stage liver disease, acute liver failure, disease history, surgical history, treatment history, history of drug allergy, and genetic disease;

[0047] Multi-categorical variables included: drug metabolism ability, metabolic enzyme genotype, other immunosuppressive drug concentration tests, and tacrolimus blood concentration;

[0048] Continuous variables included: blood routine, coagulation function, renal function, electrolytes, liver function, C-reactive protein, immunoglobulin levels, liver tissue stiffness, immune cell analysis, and cytokine levels.

[0049] S13. performing enhancement processing on image data in the clinical data of the same liver transplant recipient to obtain image samples, wherein the enhancement processing includes scaling and normalization;

[0050] S14. Combine the table sample and the image sample into one sample.

[0051] S2. Construct a multimodal linear regression model, which includes a feature extraction module, a feature dimension reduction module, an attention mechanism module, a multimodal fusion module, and a prediction module.

[0052] S3. Pass the samples in the training set through the feature extraction module to obtain the original feature sequence.

[0053] Specifically, step S3 includes:

[0054] S31. Use the pre-trained ResNet50 to process the image samples in the sample to obtain the original image feature sequence.

[0055] Specifically, a convolution layer is used to perform preliminary feature extraction on the image samples in the input layer of the pre-trained ResNet50, and then a series of residual blocks are used to further extract features. Each residual block contains multiple convolution layers, batch normalization layers, and ReLU activation functions. The residual connection helps solve the problem of gradient disappearance. Finally, the global average pooling layer is used to reduce the spatial dimension and improve feature stability. The output of the last layer of the pre-trained ResNet50 is used as the original image feature sequence.

[0056] S32. Use the fully connected network layer to process the table samples in the sample to obtain the original table feature sequence.

[0057] Specifically, the fully connected network layer includes a batch normalization layer and a ReLU activation function. The hidden layer of the fully connected network layer can learn the nonlinear characteristics of the data. In addition, the batch normalization layer helps to speed up the training process, while reducing the sensitivity of the initialization weights and helping to prevent overfitting. The ReLU activation function is added after the batch normalization layer. The ReLU activation function can help introduce nonlinearity, so that the model can learn more complex data patterns.

[0058] S33. The original image feature sequence and the original table feature sequence are combined into an original feature sequence.

[0059] S4. Input the original feature sequence into the feature dimension reduction module VAE to obtain a reduced dimension feature sequence.

[0060] Specifically, Figure 3 As shown, the feature dimension reduction module includes an encoder and a decoder; the original feature sequence is input into the feature dimension reduction module to obtain a reduced dimension feature sequence including:

[0061] S41. Inputting the original image feature sequence into the encoder to obtain a first mean and a first standard deviation; generating a first latent variable using a reparameterization method;

[0062] S42. The decoder reconstructs the first latent variable to obtain an image dimension reduction feature sequence;

[0063] S43. Inputting the original table feature sequence into the encoder to obtain a second mean and a second standard deviation; generating a second latent variable using a reparameterization method;

[0064] S44. The decoder reconstructs the second latent variable to obtain a table dimensionality reduction feature sequence;

[0065] S45. Combine the image dimension reduction feature sequence and the table dimension reduction feature sequence into a dimension reduction feature sequence.

[0066] Specifically, VAE first maps high-dimensional input data (original image feature sequence or original table feature sequence) to a parameterized representation of a low-dimensional latent space through an encoder, that is, learning the latent variable distribution of the input data; in this process, the encoder outputs the latent variable mean and the latent variable standard deviation; a reparameterization method is used to add a normally distributed noise with the latent variable standard deviation as the standard deviation around the latent variable mean to generate a latent variable, which is expressed as:

[0067] c=m+n⊙e

[0068] Among them, c represents the first latent variable or the second latent variable, m represents the first mean or the second mean, n represents the first standard deviation or the second standard deviation; e represents the first normal distribution noise with the first standard deviation as the standard deviation, or represents the second normal distribution noise with the second standard deviation as the standard deviation; ⊙ represents element-wise multiplication; the latent variable can be regarded as a compressed representation of the input data, which encodes the main features and structure of the input data.

[0069] The decoder attempts to reconstruct the original feature sequence from this low-dimensional latent variable c. The training goal of VAE is to maximize the evidence lower bound (ELBO):

[0070]

[0071] ELBO measures the weight between the log-likelihood of the reconstructed data and the latent space regularization term, where the first term is the log-likelihood of the data reconstruction expected under the latent variable c, which measures the quality of reconstructing the input data x using the latent variable c. The second term is the Kullback-Leibler divergence between the latent representation distribution q(z|x) and the prior distribution p(z), which measures the difference between the two probability distributions. By maximizing the lower bound of evidence, a balance point is found so that the model can both reconstruct the data well and keep the distribution of the latent representation close to the prior distribution, thereby generating high-quality data representations with strong generalization capabilities.

[0072] S5. Input the reduced-dimensional feature sequence into the attention module to obtain an enhanced feature sequence.

[0073] Specifically, the attention module includes a PLGA module and a PLGCroA module; the PLGA module captures global and local features within the modality; the PLGCroA module combines multimodal data to capture global and local features between modalities. The reduced-dimensional feature sequence is input into the attention module to obtain an enhanced feature sequence, including:

[0074] S51. Inputting the image dimension reduction feature sequence into the PLGA module to obtain an image splicing feature representation;

[0075] S52. Inputting the table dimension reduction feature sequence into the PLGA module to obtain a table splicing feature representation;

[0076] Specifically, Figure 4 As shown, the PLGA module includes a local attention unit and a global attention unit. The image dimension reduction feature sequence is input into the PLGA module to obtain the image splicing feature representation including:

[0077] S511. Input the image dimension reduction feature sequence into four linear layers to obtain key value, query value, global value and local value, which can be expressed as:

[0078] Q=WQ ×X

[0079] K=W K ×X

[0080]

[0081] Where Q represents the query value, W Q represents the query weight matrix, X represents the image dimension reduction feature sequence, K represents the key value, and W K represents the bond weight matrix, V l Represents the local value, V g Represents a global value, represents the local weight matrix, Represents the global weight matrix, where the key value and query value are shared between the local attention unit and the global attention unit.

[0082] S512. Focus on learning spatial information in the local attention unit. First, the key value is transposed and then multiplied by the query value to obtain a local attention map. Then the local attention map is input into the softmax layer to apply the softmax operation to quantify the similarity between each feature and the remaining spatial features to obtain the local feature similarity. Then, the local feature similarity is multiplied by the local value to obtain the local feature, which can be expressed as:

[0083]

[0084] Where d is the feature dimension, Represents local features.

[0085] S513. The global attention module can effectively capture the interdependence between feature channels. First, the query value is transposed and multiplied by the key value to obtain a global attention map. The global attention map is used to quantify the similarity between each individual channel and other channels. Then the global attention map is input into the softmax layer to apply the softmax operation to quantify the similarity between each feature and the remaining spatial features to obtain the global feature similarity. Then, the global feature is obtained by multiplying the local feature similarity with the global value, which can be expressed as:

[0086]

[0087] in, Represents global features.

[0088] S514. The local features and the global features are fused, and the fusion result is skip-joined with the image dimension reduction feature sequence to obtain the image splicing feature representation, which can be expressed as:

[0089]

[0090] in, In other words, the image splicing feature representation is represented. Similarly, the table dimension reduction feature sequence is input into the PLGA module, and the table splicing feature representation can be obtained through the above operation.

[0091] S53. Input the image splicing feature representation and the table splicing feature representation into the PLGCrOA module to obtain the image enhancement feature representation and the table enhancement feature representation.

[0092] Specifically, Figure 5 As shown, the PLGRoA module also includes a local attention unit and a global attention unit, and step S53 includes:

[0093] S531. Input the table concatenation feature representation into 4 linear layers to obtain the key value K t 、Query value Q t , global value V gt and the local value V lt ; Input the image splicing feature representation into the other 4 linear layers to obtain the key value K i 、Query value Q i , global value V gi and the local value V li ;

[0094]

[0095]

[0096] Among them, W i Q , W i K , W t Q , are all weight matrices.

[0097] S532. Key value K t Transpose and multiply by the query value Q i Get the first local attention map, pass the first local attention map through the softmax layer to get the first local similarity, and compare the first local similarity with the local value V lt Multiply to get the first cross local feature

[0098]

[0099] S533. Through the fully connected layer, the query value Q i Projection is performed to align with the spatial dimension of the table concatenated feature representation to obtain the aligned query value Q proi , for the alignment query value Q proi Transpose and multiply by the key value Kt Get the first global attention map, pass the first global attention map through the softmax layer to get the first global similarity, and compare the first global similarity with the global value V gi Multiply to get the first cross global feature

[0100]

[0101] S534. Fuse the first cross local feature and the first cross global feature, and perform skip splicing on the fusion result and the image splicing feature representation to obtain the image enhancement feature representation

[0102]

[0103] S535. Key value K i Transpose and multiply by the query value Q t Get the second local attention map, pass the second local attention map through the softmax layer to get the second local similarity, and compare the second local similarity with the local value V li Multiply to obtain the second cross local feature;

[0104] S536. Through the fully connected layer, the query value Q t Projection is performed to obtain the alignment query value Q prot , for the alignment query value Q prot Transpose and multiply by the key value K i Get the second global attention map, pass the second global attention map through the softmax layer to get the second global similarity, and compare the second global similarity with the global value V gt Multiply to get the second cross global feature;

[0105] S537. Fuse the second cross local feature and the second cross global feature, and concatenate the fusion result with the table concatenation feature representation to obtain the table enhanced feature representation.

[0106] PLGCroA combines the features of the two modalities to create a unified representation, resulting in a more comprehensive and accurate understanding of the data; in addition, PLGCroA helps extract features based on the context provided by the features of one modality with the guidance of the features of another modality, which will produce a richer and more unique feature representation.

[0107] S6. Input the enhanced feature sequence into the multimodal feature fusion module to obtain a fused feature sequence.

[0108] Specifically, the multimodal fusion module is mainly implemented by a gated attention mechanism with a bilinear product. The bilinear product is usually used to calculate the interaction between two embedding vectors, which can help the model understand the relationship between different modal features. It mainly includes:

[0109] S61. Set the first weight and the second weight through a gating mechanism; specifically, for the enhanced feature representation of the two modalities, use a gating mechanism to dynamically adjust the weights in the fusion process, which is achieved through a learnable parameterized function. Here, a fully connected layer with a sigmoid activation function is used to output a 0-1 weight, indicating the importance of the modality, and the corresponding weights are set to g1 and g2;

[0110] S62. Multiply the image enhancement feature representation and the first weight to obtain the image feature embedding Multiply the table enhanced feature representation by the second weight to get the table feature embedding It can be expressed as:

[0111]

[0112] S63. The interaction between the two modalities of image feature embedding and table feature embedding is calculated by bilinear product to obtain the fused feature sequence, which is expressed as:

[0113]

[0114] Where W ij It is a learnable weight matrix used to transform and fuse the information of the two modalities, and finally calculate the output fusion feature sequence

[0115] S7. The output features are passed through the prediction module to obtain the prediction results; the loss is calculated based on the prediction results and the initial tacrolimus dosage, and the model parameters are optimized by back-propagation training until the model parameters converge.

[0116] Specifically, the prediction module is composed of MLP, which consists of an input layer, multiple hidden layers and an output layer. Each layer is composed of multiple neural sources, and there are weighted connections between neurons. The data passes from the input layer through the hidden layer, and each layer uses weights and biases for linear combination, and then passes through a nonlinear activation function to the output layer. The hidden layer usually uses a nonlinear activation function sigmoid to introduce nonlinear characteristics and enhance the expressiveness of the model. The mean square error loss function is used to calculate the difference between the predicted value and the true value, and the gradient of the loss function with respect to the network parameters is calculated through the back-propagation algorithm, and then the Adam optimizer is used to update the weights and biases.

[0117] Penalize large errors: When the prediction error is large, the value of the loss function will increase rapidly, causing the model to pay more attention to samples with large prediction errors during the optimization process, thereby better fitting extreme cases in the data.

[0118] Minimization strategy: Minimizing the mean square error means that the predicted value is as close to the actual value as possible, which makes it easier for the optimization algorithm to update the parameters according to the loss function and gradually approach the global or local minimum point.

[0119] The Adam optimizer is an extension of the gradient descent optimization algorithm, combining the advantages of the adaptive gradient algorithm and the root mean square propagation algorithm.

[0120] The parameters of the neural network model are optimized by adaptively adjusting the learning rate. During the training process, the learning rate is adjusted according to the size and stability of the gradient, thereby improving the convergence speed and stability of the model.

[0121] S8. Obtain the clinical data to be processed and input it into the trained multimodal information linear regression model to obtain the prediction results of the initial tacrolimus dosage.

[0122] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0123] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for predicting drug dosage for liver transplantation based on multimodality, characterized in that: The following steps are involved: S1. collecting clinical data of liver transplant recipients, wherein the clinical data includes table data and image data; processing all clinical data to obtain a sample set, and dividing the sample set into a training set and a test set; each sample in the sample set includes a table sample and an image sample; S2. Construct a multimodal information linear regression model, which includes a feature extraction module, a feature dimension reduction module, an attention mechanism module, a multimodal fusion module, and a prediction module; S3. The samples in the training set are extracted by a feature extraction module to obtain an original feature sequence, wherein the original feature sequence includes an original image feature sequence and an original table feature sequence; S4. Inputting the original feature sequence into the feature dimension reduction module to obtain a reduced dimension feature sequence, wherein the reduced dimension feature sequence includes an image reduced dimension feature sequence and a table reduced dimension feature sequence; S5. Inputting the dimension reduction feature sequence into the attention mechanism module to obtain an enhanced feature sequence, wherein the enhanced feature sequence includes an image enhanced feature representation and a table enhanced feature representation; S6. inputting the enhanced feature sequence into the multimodal fusion module, and outputting a fused feature sequence; S7. The fused feature sequence is passed through the prediction module to obtain the prediction result, and the loss is calculated according to the prediction result and the model parameters are optimized through back propagation training until the model parameters converge; S8. Obtain the clinical data to be processed and input it into the trained multimodal information linear regression model to obtain the drug dosage prediction result.

2. The method for predicting drug dosage for liver transplantation based on multimodality according to claim 1, characterized in that: Tabular data include case data, routine laboratory test data, tissue elasticity data, pharmacokinetic data, and important immune checkpoint data, including: Case data included primary disease, past history, and family history of liver transplantation; primary disease included end-stage liver disease and acute liver failure, past history included disease history, surgical history, treatment history, and drug allergy history, and family history included hereditary diseases and drug metabolism ability; Routine laboratory test data include blood routine, coagulation function, renal function, electrolytes, liver function, C-reactive protein, and immunoglobulin levels, a total of 7 test data; Tissue elasticity data included liver tissue stiffness; Pharmacokinetic data included tacrolimus plasma concentrations, metabolizing enzyme genotypes; Important immune checkpoint data include other immunosuppressive drug concentration testing, immune cell analysis, and cytokine levels; The image data includes imaging result data, and the imaging result data includes liver ultrasound, liver CT, and liver MRI.

3. The method for predicting drug dosage for liver transplantation based on multimodality according to claim 1, characterized in that: Step S1: The process of processing the clinical data of each liver transplant recipient includes: S11. Preprocessing the tabular data in the clinical data of the liver transplant recipient to obtain preprocessed data; the preprocessing includes missing value and abnormal value processing, normalization and standardization processing; S12. Use univariate analysis to screen out multiple significant variables from the preprocessed data, and combine all significant variables into a table sample; S13. performing enhancement processing on image data in the clinical data of the same liver transplant recipient to obtain image samples, wherein the enhancement processing includes scaling and normalization; S14. Combine the table sample and the image sample into one sample.

4. The method for predicting drug dosage for liver transplantation based on multimodality according to claim 3, characterized in that: Step S12 uses univariate analysis to screen out multiple significant variables from the preprocessed data, including: For each binary variable in the preprocessed data, the chi-square test method was used to determine whether its relationship with the tacrolimus dosage per unit time was significant. If so, the binary variable was considered a significant variable. For each multi-classification variable in the pre-processed data, the variance analysis method was used to determine whether its relationship with the tacrolimus dosage per unit time was significant. If so, the multi-classification variable was considered a significant variable. For each continuous variable in the preprocessing data, the Spearman rank correlation coefficient analysis method was used to determine whether its relationship with the tacrolimus dosage per unit time was significant. If so, the continuous variable was regarded as a significant variable.

5. The method for predicting drug dosage for liver transplantation based on multimodality according to claim 1, characterized in that: The samples in the training set are passed through the feature extraction module to obtain the original feature sequence, including: S31. Use the pre-trained ResNet50 to process the image samples in the sample to obtain the original image feature sequence; S32. Use the fully connected network layer to process the table samples in the sample to obtain the original table feature sequence.

6. The method for predicting drug dosage for liver transplantation based on multimodality according to claim 1, characterized in that: The feature dimension reduction module includes an encoder and a decoder; the original feature sequence is input into the feature dimension reduction module to obtain a reduced dimension feature sequence including: S41. Inputting the original image feature sequence into the encoder to obtain a first mean and a first standard deviation, and generating a first latent variable using a reparameterization method; S42. The decoder reconstructs the first latent variable to obtain an image dimension reduction feature sequence; S43. Inputting the original table feature sequence into the encoder to obtain a second mean and a second standard deviation, and generating a second latent variable using a reparameterization method; S44. The decoder reconstructs the second latent variable to obtain a table dimensionality reduction feature sequence; S45. Combine the image dimension reduction feature sequence and the table dimension reduction feature sequence into a dimension reduction feature sequence.

7. The method for predicting drug dosage for liver transplantation based on multimodality according to claim 1, characterized in that: The attention module includes a PLGA module and a PLGCroA module; the reduced-dimensional feature sequence is input into the attention module to obtain an enhanced feature sequence, including: S51. Inputting the image dimension reduction feature sequence into the PLGA module to obtain an image splicing feature representation; S52. Inputting the table dimension reduction feature sequence into the PLGA module to obtain a table splicing feature representation; S53. Input the image splicing feature representation and the table splicing feature representation into the PLGCrOA module to obtain the image enhancement feature representation and the table enhancement feature representation.

8. The method for predicting drug dosage for liver transplantation based on multimodality according to claim 7, characterized in that: Input the image dimension reduction feature sequence into the PLGA module to obtain the image splicing feature representation including: S511. Input the image dimension reduction feature sequence into four linear layers to obtain key value, query value, global value and local value; S512. Transpose the key value and multiply it by the query value to obtain a local attention map, input the local attention map into the softmax layer to obtain local feature similarity, and multiply the local feature similarity by the local value to obtain the local feature; S513. Transpose the query value and multiply it by the key value to obtain a global attention map, input the global attention map into the softmax layer to obtain the global feature similarity, and multiply the local feature similarity by the global value to obtain the global feature; S514. Fuse the local features and the global features, and concatenate the fusion result with the image dimension reduction feature sequence to obtain an image concatenation feature representation.

9. The method for predicting drug dosage for liver transplantation based on multimodality according to claim 7, characterized in that: Step S53 specifically includes: S531. Calculate the key value K corresponding to the image splicing feature representation through the linear layer i 、Query value Q i , global value V gi and the local value V li ; The corresponding key value K is represented by the concatenated feature representation of the table through the linear layer calculation t 、Query value Q t , global value V gt and the local value V lt ; S532. Key value K t Transpose and multiply by the query value Q i Get the first local attention map, pass the first local attention map through the softmax layer to get the first local similarity, and compare the first local similarity with the local value V lt Multiply to get the first cross local feature; S533. Through the fully connected layer, the query value Q i Projection is performed to obtain the alignment query value Q proi , for the alignment query value Q proi Transpose and multiply by the key value K t Get the first global attention map, pass the first global attention map through the softmax layer to get the first global similarity, and compare the first global similarity with the global value V gi Multiply to get the first cross global feature; S534. Fusing the first cross local feature and the first cross global feature, and splicing the fusion result with the image splicing feature representation to obtain an image enhancement feature representation; S535. Key value K i Transpose and multiply by the query value Q t Get the second local attention map, pass the second local attention map through the softmax layer to get the second local similarity, and compare the second local similarity with the local value V li Multiply to obtain the second cross local feature; S536. Through the fully connected layer, the query value Q t Projection is performed to obtain the alignment query value Q prot , for the alignment query value Q prot Transpose and multiply by the key value K i Get the second global attention map, pass the second global attention map through the softmax layer to get the second global similarity, and compare the second global similarity with the global value V gt Multiply to get the second cross global feature; S537. Fuse the second cross local feature and the second cross global feature, and concatenate the fusion result with the table concatenation feature representation to obtain the table enhanced feature representation.

10. The method for predicting drug dosage for liver transplantation based on multimodality according to claim 1, characterized in that: The multimodal fusion module is implemented by a gated attention mechanism based on bilinear products, including: S61. Setting the first weight and the second weight through a gating mechanism; S62. Multiplying the image enhancement feature representation and the first weight to obtain an image feature embedding; multiplying the table enhancement feature representation and the second weight to obtain a table feature embedding; S63. Calculate the image feature embedding and the table feature embedding through the bilinear product to obtain a fused feature sequence.

Images