Method and system for evaluating prognosis of kidney transplantation graft based on multi-modal deep learning

Abstract

The application relates to the technical field of intelligent medical treatment, and provides a kidney transplantation graft prognosis evaluation method and system based on multi-modal deep learning, wherein the method comprises the following steps: pre-processing and feature extraction are performed on text data and image data; text vector features and image features are fused to obtain fused training features; the fused training features are input into a deep neural network model for model training; donor medical record data is input into an optimized training model, and the last layer features output by the optimized training model are saved into a database; the receiver medical record data is input into the optimized training model, the last layer features of the receiver are extracted, the last layer features of the receiver are matched with all the last layer features of the donors in the database in terms of similarity, and several donor medical records with the highest scores are screened out for transplantation selection. The application combines the existing transplantation patient medical history and prognosis, and predicts and evaluates a kidney transplantation scheme with good postoperative matching and small rejection reaction through a deep neural network.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical technology, and in particular to a method and system for prognostic assessment of kidney transplant grafts based on multimodal deep learning. Background Technology

[0002] Kidney transplantation is a crucial treatment for end-stage renal disease, thus requiring a variety of complex methods to assess its feasibility. These assessments include patient general information, kidney pathology type, blood typing, histological typing, population reaction antibody testing, and lymphocyte virus testing. However, while these existing protocols can assess suitability for matching, they fall short in assessing post-transplant rejection.

[0003] Deep learning, a branch of machine learning, is a model based on artificial neural networks that can be used to process and analyze large-scale, complex data. In recent years, with the continuous improvement of computing power and data volume, deep learning has been widely applied and developed in various fields, especially in the medical field, where its applications are constantly expanding and deepening.

[0004] Despite significant progress in kidney transplantation technology, the challenges faced by physicians during the evaluation process cannot be ignored. Physicians in this field need to leverage complex multidisciplinary knowledge, including immunology, surgery, and cardiovascular medicine, to make comprehensive judgments and decisions. In selecting immunosuppressive drugs, physicians must balance efficacy with side effects. Assessing the suitability of donor kidneys is equally challenging, requiring decisions made with limited donor resources. Therefore, seeking more scientific and precise evaluation methods to improve treatment outcomes and quality of life after kidney transplantation has become a direction that requires continuous exploration in the field of kidney transplantation evaluation. Summary of the Invention

[0005] The purpose of this invention is to solve at least one technical problem in the background art and to provide a method and system for prognostic assessment of kidney transplant grafts based on multimodal deep learning.

[0006] To achieve the above objectives, this invention provides a method for prognostic assessment of kidney transplant grafts based on multimodal deep learning, comprising:

[0007] Collect kidney transplant information data, which includes text data and image data;

[0008] Preprocessing and feature extraction are performed on text data and image data respectively to obtain text vector features and image features;

[0009] Text vector features and image features are fused to obtain fused training features for model training;

[0010] The fused training features are input into the deep neural network model for model training. The output of the deep neural network model is compared with the successfully paired donor and recipient labels of the same category. Then, the difference between the output of the deep neural network model and the donor and recipient labels is reduced by the backpropagation algorithm to obtain the optimized training model.

[0011] The donor's medical record data is input into the optimized training model, and the last layer of donor features output by the optimized training model are saved to the database.

[0012] The recipient's medical record data is input into the optimized training model. The last layer features of the recipient output by the optimized training model are extracted. The last layer features of the recipient are matched with the last layer features of all donors in the database. The donor medical records with the highest scores are selected for transplantation.

[0013] According to one aspect of the present invention, the text data is preprocessed and feature extracted to obtain text vector features, including:

[0014] First, the text data is segmented into words, then stop words are removed, and then special characters are removed;

[0015] Create a vocabulary of all words in the text data, assigning a unique integer index to each word, and then use the word2vec model to vectorize the vocabulary to obtain text vector features.

[0016] According to one aspect of the present invention, preprocessing and feature extraction are performed on the image data to obtain image features, including:

[0017] The image data is cleaned to remove noise, artifacts, and incomplete images.

[0018] Multiple images of the same scene are normalized and registered in brightness and then merged into a single image.

[0019] The merged image is randomly flipped, rotated, scaled, translated, and Gaussian noise is added.

[0020] The image data is divided into blocks, and the image features are extracted by CNN encoding of the blocks.

[0021] According to one aspect of the present invention, the word2vec model is used to vectorize words to obtain text vector features, which includes: vectorizing words using the CBOW model to obtain text vector features, comprising:

[0022] To construct a vocabulary: First, segment the text data into words, remove noisy words, and then construct a vocabulary of all unique words.

[0023] Create a context window, select a target word from the vocabulary and a fixed-size context window, where the words in the window are the context related to the target word;

[0024] Input representation: Map the context words in the window to word vectors, then average these word vectors to obtain the average vector of the context;

[0025] Prediction target: Using the average vector of the context as input, a neural network model is used to predict the target word;

[0026] Training the model: By using the backpropagation algorithm, the parameters of the neural network model are adjusted so that it can accurately predict the target word;

[0027] Obtaining text vector features: After the model is trained, each feature word is encoded and vectorized to obtain text vector features.

[0028] According to one aspect of the present invention, the step of segmenting image data and extracting image features by performing CNN encoding on the segmented image data includes:

[0029] Divide the image into 32x32 image blocks;

[0030] Image patches are input into the resNet-50 convolutional neural network model for encoding to generate image features.

[0031] According to one aspect of the present invention, the fusion of text vector features and image features to obtain fused training features for model training includes:

[0032] Text vector features and image features are fused through a cross-attention mechanism module;

[0033] The classification is achieved by outputting the fused training features for model training through the fully connected (fc) layer in the cross-attention mechanism module.

[0034] According to one aspect of the present invention, the method of inputting fused training features into a deep neural network model for model training employs a two-stage training approach:

[0035] The first stage uses the ArcFace Loss function to classify donors and recipients into one class to learn the differences between classes.

[0036] The second stage involves fine-tuning the deep neural network model to account for differences in the degree of matching between samples within the same class.

[0037] To achieve the above objectives, the present invention also provides a kidney transplant graft prognostic assessment system based on multimodal deep learning, comprising:

[0038] The data acquisition module collects kidney transplant information data, which includes text data and image data.

[0039] The feature extraction module preprocesses and extracts features from text data and image data respectively, to obtain text vector features and image features;

[0040] The feature fusion module fuses text vector features and image features to obtain fused training features for model training.

[0041] The model training module inputs the fused training features into the deep neural network model for model training. It compares the output of the deep neural network model with the successfully paired donor and recipient labels of the same category. Then, it uses the backpropagation algorithm to reduce the difference between the output of the deep neural network model and the donor and recipient labels, thus obtaining an optimized training model.

[0042] The donor feature module inputs donor medical record data into the optimization training model and saves the last layer of donor features output by the optimization training model to the database.

[0043] The recipient matching module inputs recipient medical record data into the optimized training model, extracts the last layer features of the recipient output by the optimized training model, performs similarity matching between the last layer features of the recipient and the last layer features of all donors in the database, and selects the few donor medical records with the highest scores for transplantation.

[0044] To achieve the above objectives, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the multimodal deep learning-based prognostic assessment method for kidney transplant grafts as described above.

[0045] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal deep learning-based prognostic assessment method for kidney transplant grafts as described above.

[0046] According to the present invention, the present invention combines the existing transplant patient's medical history and prognosis, and uses a deep neural network to predict and evaluate kidney transplantation options with good postoperative matching and low rejection rate.

[0047] According to the present invention, deep learning multimodal technology is used for the fusion analysis and processing of various data, such as renal imaging data, clinical data, and physiological data, to improve the efficiency and accuracy of renal disease diagnosis. Simultaneously, deep learning multimodal technology can also be used for the evaluation and prediction of kidney transplant organs. By fusing and analyzing various data such as patient clinical data, imaging data, and immunological data, the quality and adaptability of transplant organs can be predicted, reducing postoperative rejection reactions and improving the success rate of surgery and the patient's quality of life.

[0048] According to the present invention, the quality and suitability of the transplanted organ can be analyzed in depth based on the patient's clinical and imaging data, as well as the recipient patient's kidney transplant prognosis. This method allows for a more accurate assessment of the risks and outcomes of kidney transplantation, thus providing strong decision support for physicians. This method not only improves the accuracy of the assessment but also better predicts the prognosis of the transplanted organ, thereby providing physicians with more effective treatment options. Attached Figure Description

[0049] Figure 1 The flowchart schematically illustrates a method for prognostic assessment of kidney transplant grafts based on multimodal deep learning according to one embodiment of the present invention. Detailed Implementation

[0050] The invention will now be discussed with reference to exemplary embodiments. It should be understood that the described embodiments are merely intended to enable those skilled in the art to better understand and thus implement the invention, and are not intended to imply any limitation on the scope of the invention.

[0051] As used herein, the term "comprising" and its variations are to be interpreted as open-ended terms meaning "including but not limited to". The term "based on" is to be interpreted as "at least partially based on". The terms "one embodiment" and "an embodiment" are to be interpreted as "at least one embodiment".

[0052] Figure 1 This diagram illustrates a flowchart of a multimodal deep learning-based prognostic assessment method for kidney transplant grafts according to one embodiment of the present invention. Figure 1 As shown, in this embodiment, the prognostic assessment method for kidney transplant grafts based on multimodal deep learning includes:

[0053] a. Collect kidney transplant information data, which includes text data and image data;

[0054] b. Perform preprocessing and feature extraction on the text data and image data respectively to obtain text vector features and image features;

[0055] c. Fuse text vector features and image features to obtain fused training features for model training;

[0056] d. Input the fused training features into the deep neural network model for model training, compare the output of the deep neural network model with the successfully paired donor and recipient labels of the same category, and then reduce the difference between the output of the deep neural network model and the donor and recipient labels through the backpropagation algorithm to obtain the optimized training model;

[0057] e. Input the donor's medical record data into the optimized training model, and save the last layer of donor features output by the optimized training model to the database;

[0058] f. Input the recipient's medical record data into the optimized training model, extract the last layer features of the recipient output by the optimized training model, perform similarity matching between the last layer features of the recipient and all the last layer features of the donors in the database, and select the few donor medical records with the highest scores for transplantation.

[0059] According to one embodiment of the present invention, in step a above, the information data collected on kidney transplantation includes the complete medical history of the kidney transplant patient and the corresponding information data on postoperative recovery. The complete medical history includes basic information such as the patient's height, weight, age, and medical history, as well as some laboratory tests such as blood routine, liver and kidney function, and imaging examinations such as CT and MRI. The corresponding postoperative recovery status is scored by the doctor on each postoperative condition, with a value range of 0 to 100.

[0060] Furthermore, according to one embodiment of the present invention, in step b above, the text data is preprocessed and features are extracted to obtain text vector features, including:

[0061] First, the text data is segmented into words, then stop words are removed, and then special characters are removed;

[0062] A vocabulary of all words in the text data is created, with each word assigned a unique integer index. The word2vec model is then used to vectorize the vocabulary, resulting in text vector features. This setup effectively removes a significant amount of noise and redundant information from the medical text data, significantly improving the accuracy of the text vector features used to train the model. This ensures the accuracy of the model's output and ultimately enhances the quality and success rate of transplant surgeries.

[0063] Furthermore, according to one embodiment of the present invention, in step b above, the image data is preprocessed and feature extracted to obtain image features, including:

[0064] The image data is cleaned to remove noise, artifacts, and incomplete images.

[0065] Multiple images of the same scene are normalized and registered in brightness and then merged into a single image.

[0066] The merged image is randomly flipped, rotated, scaled, translated, and Gaussian noise is added.

[0067] Image data is segmented into blocks, and CNN encoding is used to extract image features from these blocks. This approach expands the data sample set, improves the model's generalization ability, ensures the accuracy of the trained model's output, and effectively improves the quality and success rate of transplantation surgery.

[0068] Furthermore, in this embodiment, the word2vec model is selected to vectorize words, resulting in text vector features: The CBOW model is used to vectorize words, resulting in text vector features, including:

[0069] To construct a vocabulary: First, segment the text data into words, remove noisy words, and then construct a vocabulary of all unique words.

[0070] Create a context window, select a target word from the vocabulary and a fixed-size context window, where the words in the window are the context related to the target word;

[0071] Input representation: Map the context words in the window to word vectors, then average these word vectors to obtain the average vector of the context;

[0072] Prediction target: Using the average vector of the context as input, a neural network model is used to predict the target word;

[0073] Training the model: By using the backpropagation algorithm, the parameters of the neural network model are adjusted so that it can accurately predict the target word;

[0074] Obtaining text vector features: After model training, each feature word is encoded and vectorized to obtain text vector features. This setup allows for the capture of semantic relationships between words by mapping them to positions in a high-dimensional vector space, effectively improving the accuracy of predicting target words and thus enhancing the accuracy of text vector features.

[0075] Furthermore, in this embodiment, the image data is segmented into blocks, and the segmented image data is encoded using CNN to extract image features, including:

[0076] Divide the image into 32x32 image blocks;

[0077] Image patches are input into the resNet-50 convolutional neural network model for encoding to generate image features.

[0078] In this embodiment, convolutional neural networks can efficiently capture local features in images. Through convolution operations, basic elements such as edges, textures, and shapes can be detected, thereby extracting important information from the image. In this embodiment, resNet-50 is selected as the base network for feature extraction. The base network is pre-trained on ImageNet, and the pre-trained network has a good understanding of both high-frequency and low-frequency textures in the image, which is beneficial for the convergence of subsequent downstream tasks when extracting image features.

[0079] Furthermore, according to one embodiment of the present invention, in step c above, the text vector features and image features are fused to obtain fused training features for model training, including:

[0080] Text vector features and image features are fused through a cross-attention mechanism module;

[0081] The classification is achieved by outputting the fused training features for model training through the fully connected (fc) layer in the cross-attention mechanism module.

[0082] In this implementation, the input text vector features and image features are fused through a cross-attention module. The text vector features and image features are treated as tokens and input into the cross-attention module. After fusion by the cross-attention module, the output is classified through one of the fully connected (FC) layers.

[0083] Furthermore, according to one embodiment of the present invention, in step d above, the fused training features are input into the deep neural network model for model training using a two-stage training method:

[0084] The first stage uses the ArcFace Loss function to classify donors and recipients into one class to learn the differences between classes.

[0085] The second stage involves fine-tuning the deep neural network model to account for differences in the degree of matching between samples within the same class.

[0086] In this embodiment, during model training, the output of the model needs to be compared with the labels, and backpropagation is used to reduce the difference with the labels to obtain a model with better prediction accuracy. Here, the data labels classify donors and successfully matched recipients into the same category and mark the matching degree (0-1). This invention adopts a two-stage training method. The first stage first classifies donors and recipients into one class to learn the difference between classes. The second stage fine-tunes the model training to learn the difference between samples within the same class based on the different matching degrees.

[0087] The first stage is a classification task, and the loss function used in this invention is ArcFace Loss.

[0088]

[0089] In the formula, L is the error; N is the number of samples in the training data; s is the scaling parameter, used to control the distribution of the feature space; θ yi Let the feature vector of the i-th sample be its true class y. i The angle between the corresponding category feature vectors; m is the angle boundary parameter, introduced to enhance the discriminative performance of angle information; e is the base of the natural logarithm, approximately equal to 2.7; j represents the angle between the target category y and the target category y. i Different from all other categories; θj represents the angle between the feature vector of this sample and the feature vectors of all other categories j.

[0090] In this formula, cos(θ) yi +m) represents category y i The cosine similarity between the feature vectors of the sample and the feature vectors of the class is used to measure the distribution of the feature vectors along the angle. The numerator in the formula corresponds to the true class of the sample, and the denominator corresponds to all other classes. The goal of the entire formula is to maximize the similarity of the true class while minimizing the similarity of the other classes.

[0091] The optimizer uses stochastic gradient descent with an initial learning rate of 0.01. After 60,000 iterations, the learning rate is reduced. The first training phase converges and the loss stabilizes after 100,000 iterations.

[0092] The second stage trains the differences in intra-class kidney matching, distinguishing different matching degrees based on the rejection response of recipient patients after kidney transplantation. No rejection response is labeled as 1. The matching degree is then scored based on the magnitude of rejection response and the presence of other side effects and complications. The second stage first loads the model parameters from the first stage, and then uses CrossEntropy Loss to regress the current task.

[0093] Furthermore, according to one embodiment of the present invention, in steps e and f above, after the model training is completed, the donor medical record data is first processed through the model, and the output last layer of features is saved to the database. In this invention, the last layer is a 512-dimensional feature. Then, when a recipient patient needs a kidney transplant, the recipient patient's medical record, laboratory tests, imaging examinations, and other data are input into the model to extract features. Finally, the 512-dimensional features of the recipient patient are matched with the donor features in the database to determine similarity, and the highest-scoring medical records are selected for doctors and patients to choose from.

[0094] According to the above-described scheme of the present invention, the present invention combines the existing transplant patient's medical history and prognosis, and uses a deep neural network to predict and evaluate kidney transplantation schemes with good postoperative matching and low rejection rate.

[0095] According to the above-described scheme of the present invention, the present invention utilizes deep learning multimodal technology for the fusion analysis and processing of various data such as renal imaging data, clinical data, and physiological data, thereby improving the efficiency and accuracy of renal disease diagnosis. Simultaneously, deep learning multimodal technology can also be used for the evaluation and prediction of kidney transplant organs. By fusing and analyzing various data such as the patient's clinical data, imaging data, and immunological data, the quality and adaptability of transplant organs can be predicted, reducing postoperative rejection reactions and improving the success rate of surgery and the patient's quality of life.

[0096] According to the above-described scheme of the present invention, the quality and suitability of the transplanted organ can be analyzed in depth based on the patient's clinical and imaging data, as well as the recipient patient's kidney transplant prognosis. This method enables a more accurate assessment of the risks and outcomes of kidney transplantation, thereby providing strong decision support for physicians. This method not only improves the accuracy of the assessment but also better predicts the prognosis of the transplanted organ, thus providing physicians with more effective treatment options.

[0097] Furthermore, to achieve the above objectives, the present invention also provides a kidney transplant graft prognostic assessment system based on multimodal deep learning, comprising:

[0098] The data acquisition module collects kidney transplant information data, which includes text data and image data.

[0099] The feature extraction module preprocesses and extracts features from text data and image data respectively, to obtain text vector features and image features;

[0100] The feature fusion module fuses text vector features and image features to obtain fused training features for model training.

[0101] The model training module inputs the fused training features into the deep neural network model for model training. It compares the output of the deep neural network model with the successfully paired donor and recipient labels of the same category. Then, it uses the backpropagation algorithm to reduce the difference between the output of the deep neural network model and the donor and recipient labels, thus obtaining an optimized training model.

[0102] The donor feature module inputs donor medical record data into the optimization training model and saves the last layer of donor features output by the optimization training model to the database.

[0103] The recipient matching module inputs recipient medical record data into the optimized training model, extracts the last layer features of the recipient output by the optimized training model, performs similarity matching between the last layer features of the recipient and the last layer features of all donors in the database, and selects the few donor medical records with the highest scores for transplantation.

[0104] According to one embodiment of the present invention, in the above-mentioned data acquisition module, the acquired kidney transplant information data includes the complete medical history of the kidney transplant patient and the corresponding postoperative recovery information data; the complete medical history includes basic information such as the patient's height, weight, age, and medical history, as well as some laboratory tests such as blood routine, liver and kidney function and imaging examinations such as CT and MRI; the corresponding postoperative recovery status is scored by the doctor for each postoperative condition, with a value range of 0 to 100.

[0105] Furthermore, according to one embodiment of the present invention, in the above-mentioned feature extraction module, preprocessing and feature extraction are performed on the text data to obtain text vector features, including:

[0106] First, the text data is segmented into words, then stop words are removed, and then special characters are removed;

[0107] A vocabulary of all words in the text data is created, with each word assigned a unique integer index. The word2vec model is then used to vectorize the vocabulary, resulting in text vector features. This setup effectively removes a significant amount of noise and redundant information from the medical text data, significantly improving the accuracy of the text vector features used to train the model. This ensures the accuracy of the model's output and ultimately enhances the quality and success rate of transplant surgeries.

[0108] Furthermore, according to one embodiment of the present invention, in the above-described feature extraction module, image data is preprocessed and feature extracted to obtain image features, including:

[0109] The image data is cleaned to remove noise, artifacts, and incomplete images.

[0110] Multiple images of the same scene are normalized and registered in brightness and then merged into a single image.

[0111] The merged image is randomly flipped, rotated, scaled, translated, and Gaussian noise is added.

[0112] Image data is segmented into blocks, and CNN encoding is used to extract image features from these blocks. This approach expands the data sample set, improves the model's generalization ability, ensures the accuracy of the trained model's output, and effectively improves the quality and success rate of transplantation surgery.

[0113] Furthermore, in this embodiment, the word2vec model is selected to vectorize words, resulting in text vector features: The CBOW model is used to vectorize words, resulting in text vector features, including:

[0114] To construct a vocabulary: First, segment the text data into words, remove noisy words, and then construct a vocabulary of all unique words.

[0115] Create a context window, select a target word from the vocabulary and a fixed-size context window, where the words in the window are the context related to the target word;

[0116] Input representation: Map the context words in the window to word vectors, then average these word vectors to obtain the average vector of the context;

[0117] Prediction target: Using the average vector of the context as input, a neural network model is used to predict the target word;

[0118] Training the model: By using the backpropagation algorithm, the parameters of the neural network model are adjusted so that it can accurately predict the target word;

[0119] Obtaining text vector features: After model training, each feature word is encoded and vectorized to obtain text vector features. This setup allows for the capture of semantic relationships between words by mapping them to positions in a high-dimensional vector space, effectively improving the accuracy of predicting target words and thus enhancing the accuracy of text vector features.

[0120] Furthermore, in this embodiment, the image data is segmented into blocks, and the segmented image data is encoded using CNN to extract image features, including:

[0121] Divide the image into 32x32 image blocks;

[0122] Image patches are input into the resNet-50 convolutional neural network model for encoding to generate image features.

[0123] In this embodiment, convolutional neural networks can efficiently capture local features in images. Through convolution operations, basic elements such as edges, textures, and shapes can be detected, thereby extracting important information from the image. In this embodiment, resNet-50 is selected as the base network for feature extraction. The base network is pre-trained on ImageNet, and the pre-trained network has a good understanding of both high-frequency and low-frequency textures in the image, which is beneficial for the convergence of subsequent downstream tasks when extracting image features.

[0124] Furthermore, according to one embodiment of the present invention, in the above-mentioned feature fusion module, text vector features and image features are fused to obtain fused training features for model training, including:

[0125] Text vector features and image features are fused through a cross-attention mechanism module;

[0126] The classification is achieved by outputting the fused training features for model training through the fully connected (fc) layer in the cross-attention mechanism module.

[0127] In this implementation, the input text vector features and image features are fused through a cross-attention module. The text vector features and image features are treated as tokens and input into the cross-attention module. After fusion by the cross-attention module, the output is classified through one of the fully connected (FC) layers.

[0128] Furthermore, according to one embodiment of the present invention, in the above-mentioned model training module, the fused training features are input into the deep neural network model for model training using a two-stage training method:

[0129] The first stage uses the ArcFace Loss function to classify donors and recipients into one class to learn the differences between classes.

[0130] The second stage involves fine-tuning the deep neural network model to account for differences in the degree of matching between samples within the same class.

[0131] In this embodiment, during model training, the output of the model needs to be compared with the labels, and backpropagation is used to reduce the difference with the labels to obtain a model with better prediction accuracy. Here, the data labels classify donors and successfully matched recipients into the same category and mark the matching degree (0-1). This invention adopts a two-stage training method. The first stage first classifies donors and recipients into one class to learn the difference between classes. The second stage fine-tunes the model training to learn the difference between samples within the same class based on the different matching degrees.

[0132] The first stage is a classification task, and the loss function used in this invention is ArcFace Loss.

[0133]

[0134] In the formula, L is the error; N is the number of samples in the training data; s is the scaling parameter, used to control the distribution of the feature space; θ yi Let the feature vector of the i-th sample be its true class y. iThe angle between the corresponding category feature vectors; m is the angle boundary parameter, introduced to enhance the discriminative performance of angle information; e is the base of the natural logarithm, approximately equal to 2.7; j represents the angle between the target category y and the target category y. i Different from all other categories; θ j This represents the angle between the feature vector of this sample and the feature vectors of all other categories j.

[0135] In this formula, cos(θ) yi +m) represents category y i The cosine similarity between the feature vectors of the sample and the feature vectors of the class is used to measure the distribution of the feature vectors along the angle. The numerator in the formula corresponds to the true class of the sample, and the denominator corresponds to all other classes. The goal of the entire formula is to maximize the similarity of the true class while minimizing the similarity of the other classes.

[0136] The optimizer uses stochastic gradient descent with an initial learning rate of 0.01. After 60,000 iterations, the learning rate is reduced. The first training phase converges and the loss stabilizes after 100,000 iterations.

[0137] The second stage trains the differences in intra-class kidney matching, distinguishing different matching degrees based on the rejection response of recipient patients after kidney transplantation. No rejection response is labeled as 1. The matching degree is then scored based on the magnitude of rejection response and the presence of other side effects and complications. The second stage first loads the model parameters from the first stage, and then uses CrossEntropy Loss to regress the current task.

[0138] Furthermore, according to one embodiment of the present invention, in the donor feature module and recipient matching module described above, after the model training is completed, the donor medical record data is first processed through the model, and the output last layer of features is saved to the database. In this invention, the last layer is a 512-dimensional feature. Then, when a recipient patient needs a kidney transplant, the recipient patient's medical record, laboratory tests, imaging examinations, and other data are input into the model to extract features. Finally, the 512-dimensional features of the recipient patient are matched with the donor features in the database to determine similarity, and the highest-scoring medical records are selected for doctors and patients to choose from.

[0139] According to the above-described scheme of the present invention, the present invention combines the existing transplant patient's medical history and prognosis, and uses a deep neural network to predict and evaluate kidney transplantation schemes with good postoperative matching and low rejection rate.

[0140] According to the above-described scheme of the present invention, the present invention utilizes deep learning multimodal technology for the fusion analysis and processing of various data such as renal imaging data, clinical data, and physiological data, thereby improving the efficiency and accuracy of renal disease diagnosis. Simultaneously, deep learning multimodal technology can also be used for the evaluation and prediction of kidney transplant organs. By fusing and analyzing various data such as the patient's clinical data, imaging data, and immunological data, the quality and adaptability of transplant organs can be predicted, reducing postoperative rejection reactions and improving the success rate of surgery and the patient's quality of life.

[0141] According to the above-described scheme of the present invention, the quality and suitability of the transplanted organ can be analyzed in depth based on the patient's clinical and imaging data, as well as the recipient patient's kidney transplant prognosis. This method enables a more accurate assessment of the risks and outcomes of kidney transplantation, thereby providing strong decision support for physicians. This method not only improves the accuracy of the assessment but also better predicts the prognosis of the transplanted organ, thus providing physicians with more effective treatment options.

[0142] Furthermore, to achieve the above objectives, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the above-mentioned prognostic assessment method for kidney transplant grafts based on multimodal deep learning.

[0143] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for prognostic assessment of kidney transplant grafts based on multimodal deep learning.

[0144] Those skilled in the art will recognize that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and equipment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0146] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0147] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0148] In addition, the functional modules in the embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0149] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the energy-saving signal transmission / reception methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0150] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

[0151] It should be understood that the sequence number of each step in the invention and embodiments of the present invention does not absolutely imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

Claims

1. A prognostic assessment method for kidney transplant grafts based on multimodal deep learning, characterized in that, include: Collect kidney transplant information data, which includes text data and image data; Preprocessing and feature extraction are performed on text data and image data respectively to obtain text vector features and image features; Text vector features and image features are fused to obtain fused training features for model training; The fused training features are input into the deep neural network model for model training. The output of the deep neural network model is compared with the donor and recipient labels of the same category that have been successfully paired. Then, the difference between the output of the deep neural network model and the donor and recipient labels is reduced by the backpropagation algorithm to obtain an optimized training model. The donor and recipient labels are used to classify the donor and the successfully paired recipient into the same category and mark the degree of matching in the range of 0 to 1. The donor's medical record data is input into the optimized training model, and the last layer of donor features output by the optimized training model are saved to the database. The recipient's medical record data is input into the optimized training model. The last layer features of the recipient output by the optimized training model are extracted. The last layer features of the recipient are matched with the last layer features of all donors in the database. The highest-scoring donor medical records are selected for transplantation. The method of inputting fused training features into the deep neural network model for model training employs a two-stage training approach: The first stage uses the ArcFace Loss function to classify donors and recipients into one class and learn the differences between classes; The ArcFace Loss function is as follows: ； In the formula, L is the error; N is the number of samples in the training data; s is the scaling parameter, used to control the distribution of the feature space; θ yi Let the feature vector of the i-th sample be its true class y. i The angle between the corresponding category feature vectors; m is the angle boundary parameter, introduced to enhance the discriminative performance of angle information; e is the base of the natural logarithm, equal to 2.7; j represents the angle between the target category y and the target category y. i Different from all other categories; θj represents the angle between the feature vector of this sample and the feature vectors of all other categories j; cos(θ) yi +m) represents category y i The cosine similarity between the feature vectors of the sample and the feature vectors of the class is used to measure the distribution of the feature vectors in terms of angle; the numerator in the formula corresponds to the true class of the sample, and the denominator corresponds to all other classes. The second stage involves fine-tuning the deep neural network model based on varying matching levels, training on the differences between intra-class samples, including: To differentiate the degree of matching in kidney transplant recipients based on their rejection response, a score of 1 is assigned for those without rejection. When rejection occurs, the degree of matching is scored based on the severity of the rejection and the presence of other side effects and complications.

2. The method for prognostic assessment of kidney transplant grafts based on multimodal deep learning according to claim 1, characterized in that, The text data is preprocessed and features are extracted to obtain text vector features, including: First, the text data is segmented into words, then stop words are removed, and then special characters are removed; Create a vocabulary of all words in the text data, assigning a unique integer index to each word, and then use the word2vec model to vectorize the vocabulary to obtain text vector features.

3. The method for prognostic assessment of kidney transplant grafts based on multimodal deep learning according to claim 1, characterized in that, The image data is preprocessed and feature extracted to obtain image features, including: The image data is cleaned to remove noise, artifacts, and incomplete images. Multiple images of the same scene are normalized and registered in brightness and then merged into a single image. The merged image is randomly flipped, rotated, scaled, translated, and Gaussian noise is added. The image data is divided into blocks, and the image features are extracted by CNN encoding of the blocks.

4. The prognostic assessment method for kidney transplant grafts based on multimodal deep learning according to claim 2, characterized in that, The word2vec model is selected to vectorize words, resulting in text vector features: The CBOW model is used to vectorize words, resulting in text vector features, including: To construct a vocabulary: First, segment the text data into words, remove noisy words, and then construct a vocabulary of all unique words. Create a context window, select a target word from the vocabulary and a fixed-size context window, where the words in the window are the context related to the target word; Input representation: Map the context words in the window to word vectors, then average these word vectors to obtain the average vector of the context; Prediction target: Using the average vector of the context as input, a neural network model is used to predict the target word; Training the model: By using the backpropagation algorithm, the parameters of the neural network model are adjusted so that it can accurately predict the target word; Obtaining text vector features: After the model is trained, each feature word is encoded and vectorized to obtain text vector features.

5. The method for prognostic assessment of kidney transplant grafts based on multimodal deep learning according to claim 3, characterized in that, The step of segmenting the image data into blocks and extracting image features by performing CNN encoding on the segmented image data includes: Divide the image into 32x32 image blocks; Image patches are input into the resNet-50 convolutional neural network model for encoding to generate image features.

6. The method for prognostic assessment of kidney transplant grafts based on multimodal deep learning according to claim 1, characterized in that, The process of fusing text vector features and image features to obtain fused training features for model training includes: Text vector features and image features are fused through a cross-attention mechanism module; The classification is achieved by outputting the fused training features for model training through the fully connected (fc) layer in the cross-attention mechanism module.

7. A prognostic assessment system for kidney transplant grafts based on multimodal deep learning, characterized in that, include: The data acquisition module collects kidney transplant information data, which includes text data and image data. The feature extraction module preprocesses and extracts features from text data and image data respectively, to obtain text vector features and image features; The feature fusion module fuses text vector features and image features to obtain fused training features for model training. The model training module inputs the fused training features into the deep neural network model for model training. The output of the deep neural network model is compared with the donor and recipient labels of the same category that have been successfully paired. Then, the difference between the output of the deep neural network model and the donor and recipient labels is reduced by the backpropagation algorithm to obtain an optimized training model. The donor and recipient labels are used to classify the donor and the successfully paired recipient into the same category and mark the degree of matching in the range of 0 to 1. The donor feature module inputs donor medical record data into the optimization training model and saves the last layer of donor features output by the optimization training model to the database. The recipient matching module inputs recipient medical record data into the optimized training model, extracts the last layer features of the recipient output by the optimized training model, performs similarity matching between the last layer features of the recipient and the last layer features of all donors in the database, and selects the few donor medical records with the highest scores for transplantation. The method of inputting fused training features into the deep neural network model for model training employs a two-stage training approach: The first stage uses the ArcFace Loss function to classify donors and recipients into one class and learn the differences between classes; The ArcFace Loss function is as follows: ； In the formula, L is the error; N is the number of samples in the training data; s is the scaling parameter, used to control the distribution of the feature space; θ yi Let the feature vector of the i-th sample be its true class y. i The angle between the corresponding category feature vectors; m is the angle boundary parameter, introduced to enhance the discriminative performance of angle information; e is the base of the natural logarithm, equal to 2.7; j represents the angle between the target category y and the target category y. i Different from all other categories; θj represents the angle between the feature vector of this sample and the feature vectors of all other categories j; cos(θ) yi +m) represents category y i The cosine similarity between the feature vectors of the sample and the feature vectors of the class is used to measure the distribution of the feature vectors in terms of angle; the numerator in the formula corresponds to the true class of the sample, and the denominator corresponds to all other classes. The second stage involves fine-tuning the deep neural network model based on varying matching levels, training on the differences between intra-class samples, including: To differentiate the degree of matching in kidney transplant recipients based on their rejection response, a score of 1 is assigned for those without rejection. When rejection occurs, the degree of matching is scored based on the severity of the rejection and the presence of other side effects and complications.

8. An electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the prognostic assessment method for kidney transplant grafts based on multimodal deep learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the prognostic assessment method for kidney transplant grafts based on multimodal deep learning as described in any one of claims 1 to 6.

Images