Chronic kidney disease progression prediction method and device

Abstract

The invention provides a chronic kidney disease progress prediction method and device, and the method comprises the steps: collecting multi-modal data related to a chronic kidney disease, inputting the multi-modal data into a constructed prediction model for training, and if the multi-modal data is recognized to belong to table data, determining the progress of the chronic kidney disease; performing feature aggregation on the modal data through the table feature extractor, encoding the modal data into a plurality of one-dimensional feature vectors, and integrating the plurality of one-dimensional feature vectors into a table data representative vector; if the modal data is identified to belong to the image data, training the whole modal data graph through the image feature extractor, and integrating to obtain an image representative vector; and splicing and fusing all the table data representative vectors and all the image representative vectors through the multi-modal fusion classifier, and then carrying out classification prediction. The method can predict the progress of chronic kidney disease with extremely high precision.

Description

Technical Field

[0001] The present invention relates to the multimodal field of computational medicine, and in particular to a method, device, storage medium, and computer program product for predicting the progression of chronic kidney disease. Background Art

[0002] Existing chronic kidney disease progression prediction technologies mainly include clinical models based on electronic medical record data, simple models using machine learning algorithms, and extended models using multimodal data.

[0003] Among them: In the prediction methods of clinical models based on electronic medical record data, many models use clinical indicators recorded in electronic medical records to predict the progression of chronic kidney disease. The most common predictors in the prediction models are age, gender, body mass index, diabetes status, systolic blood pressure, serum creatinine, proteinuria measurement, and serum albumin or total protein. This type of technology is mostly transformed from clinical applications. In order to facilitate use in real-world scenarios, it usually follows the two principles of easy data acquisition and wide applicability. However, since this type of method is a summary of clinical experience, it also leads to the use of fewer indicators in the method, making it difficult to discover new knowledge, resulting in poor prediction accuracy for new symptoms.

[0004] In the prediction method of simple models using machine learning algorithms, machine learning algorithms are used to process a wider range and a larger number of variables, which enhances the analysis capabilities of structured data and classification problems and improves the accuracy of disease prediction to a certain extent. However, the limitations of this type of technology mainly come from the model itself, including: it is difficult to use large-scale data, it is difficult for simple structures to capture complex nonlinear relationships in data, manual feature engineering is complex and it is easy to ignore unknown important features, and there is also the defect of poor prediction accuracy for new symptoms.

[0005] In the prediction method of the extended model using multimodal data, the accuracy of disease prediction is improved by mining the influence of other modal data (such as important omics data and image data in the diagnosis and treatment process) on the prediction accuracy. However, the disadvantage of the existing prediction method of the extended model using multimodal data is that the expansion of new clinical markers requires a large number of wet experiments, and the data analysis and utilization of each mode are independent, without paying attention to the complementary information between the modes.

[0006] Therefore, it is necessary to improve the above prediction method. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a method, device, storage medium, and computer program product for predicting the progression of chronic kidney disease.

[0008] Among them: On the one hand, the present invention provides a method for predicting the progression of chronic kidney disease, comprising:

[0009] Collect multimodal data on chronic kidney disease, including clinical data at the individual scale, proteomic data and metabolomic data at the molecular scale, and digital pathological images at the cell tissue scale; wherein the clinical data, proteomic data, and metabolomic data belong to tabular data, and the digital pathological images at the cell tissue scale belong to image data; input the multimodal data into a constructed prediction model for training, and the prediction model includes a tabular feature extractor, an image feature extractor, and a multimodal fusion classifier; wherein: identify the category of each modal data, if it is identified as tabular data, perform feature aggregation on the modal data through the tabular feature extractor and encode it into multiple one-dimensional feature vectors, and integrate the multiple one-dimensional feature vectors into a tabular data representative vector; if it is identified as image data, train the entire modal data through the image feature extractor, and integrate to obtain an image representative vector; and perform classification prediction after splicing and fusing all the tabular data representative vectors and all the image representative vectors through the multimodal fusion classifier.

[0010] In some embodiments of the present invention, each of the table feature extractors includes a mapping coding layer, an auxiliary training layer, and a fully connected feature extraction layer; the features contained in the corresponding modal data are bucketed in turn through the mapping coding layer to unify the value range of the data, and then each feature is encoded into a one-dimensional feature vector; each one-dimensional feature vector is updated and optimized through the auxiliary training layer; and the multiple one-dimensional feature vectors are integrated into the representative vector of the table data through multiple hidden layers of the fully connected feature extraction layer.

[0011] In some embodiments of the present invention, the mapping coding layer uses a hash equation to map the original data type corresponding to the modal data to multiple categories to unify the value range of the data.

[0012] In some embodiments of the present invention, in the process of updating each one-dimensional feature vector through the auxiliary training layer, each one-dimensional feature vector representative vector is extended to contain more information, and gradient backpropagation is performed to train each one-dimensional feature vector so that the information it contains is consistent with the prediction task.

[0013] In some embodiments of the present invention, the image feature extractor adopts the Resnet50 model as the basic architecture, and replaces the last two layers of the aggregation layer and the classification layer of the Resnet50 model with new hidden layers, while the other layers remain unchanged.

[0014] In some embodiments of the present invention, each of the table data representative vectors and each of the image representative vectors are configured to be applied to the predicted classification target respectively; the multimodal fusion classifier is composed of an n-layer fully connected neural network in conjunction with a gating network, wherein the gating network is composed of m fully connected layers, generating a first weight corresponding to each of the table data representative vectors and a second weight corresponding to each of the image representative vectors, and each of the table representative vectors is multiplied by its own first weight, and each of the image representative vectors is multiplied by its own second weight, and then spliced; the spliced ​​features enter the n-layer fully connected neural network for classification prediction; wherein n and m are both at least 2.

[0015] In some embodiments of the present invention, a five-fold cross-validation dataset is used to train the prediction model.

[0016] In some embodiments of the present invention, the importance of each modal data in the multimodal data is analyzed, and the importance of each feature obtained at each stage in the process of inputting the multimodal data into the constructed prediction model for training is analyzed, and the key features are screened out, and the key features are preliminarily identified as predictive markers. In some embodiments of the present invention, the table feature extractor includes a mapping coding layer, an auxiliary training layer, and a fully connected feature extraction layer; wherein: the features contained in each tabular data are sequentially bucketed through the mapping coding layer, the value range of the data is unified, and each feature is encoded into a one-dimensional feature vector; each one-dimensional feature vector is updated and optimized through the auxiliary training layer; and the multiple one-dimensional feature vectors are integrated into the representative vector of the tabular data through the multiple hidden layers of the fully connected feature extraction layer.

[0017] And, in another aspect, the present invention provides a chronic kidney disease progression prediction device, the device comprising:

[0018] A data set acquisition unit, used to collect multimodal data on chronic kidney disease, including clinical data at the individual scale, proteomic data and metabolomic data at the molecular scale, and digital pathological images at the cell tissue scale; wherein the clinical data, proteomic data, and metabolomic data belong to table data, and the digital pathological images at the cell tissue scale belong to image data;

[0019] A model training and prediction unit, used to input the multimodal data into a constructed prediction model for training, the prediction model comprising a table feature extractor, an image feature extractor, and a multimodal fusion classifier; wherein: the model training and prediction unit further comprises:

[0020] An encoding module, for performing feature aggregation on the modal data identified as table data through the table feature extractor and encoding it into a plurality of one-dimensional feature vectors, and integrating the plurality of one-dimensional feature vectors into the representative vector of the table data; and for training the modal data identified as image data through the image feature extractor on the entire image, and integrating to obtain the image representative vector;

[0021] The classification module is used to perform classification prediction after splicing and fusing all the table data representative vectors and all the image representative vectors through the multimodal fusion classifier.

[0022] In some embodiments of the present invention, each of the table feature extractors includes a mapping coding layer, an auxiliary training layer, and a fully connected feature extraction layer; wherein the mapping coding layer is used to sequentially bucket the features contained in the corresponding modal data, unify the value range of the data, and then encode each feature into a one-dimensional feature vector; the auxiliary training layer is connected to the output end of the mapping coding layer to update and optimize each one-dimensional feature vector; the fully connected feature extraction layer is connected to the output end of the auxiliary training layer to integrate the multiple one-dimensional feature vectors into the table data representative vector through multiple hidden layers of the fully connected feature extraction layer.

[0023] In yet another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for predicting the progression of chronic kidney disease.

[0024] In yet another aspect, the present invention provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned chronic kidney disease progression prediction method are implemented.

[0025] It can be seen from the above scheme that the advantages of the present invention are:

[0026] The method for predicting the progression of chronic kidney disease disclosed in the present invention integrates data of multiple scales and modes such as clinical data, proteomic data, metabolomic data and digital pathological images, and predicts the progression of chronic kidney disease through a deep learning model. Specifically, according to the different categories of each modal data, the clinical data, proteomic data and metabolomic data belonging to the tabular data are respectively subjected to feature aggregation and encoding into multiple one-dimensional feature vectors through a tabular feature extractor, and the multiple one-dimensional feature vectors are integrated into the tabular data representative vector of the corresponding modality; the digital pathological image belonging to the image data is trained by the image feature extractor for the whole picture, and the image representative vector is integrated; and finally, the tabular data representative vector and the image representative vector are fused by the multimodal fusion classifier for classification prediction, and the final chronic kidney disease progression prediction result is obtained. This method effectively copes with the distribution noise caused by modal heterogeneity while enriching the feature information by aggregating features and optimizing encoding. The synchronous training of the model architecture and feature encoding greatly improves the model's ability to capture the interaction between different modalities and features in high-dimensional space, and can predict the progression of chronic kidney disease with extremely high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The overall process diagram of the method for predicting the progression of chronic kidney disease provided by one embodiment of the present invention is shown.

[0028] Figure 2(a) shows the overall architecture of the prediction model;

[0029] FIG2( b ) is a schematic diagram showing the specific structure of the table feature extractor;

[0030] FIG2( c ) is a schematic diagram showing three types of table data input into a table feature extractor;

[0031] Figure 3 A detailed schematic diagram of the process of step S2 is shown;

[0032] Figure 4 A schematic diagram showing the structure of a chronic kidney disease progression prediction device provided by an embodiment of the present invention;

[0033] Wherein, the reference numerals are:

[0034] 10: Prediction model;

[0035] 11: table feature extractor;

[0036] 111: mapping coding layer;

[0037] 112: auxiliary training layer;

[0038] 113: fully connected feature extraction layer;

[0039] 12: Image feature extractor;

[0040] 13: Multimodal fusion classifier;

[0041] 131: fully connected neural network;

[0042] 132: gated network;

[0043] 200: Chronic kidney disease progression prediction device;

[0044] 210: data set acquisition unit;

[0045] 220: model training and prediction unit;

[0046] 2201: encoding module;

[0047] 2202: Classification module. DETAILED DESCRIPTION

[0048] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0049] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0050] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0051] Please refer to Figure 1 , as shown in Figure 2(a), Figure 2(b), and Figure 2(c), where Figure 1 FIG2(a) shows an overall schematic diagram of a method for predicting the progression of chronic kidney disease provided by an embodiment of the present invention. FIG2(b) shows a schematic diagram of the overall structure of a prediction model. FIG2(c) shows a schematic diagram of the specific structure of a table feature extractor. FIG2(c) shows a schematic diagram of inputting three types of table data, namely, clinical data, proteomic data, and metabolomic data, into a table feature extractor. Figure 3 A detailed flow chart of step S2 is shown.

[0052] Specifically, see Figure 1 As shown in , the chronic kidney disease progression prediction method specifically includes the following steps:

[0053] Step S1, collecting multimodal data on chronic kidney disease, including clinical data at the individual scale, proteomic data and metabolomic data at the molecular scale, and digital pathological images at the cell tissue scale. The clinical data, proteomic data, and metabolomic data belong to table data, and the digital pathological images at the cell tissue scale belong to image data.

[0054] In this embodiment, multimodal data on chronic kidney disease need to be collected first, mainly including four commonly used modal data, specifically: clinical data of chronic kidney disease patients at the individual scale obtained from electronic medical records, molecular-scale proteomics data and metabolomics data collected from patient urine, and digital pathology images at the cell tissue scale obtained from patient kidney puncture images. The digital pathology image can not only see a piece of tissue, but also see the morphological characteristics of cells in the tissue.

[0055] In addition, in this embodiment, the multimodal data is not limited to the above four types of modal data. In practice, new modal data can be added according to the needs of specific experiments and the development of related disease monitoring technologies.

[0056] Then, before the multimodal data collected above are input into the prediction model for training, these multimodal data need to be further preprocessed. For example, for clinical data, proteomics data and metabolomics data, perform preprocessing operations such as "deleting features with more than 20% missing values ​​and only one value, filtering features with a P value less than 0.05, retaining features with internal and external Spearman correlation coefficients less than 0.7, etc."; for digital pathology images, perform preprocessing operations such as "removing pathology image background". Through the above preprocessing process, the data accuracy, completeness and consistency of the multimodal data are guaranteed, and the training efficiency and prediction accuracy of the subsequent model are improved.

[0057] In addition, since the above-mentioned multimodal data is mainly divided into two formats, tabular data and image data, clinical data, proteomic data, and metabolomic data belong to tabular data, and the digital pathological images of the cell tissue scale belong to image data. Due to the large difference in the effective value range of data in different formats, clinical data includes two types of continuous data and classified data, proteomic data and metabolomic data are mainly continuous numbers less than 1, and a single picture of image data contains tens of thousands of pixels, so the number of effective cases that can match the four modalities is small. Therefore, in this embodiment, different model architectures are used for data in different formats to build a prediction model based on deep learning.

[0058] Specifically, refer to Figures 2(a) and 2(b), where 2(a) represents a schematic diagram of the overall structure of the prediction model; Figure 2(b) represents a schematic diagram of the specific structure of the table feature extractor. As shown in Figure 2(a), the prediction model 10 specifically includes a table feature extractor 11, an image feature extractor 12, and a multimodal fusion classifier 13. Among them, the table feature extractor 11 is used to extract features from modal data belonging to the table data; the image feature extractor 12 is used to extract features from modal data belonging to the image data; the multimodal fusion classifier 13 is connected to the output ends of the table feature extractor 11 and the image feature extractor 12. In one embodiment, the multimodal fusion classifier 13 is composed of an n-layer fully connected neural network 131 and a gated network 132, wherein the gated network is composed of m fully connected layers.

[0059] It should be noted that in actual design, the table feature extractor 11 and the image feature extractor 12 are not limited to one, and the specific number of table feature extractors 11 and image feature extractors 12 needs to be determined according to the number of modal data processed simultaneously. For example, in this embodiment, the three modal data of clinical data, proteomic data, and metabolomic data belong to the same table data, and then the table feature extractors 11 are configured as three, and the clinical data, proteomic data, and metabolomic data are simultaneously input into the prediction model, and feature extraction is performed by one table feature extractor 11 respectively, and one table feature extractor 11 corresponds to one type of modal data, and feature extraction processing is performed on the corresponding modal data.

[0060] Referring to FIG. 2( b ), in this embodiment, in order to better process tabular data, a table feature extractor specifically for processing tabular data is designed. Each table feature extractor 11 specifically includes a mapping coding layer 111, an auxiliary training layer 112, and a fully connected feature extraction layer 113 connected in sequence. The specific processing process of the table feature extractor 11 on the tabular data will be specifically introduced in the subsequent step S2.

[0061] Step S2, inputting the multimodal data into the constructed prediction model for training, wherein: identifying the category of each modal data, if it is identified as tabular data, performing feature aggregation on the modal data through the tabular feature extractor and encoding it into multiple one-dimensional feature vectors, and integrating the multiple one-dimensional feature vectors into a representative vector of the tabular data.

[0062] In this embodiment, in order to better process tabular data, a table feature extractor specifically for tabular data is designed. The table feature extractor can help allow the algorithm to train and update feature encoding while training model parameters end-to-end, thereby improving the ability to extract high-dimensional abstract features. Specifically, by identifying the category of each modal data of the multimodal data input to the prediction model, if it is identified as tabular data, the modal data will be feature aggregated and encoded into multiple one-dimensional feature vectors through a table feature extractor, and the multiple one-dimensional feature vectors will be integrated into a representative vector of the tabular data of the corresponding modality.

[0063] Specifically, in combination with FIG. 2(b), Figure 3 As shown, Figure 3 The detailed flow diagram of step S2 is shown. First, step S21: the features contained in the corresponding modal data are bucketed by the mapping coding layer 111 to unify the value range of the data, realize feature aggregation, and then encode each feature into a one-dimensional feature vector.

[0064] For step S21, the specific table data is divided into discrete data and continuous data, and the data types are statistically shown in Table 1 below. In this embodiment, the three modal data of clinical data, proteomic data, and metabolomic data belong to the same tabular data, and the table feature extractor 11 is configured as three. The clinical data, proteomic data, and metabolomic data are input into the prediction model at the same time, and feature extraction is performed by each table feature extractor 11. One table feature extractor 11 corresponds to one type of modal data, and feature extraction is performed on the corresponding modal data.

[0065] Taking the feature extraction of clinical data as an example, the mapping coding layer 111 of the table feature extractor 11 corresponding to the clinical data as the modality data performs bucketing processing on the features contained in the clinical data, that is, feature aggregation.

[0066] The mapping coding layer specifically uses a hash equation to map the original data type of clinical data to multiple categories to unify the value range of the data, thereby achieving feature aggregation of the modality data of clinical data. The hash equation H(x) represents the mapping of the original variable X={x 1 ,x 2 ,,x n} to k categories (k≤10):

[0067]

[0068] The processing of the two types of modal data, proteomic data and metabolomic data, is similar to the above-mentioned clinical data, and will not be described in detail in this embodiment.

[0069] Table 1: Specific data types of table data before and after feature aggregation

[0070]

[0071]

[0072] Then, step S22: after the corresponding modality data is bucketed by the mapping coding layer 111, each one-dimensional feature vector is further updated and optimized by the auxiliary training layer 112 to achieve coding optimization.

[0073] Specifically, in one embodiment, the table feature extractor is an end-to-end structure. During the encoding optimization process, the one-dimensional feature vector of each feature is extended to contain more information, and gradient feedback is further performed. Each one-dimensional feature vector is trained so that the information it contains is consistent with the prediction task. At the same time, each one-dimensional feature vector can potentially integrate the features of other modal data through gradients, thereby achieving encoding optimization.

[0074] In conventional methods, the expression of features is fixed when data or features are input into a neural network. Generally, it is a vector with a length of 1. However, when inputting features, the present invention randomly generates a one-dimensional feature vector, and the length of the one-dimensional feature vector can be arbitrary, taking a length of 10 as an example. If a vector with a length of 1 represents a position on a one-dimensional coordinate axis, and a vector with a length of 2 can represent a position on a two-dimensional coordinate axis, then the longer the one-dimensional feature vector is, the more information the vector contains. However, if the one-dimensional feature vector is a random vector, it contains too much noise and needs to be optimized according to the current data distribution and prediction task to achieve better prediction performance.

[0075] Therefore, in this embodiment, step S23: the multiple one-dimensional feature vectors are integrated into the table data representative vector through the multiple hidden layers of the fully connected feature extraction layer. In this embodiment, each one-dimensional feature vector passes through the fully connected feature extraction layer 113 to splice the vectors in sequence, and the spliced ​​vectors are uniformly passed through n hidden layers with activation functions (the activation function and the length and number of hidden layers here can be adjusted according to actual conditions, and n is at least 2), and the last hidden layer outputs the integrated modal representative vector, which is the overall representative vector of the current modal table data.

[0076] Specifically, the process of processing three kinds of tabular data, namely clinical data, proteomic data and metabolomic data, by a tabular feature extractor is taken as an example, as shown in FIG2(c), FIG2(c) shows a schematic diagram of processing different tabular data by a tabular feature extractor. The tabular feature extractor is applicable to three kinds of tabular data, namely clinical data, proteomic data and metabolomic data. There are three tabular feature extractors, and each tabular feature extractor adopts the same basic architecture (i.e., all include a mapping coding layer, an auxiliary training layer and a fully connected feature extraction layer), and the specific parameters are configured with different parameters, which are respectively used for the processing of clinical data, proteomic data and metabolomic data. Specifically, the clinical data, proteomic data and metabolomic data are respectively passed through the corresponding tabular feature extractors through the training process of executing the above steps S21-S23, and the tabular data representative vectors of the corresponding modalities are respectively obtained.

[0077] In this embodiment, a table feature extractor is designed, which is an end-to-end structure that can handle situations where the data volume is small and the number of valid cases of multimodal data is small, while covering tabular data with different values ​​and reasonably integrating data of each modality.

[0078] Step S3: If the recognition belongs to image data, the entire modal data is trained by the image feature extractor to obtain an image representative vector through integration.

[0079] In one embodiment, by specifically identifying the category of each modal data of the multimodal data input into the prediction model, if the identification belongs to image data, the entire image of the modal data is trained by an image feature extractor, and the image representative vector is integrated. In this embodiment, the image data is a dimensional difference between the balanced and tabular data, and the whole image processing method is adopted to avoid the common method of dividing the image into tens of thousands of slices and then extracting features. For example, in this embodiment, digital pathology image recognition belongs to image data, and the entire image of the modal data is trained by an image feature extractor, and the image representative vector corresponding to the digital pathology image is integrated.

[0080] In one embodiment, the image feature extractor specifically adopts the Resnet50 model as the basic architecture, replaces the last two layers of the aggregation layer and the classification layer of the Resnet50 model with new hidden layers, and the other layers remain unchanged.

[0081] Furthermore, in one embodiment, the number of layers of the table feature extractor and the image feature extractor can be configured so that the image representative vector output by the image feature extractor has the same length as the table data representative vector output by the table feature extractor.

[0082] In addition, it should be noted that in this embodiment, there is actually no order of execution for the feature extraction process of step S2 for identifying as table data and the feature extraction process of step S3 for identifying as image data, and they are usually executed simultaneously and in parallel.

[0083] Step S4: Use the multimodal fusion classifier to concatenate and fuse all the table data representative vectors with all the image representative vectors and then perform classification prediction.

[0084] In one embodiment, after the modal data identified as tabular data is processed through the above step S2 to obtain a tabular data representative vector, and after the modal data identified as image data is processed to obtain an image representative vector, all the tabular data representative vectors and all the image representative vectors are further input into a multimodal fusion classifier, and the tabular data representative vectors and the image representative vectors are further fused through the multimodal fusion classifier 13 to perform classification prediction to obtain the final chronic kidney disease progression prediction result.

[0085] Specifically, each of the table data representative vectors and each of the image representative vectors are configured to be applied to the predicted classification target respectively; the multimodal fusion classifier is composed of an n-layer fully connected neural network in conjunction with a gating network, wherein the gating network is composed of m fully connected layers, generating a first weight corresponding to each of the table data representative vectors and a second weight corresponding to each of the image representative vectors, and each of the table representative vectors is multiplied by its own first weight, and each of the image representative vectors is multiplied by its own second weight, and then spliced; the spliced ​​features enter the n-layer fully connected neural network for classification prediction; wherein n and m are both at least 2.

[0086] For example, taking the multimodal data of clinical data, proteomic data, metabolomic data, and digital pathological images as an example, after the clinical data, proteomic data, and metabolomic data identified as table data are processed through the above step S2 to obtain three table data representative vectors, and after the digital pathological image identified as image data is processed to obtain an image representative vector, the four representative vectors are further input into the multimodal fusion classifier, and the four representative vectors of the table data representative vector and the image representative vector are further fused through the multimodal fusion classifier 13 for classification prediction to obtain the final chronic kidney disease progression prediction result. Specifically, three table data representative vectors and one image representative vector are configured to be applied to the predicted classification targets respectively. The multimodal fusion classifier 13 is composed of an n-layer fully connected neural network 131 and a gating network 132, and the n-layer fully connected neural network 131 is connected to the output end of the gating network 132. The gating network 132 is specifically composed of m fully connected layers. According to the gating network 132, three table data representative vectors and one image representative vector are respectively generated with corresponding first weights, second weights, third weights, and fourth weights, and the three table data representative vectors and one image representative vector are respectively multiplied with the first weights, second weights, third weights, and fourth weights corresponding to themselves and then spliced. The spliced ​​features enter the n-layer fully connected neural network for classification prediction; wherein n and m are both at least 2.

[0087] Furthermore, in some embodiments, the predictive model is trained using a five-fold cross validation dataset.

[0088] In some embodiments, in order to verify the model, the importance of each modality in the multimodal data and the validity of each structure in the prediction model are determined by ablation analysis. After the model training is completed, the importance of each feature obtained at each stage in the training process of inputting the multimodal data into the constructed prediction model is determined by posterior analysis, and the key features are screened out and preliminarily identified as new prediction markers.

[0089] As a new standard for the diagnosis and treatment of chronic kidney disease, predictive markers are characteristic indicators that show obvious performance changes and need to be paid special attention to during the progression of chronic kidney disease. Specifically, a series of ablation analyses such as comparing AUC, F1 score, accuracy, recall, precision reading, and sensitivity changes can be used to determine the importance of each modality data and the importance of each feature.

[0090] In summary, the chronic kidney disease progression prediction method disclosed in the present invention integrates data of multiple scales and modes such as clinical data at the individual scale, pathological images at the cell tissue scale, and proteomics and metabolomics data at the molecular scale to construct a multimodal data set. According to different data formats, the clinical data, proteomic data, and metabolomics data belonging to the tabular data are respectively subjected to feature aggregation and encoding into multiple one-dimensional feature vectors through a tabular feature extractor, and the multiple one-dimensional feature vectors are integrated into the tabular data representative vectors of the corresponding modality; the digital pathological image belonging to the image data is trained by the image feature extractor for the whole picture, and the image representative vector is integrated; and finally, the four representative vectors of the tabular data representative vector and the image representative vector are fused by the multimodal fusion classifier for classification prediction, and the final chronic kidney disease progression prediction result is obtained. The method integrates clinical data, proteomic data, metabolomics data, and digital pathological images, predicts the progression of chronic kidney disease through a deep learning model, and effectively reduces the influence of distribution noise caused by modality heterogeneity while enriching feature information by aggregating features and optimizing encoding. The simultaneous training of model architecture and feature encoding greatly improves the model's ability to capture the interactions between different modalities and features in high-dimensional space, and can predict disease progression with extremely high accuracy. In addition, this method can also reasonably screen out key features to find new chronic kidney disease progression predictive markers, providing a new direction for marker research and development.

[0091] In addition, corresponding to the chronic kidney disease progression prediction method disclosed in the above embodiment, the present invention also discloses a chronic kidney disease progression prediction device 200, see Figure 4 As shown, Figure 4 The structure diagram of the chronic kidney disease progression prediction device 200 is shown, and the chronic kidney disease progression prediction device 200 at least comprises:

[0092] The data set acquisition unit 210 is used to collect multimodal data on chronic kidney disease, including clinical data at the individual scale, proteomic data and metabolomic data at the molecular scale, and digital pathological images at the cell tissue scale; wherein the clinical data, proteomic data, and metabolomic data are tabular data, and the digital pathological images at the cell tissue scale are image data.

[0093] The model training and prediction unit 220 is used to input the multimodal data into the constructed prediction model 10 for training, and the prediction model 10 includes multiple table feature extractors 11, at least one image feature extractor 12, and a multimodal fusion classifier 13; wherein: the model training and prediction unit 220 further includes:

[0094] The encoding module 2201 is used to perform feature aggregation on the modal data identified as the table data through the table feature extractor 11 and encode it into multiple one-dimensional feature vectors, and integrate the multiple one-dimensional feature vectors into the representative vector of the table data; and train the entire image of the modal data identified as the image data through the image feature extractor 12 to integrate and obtain the image representative vector;

[0095] The classification module 2202 is used to perform classification prediction after splicing and fusing all the table data representative vectors and all the image representative vectors through the multimodal fusion classifier 13.

[0096] In some embodiments, each table feature extractor 11 specifically includes a mapping coding layer 111, an auxiliary training layer 112, and a fully connected feature extraction layer 113, which are connected in sequence. The mapping coding layer 111 is used to sequentially perform bucket processing on the features contained in the corresponding modal data, unify the value range of the data, and then encode each feature into each one-dimensional feature vector; the auxiliary training layer 112 is connected to the output end of the mapping coding layer 111 to update each one-dimensional feature vector during the model training process; the fully connected feature extraction layer 113 is connected to the output end of the auxiliary training layer 112, and multiple hidden layers of the fully connected feature extraction layer 113 integrate the multiple one-dimensional feature vectors into a corresponding modality tabular data representative vector.

[0097] The image feature extractor 12 specifically adopts the Resnet50 model as the basic architecture, replaces the last two layers of the aggregation layer and the classification layer of the Resnet50 model with new hidden layers, and the other layers remain unchanged.

[0098] The multimodal fusion classifier 13 is connected to the output ends of the table feature extractor 11 and the image feature extractor 12. The multimodal fusion classifier 13 is composed of an n-layer fully connected neural network 131 and a gating network 132, wherein the gating network is composed of m fully connected layers.

[0099] In addition, those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each module of the described device can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0100] In addition, the present invention also provides a computer program product in one embodiment, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the chronic kidney disease progression prediction method provided by the above-mentioned methods.

[0101] The present invention also proposes a storage medium in another embodiment for storing a computer program for executing the chronic kidney disease progression prediction method. It should be understood that the storage medium in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0102] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A method for predicting the progression of chronic kidney disease, characterized in that: Include: Collect multimodal data on chronic kidney disease, including clinical data at the individual scale, proteomic data and metabolomic data at the molecular scale, and digital pathological images at the cell tissue scale; wherein the clinical data, proteomic data, and metabolomic data belong to tabular data, and the digital pathological images at the cell tissue scale belong to image data; The multimodal data is input into a constructed prediction model for training, and the prediction model comprises a table feature extractor, an image feature extractor, and a multimodal fusion classifier; wherein: Identify the category of each modal data. If the modal data is identified as table data, perform feature aggregation on the modal data and encode it into multiple one-dimensional feature vectors through the table feature extractor, and integrate the multiple one-dimensional feature vectors into a representative vector of the table data; If the recognition belongs to image data, the whole image of the modal data is trained by the image feature extractor, and the image representative vector is obtained by integration; The multimodal fusion classifier is used to concatenate and fuse all the table data representative vectors with all the image representative vectors and then perform classification prediction.

2. The method according to claim 1, characterized in that Each of the table feature extractors comprises a mapping encoding layer, an auxiliary training layer, and a fully connected feature extraction layer; The features contained in the corresponding modal data are sequentially bucketed through the mapping coding layer to unify the value range of the data, and then each feature is encoded into a one-dimensional feature vector; Updating and optimizing each one-dimensional feature vector through the auxiliary training layer; The multiple one-dimensional feature vectors are integrated into the table data representative vector through the multiple hidden layers of the fully connected feature extraction layer.

3. The method according to claim 2, characterized in that The mapping coding layer uses a hash equation to map the original data type of the corresponding modal data to multiple categories to unify the value range of the data.

4. The method according to claim 2, characterized in that: In the process of updating each one-dimensional feature vector through the auxiliary training layer, wherein: For each one-dimensional feature vector, the representative vector of each one-dimensional feature vector is extended to contain more information, and gradient backpropagation is performed to train each one-dimensional feature vector so that the information it contains is consistent with the prediction task.

5. The method according to claim 1, characterized in that: The image feature extractor adopts the Resnet50 model as the basic architecture, replaces the last two layers of the aggregation layer and the classification layer of the Resnet50 model with new hidden layers, and the other layers remain unchanged.

6. The method according to claim 1, characterized in that Configure each of the table data representative vectors and each of the image representative vectors to be applied to the predicted classification target respectively; The multimodal fusion classifier is composed of an n-layer fully connected neural network in conjunction with a gating network, wherein the gating network is composed of m fully connected layers, generating a first weight corresponding to each of the table data representative vectors and a second weight corresponding to each of the image representative vectors, and each of the table representative vectors is multiplied by its own first weight, and each of the image representative vectors is multiplied by its own second weight, and then spliced; the spliced ​​features enter the n-layer fully connected neural network for classification prediction; wherein n and m are both at least 2.

7. The method according to claim 1, characterized in that Also includes: The prediction model was trained using a five-fold cross-validation dataset.

8. The method according to claim 1, characterized in that Also includes: Analyzing the importance of each modality of the multimodal data, and, The importance of each feature obtained at each stage during the training process of inputting the multimodal data into the constructed prediction model is analyzed, key features are screened out, and the key features are preliminarily identified as prediction markers.

9. A chronic kidney disease progression prediction device, characterized in that: Include: A data set acquisition unit, used to collect multimodal data on chronic kidney disease, including clinical data at the individual scale, proteomic data and metabolomic data at the molecular scale, and digital pathological images at the cell tissue scale; wherein the clinical data, proteomic data, and metabolomic data belong to table data, and the digital pathological images at the cell tissue scale belong to image data; A model training and prediction unit, used to input the multimodal data into a constructed prediction model for training, the prediction model comprising a table feature extractor, an image feature extractor, and a multimodal fusion classifier; wherein: the model training and prediction unit further comprises: an encoding module, configured to perform feature aggregation on the modal data identified as tabular data through the tabular feature extractor and encode the modal data into a plurality of one-dimensional feature vectors, and integrate the plurality of one-dimensional feature vectors into a representative vector of the tabular data; and The modal data identified as image data is trained on the entire image through the image feature extractor, and integrated to obtain an image representative vector; The classification module is used to perform classification prediction after splicing and fusing all the table data representative vectors and all the image representative vectors through the multimodal fusion classifier.

10. The device according to claim 9, characterized in that Include: Each of the table feature extractors comprises a mapping encoding layer, an auxiliary training layer, and a fully connected feature extraction layer; The mapping coding layer is used to sequentially bucket the features contained in the corresponding modal data, unify the value range of the data, and then encode each feature into a one-dimensional feature vector; The auxiliary training layer is connected to the output end of the mapping coding layer to update and optimize each one-dimensional feature vector; The fully connected feature extraction layer is connected to the output end of the auxiliary training layer, and is used to integrate the multiple one-dimensional feature vectors into the table data representative vector through the multiple hidden layers of the fully connected feature extraction layer.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images