Chronic disease prognosis method, model, device, computer device and storage medium

Abstract

The application discloses a chronic disease prognosis method, model, device, computer equipment and storage medium, and the method comprises the steps of obtaining medical images and medical data of a person to be prognosed; extracting deep learning features and imaging features in the medical images; determining shared features and unique features of the deep learning features and the imaging features; performing fusion processing based on the shared features and the unique features to obtain target fusion features; constructing a multi-omics graph based on the target fusion features and the medical data; and performing prognosis processing based on the multi-omics graph to obtain a prognosis result of the person to be prognosed. The multi-omics graph constructed by the target fusion features and medical data of different modalities can improve the expression complementarity between different modalities, enhance the modeling capability of the model for complex pathological information, and significantly improve the accuracy and robustness of the prognosis prediction result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of smart medical treatment, and in particular to a chronic disease prognosis method, model, device, computer equipment and storage medium. BACKGROUND

[0002] Chronic diseases, such as chronic liver disease (CLD), have become an important global health problem, and accurate prognosis of chronic diseases is crucial. Traditional prognosis tools, including Child-Pugh score and end-stage liver disease model (MELD), often fail to take into account changes in patients, resulting in reduced accuracy of prognosis. Therefore, how to provide a chronic disease prognosis method capable of improving the accuracy of prognosis has become a problem to be solved. SUMMARY

[0003] Therefore, it is necessary to provide a chronic disease prognosis method, model, device, computer equipment and storage medium to solve the problem of low accuracy of traditional methods.

[0004] Obtaining medical images and medical data of a person to be prognosed;

[0005] Extracting deep learning features and imageomic features from the medical images;

[0006] Determining shared features and unique features from the deep learning features and the imageomic features;

[0007] Performing fusion processing based on the shared features and the unique features to obtain target fusion features;

[0008] Based on the target fusion features and the medical data, a multi-omics graph is constructed;

[0009] Based on the multi-omics graph, prognosis processing is performed to obtain the prognosis result of the person to be prognosed.

[0010] Optionally, the determining shared features and unique features from the deep learning features and the imageomic features comprises:

[0011] Calculating a first mean and a first covariance matrix of the deep learning features through a preset first encoder;

[0012] Based on the first mean and the first covariance matrix, a first latent variable corresponding to the deep learning features is determined;

[0013] Calculating a second mean and a second covariance matrix of the imageomic features through a preset second encoder;

[0014] determine a second latent variable corresponding to the radiomics features based on the second mean and the second covariance matrix;

[0015] determine shared features and unique features of both the first latent variable and the second latent variable.

[0016] Optionally, the determining shared features and unique features of both the first latent variable and the second latent variable comprises:

[0017] optimize the first latent variable and the second latent variable through a preset mutual information optimization mechanism to obtain a first target latent variable corresponding to the first latent variable and a second target latent variable corresponding to the second latent variable;

[0018] determine shared features and unique features of both the first target latent variable and the second target latent variable through a preset disentanglement mechanism.

[0019] Optionally, the optimizing the first latent variable and the second latent variable through a preset mutual information optimization mechanism to obtain a first target latent variable corresponding to the first latent variable and a second target latent variable corresponding to the second latent variable comprises:

[0020] adjust the first latent variable to obtain a first target latent variable, taking maximizing mutual information between the first latent variable and a preset target variable and minimizing mutual information between the first latent variable and the deep learning features as adjustment targets;

[0021] adjust the second latent variable to obtain a second target latent variable, taking maximizing mutual information between the second latent variable and a preset target variable and minimizing mutual information between the second latent variable and the radiomics features as adjustment targets.

[0022] Optionally, the constructing a multi-omics graph based on the target fusion features and the medical data comprises:

[0023] embedding the target fusion features and the medical data as nodes into a preset empty graph to obtain a multi-omics graph to be processed;

[0024] completely connecting nodes of the same modality and sparsely connecting nodes of different modalities in the multi-omics graph to be processed to obtain the multi-omics graph.

[0025] A chronic disease prognosis model for performing the chronic disease prognosis method, the chronic disease prognosis model comprising:

[0026] a feature extraction layer configured to extract deep learning features and radiomics features in the medical image;

[0027] a determination module configured to determine shared features and unique features in both the deep learning features and the radiomics features;

[0028] a feature fusion layer configured to perform fusion processing based on the shared features and the unique features to obtain target fusion features;

[0029] an image construction layer configured to construct a multi-omics graph based on the target fusion features and the medical data;

[0030] a graph neural network layer configured to perform prognosis processing based on the multi-omics graph to obtain a prognosis result of the person to be prognosed.

[0031] Optionally, the chronic disease prognosis model is obtained through training processing of a target loss function, the target loss function comprises a first sub-loss function and a second sub-loss function, the first sub-loss function is determined through a preset mutual information optimization mechanism, and the second sub-loss function is determined through a preset disentanglement mechanism.

[0032] A chronic disease prognosis device, comprising:

[0033] an acquisition module configured to acquire a medical image and medical data of a person to be prognosed;

[0034] an extraction module configured to extract deep learning features and radiomics features in the medical image;

[0035] a determination module configured to determine shared features and unique features in both the deep learning features and the radiomics features;

[0036] a fusion module configured to perform fusion processing based on the shared features and the unique features to obtain target fusion features;

[0037] a construction module configured to construct a multi-omics graph based on the target fusion features and the medical data;

[0038] a prognosis module configured to perform prognosis processing based on the multi-omics graph to obtain a prognosis result of the person to be prognosed.

[0039] A computer device comprises a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, and the processor implements the above-mentioned chronic disease prognosis method when executing the computer readable instructions.

[0040] A readable storage medium, on which computer readable instructions are stored, and the computer readable instructions are executed by a processor to implement the above-mentioned chronic disease prognosis method.

[0041] The above-mentioned chronic disease prognosis method, device, computer device and storage medium, obtain medical images and medical data of a person to be prognosed; extract deep learning features and imageomics features in the medical images; determine shared features and unique features in the deep learning features and the imageomics features; perform fusion processing based on the shared features and the unique features to obtain target fusion features; construct a multi-omics graph based on the target fusion features and the medical data; and perform prognosis processing based on the multi-omics graph to obtain a prognosis result of the person to be prognosed. By extracting deep learning features and imageomics features in medical images, and then performing fusion processing on the deep learning features and the imageomics features, more accurate target fusion features can be obtained, and the multi-omics graph constructed based on the target fusion features and medical data of different modalities can enhance the modeling ability of the model for complex pathological information on the basis of improving the expression complementarity between different modalities, thereby significantly improving the accuracy and robustness of the prognosis prediction result. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a flowchart of a chronic disease prognosis method in an embodiment of the present application;

[0044] Figure 2 is a structure diagram of a chronic disease prognosis model in an embodiment of the present application;

[0045] Figure 3 is a structure diagram of a chronic disease prognosis device in an embodiment of the present application;

[0046] Figure 4 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0047] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0048] In an embodiment, as shown in Figure 1 A chronic disease prognosis method is provided, comprising the following steps:

[0049] 101. Obtain medical images and medical data of a person to be prognosed.

[0050] In the embodiments of the present application, the chronic disease prognosis method described above can be applied to a medical management platform, and the medical management platform can be constructed by a server or a server cluster. The server or server cluster can be any electronic device with functions of image analysis, image recognition, image processing, data storage, data transmission, etc. The medical management platform can be deployed with a chronic disease prognosis model, so as to realize the chronic disease prognosis method through the chronic disease prognosis model. The medical management platform can receive the medical images and medical data through the data transmission function.

[0051] The person to be prognosed can be any person with a chronic disease, which can be determined according to actual needs. The medical images can be CT images (Computed Tomography, computed tomography images), or MRI images (Magnetic Resonance Imaging, magnetic resonance imaging), Ultrasound images (ultrasound imaging), etc.

[0052] The medical data can be data of the person to be prognosed during medical treatment, which can include basic demographic and physical characteristics, such as gender, height, and body mass index (BMI), 23-dimensional complete blood count (CBC) and biochemical index report, and 16-dimensional fat analysis report.

[0053] 102. Extract deep learning features and image features from the medical images.

[0054] In the embodiments of the present application, the deep learning features can be extracted by an image encoder. The image encoder can use a pre-trained three-dimensional residual network (ResNet50-3D) to perform feature encoding on the medical image, so as to obtain deep visual feature representation (i.e., deep learning features) with semantic expression capability. Alternatively, the image encoder can also use other deep neural network structures with volume image modeling capability, such as DenseNet, VGG, UNet based on 3D CNN, or Transformer structure for cross-modal feature learning. The specific structure can be adjusted according to the data type, task requirement or deployment environment.

[0055] The radiomics features can be obtained by extracting quantitative radiomics data from the medical image and performing feature selection on the quantitative radiomics data by a random forest selection mechanism. Specifically, the random forest-based feature selection mechanism can be used to filter the quantitative radiomics data, so as to retain a feature subset (i.e., the radiomics features) most relevant to the target task. The quantitative radiomics data refers to a set of statistical features of image shape, grayscale, texture, etc. expressed in a structured numerical form, which are extracted from a predefined region (such as a lesion region) in the medical image. Alternatively, other feature selection methods can be used to process the quantitative radiomics data, such as maximum relevance minimum redundancy (mRMR), L1 regularization, principal component analysis (PCA) or deep embedded feature selection network, etc. to obtain the radiomics features.

[0056] 103、In both the deep learning features and the radiomics features, shared features and unique features are determined.

[0057] In the embodiments of the present application, the shared features can include (s d , s r ), wherein s d is a shared feature in the deep learning features, and s r is a shared feature in the radiomics features. The unique features can include (u d , u r ), wherein u d is a unique feature in the deep learning features, and u r is a unique feature in the radiomics features.

[0058] Specifically, the deep learning features and the radiomics features can be vectorized to obtain first latent variables and second latent variables, and the first latent variables and the second latent variables can be optimized by a preset mutual information optimization mechanism to obtain first target latent variables corresponding to the first latent variables and second target latent variables corresponding to the second latent variables, and then shared features and unique features of the first target latent variables and the second target latent variables can be extracted by a preset disentanglement mechanism. The preset mutual information optimization mechanism can be determined by an information bottleneck (IB) mechanism, and the preset disentanglement mechanism can include two parallel projection sub-networks: a shared feature projection sub-network and a unique feature projection sub-network, which respectively map the input latent variables to a shared feature space and a modal unique space to realize disentanglement to determine the shared features and the unique features.

[0059] In a possible embodiment, only the disentanglement mechanism can be considered, and the first latent variables and the second latent variables can be directly mapped to the shared feature space and the modal unique space by the shared feature projection sub-network and the unique feature projection sub-network to realize disentanglement to determine the shared features and the unique features.

[0060] 104. The target fusion features are obtained by fusion processing based on the shared features and the unique features.

[0061] In the embodiments of the present application, the fusion processing can be realized by a transformer module. Specifically, the shared features and the unique features can be input into a preset transformer encoder, and the interaction relationship between the features of each dimension can be modeled by a self-attention mechanism. The transformer module can adaptively capture the key dependencies between different modal features, thereby generating a unified feature representation with discriminative ability as the target fusion features on the basis of global information fusion.

[0062] Alternatively, the fusion processing can also be realized by any other fusion manner, for example, can be a concatenation fusion, an attention-based fusion, a dual-branch MLP fusion or a GCN-based fusion, etc.

[0063] 105. A multi-omics graph is constructed based on the target fusion features and the medical data.

[0064] In the embodiment of the present application, the target fusion feature and the medical data can be embedded into a preset empty graph in the form of a feature vector to obtain a multi-omics graph to be processed. In the multi-omics graph to be processed, the nodes of the same modality are fully connected, and the nodes of different modalities are sparsely connected. The sparse connection is introduced between the nodes of different modalities with a probability. After the connection is completed, the above multi-omics graph can be obtained.

[0065] By fully connecting the nodes of the same modality and sparsely connecting the nodes of different modalities, complete intra-modality aggregation can be allowed, and cross-modality information propagation can be supported, thereby promoting global information aggregation through cross-modality interaction. It can be understood that the target fusion feature is one modality, and the medical data is also one modality.

[0066] It can be understood that the multi-omics graph is a graph structure data expression method for fusing and modeling information from multiple omics modalities (such as images, genes, and clinics). Specifically, the multi-omics graph organizes data (such as medical images, image omics, transcriptomes, and clinical information) from different modalities as node features or graph structure information, and models the relationships between samples, modalities, or features in the form of a graph.

[0067] 106. Perform prognosis processing based on the multi-omics graph to obtain a prognosis result of the personnel to be prognosed.

[0068] In the embodiment of the present application, the above prognosis processing can be realized by a preset graph neural network (GNN). The multi-omics graph is provided to the above preset graph neural network, and the above prognosis result is output by the above graph neural network.

[0069] The above graph neural network can be composed of two layers of TransformerConv, and a LeakyReLU activation function is integrated between each layer. The LeakyReLU activation enhances the gradient flow and improves the performance of the model. Finally, a classification head applied to the aggregated feature representation is used to complete the prognosis classification.

[0070] In a possible embodiment, the above prognosis processing can also be realized by different types of graph neural networks. The above graph neural network includes but is not limited to a GCN network based on graph convolution, a GraphSAGE network based on neighbor sampling, a GAT network based on an attention mechanism, and a graph isomorphism network (GIN) structure with strong structure expression capability. The network structure can select an appropriate number of layers according to actual task requirements, and an activation function (such as ReLU or LeakyReLU) and a classification head are used to complete the final prognosis result prediction.

[0071] Therefore, the present application can select different types of graph structure encoders to realize prognosis modeling according to the scale, graph connection density and task characteristics of multi-omics graphs, without being limited to a specific graph neural network structure, so as to balance modeling accuracy and reasoning efficiency.

[0072] It should be noted that the prognosis result can be a result after the chronic disease of the person to be prognosed is processed. The chronic disease can be any disease with long-term evolution characteristics, such as chronic liver disease, chronic kidney disease, diabetes, or the like. When the chronic disease is chronic liver disease, the medical image can be a medical image related to chronic liver disease, and the medical data can be medical data related to chronic liver disease. When the chronic disease is chronic kidney disease, the medical image can be a medical image related to chronic kidney disease, and the medical data can be medical data related to chronic kidney disease. The chronic diseases such as chronic liver disease and chronic kidney disease are exemplary, and any disease with long-term evolution characteristics can be used as a chronic disease, and the corresponding medical image and medical data can be obtained to implement any chronic disease prognosis method provided by the present application.

[0073] In the embodiment of the present application, the medical image and the medical data of the person to be prognosed are obtained. In the medical image, deep learning features and image omics features are extracted. Shared features and unique features are determined from the deep learning features and the image omics features. The shared features and the unique features are fused to obtain target fusion features. The multi-omics graph is constructed based on the target fusion features and the medical data. The prognosis result of the person to be prognosed is obtained by prognosis processing based on the multi-omics graph. By extracting deep learning features and image omics features from the medical image, and then fusing the deep learning features and the image omics features, more accurate target fusion features can be obtained. The multi-omics graph constructed based on the target fusion features and the medical data of different modalities can enhance the modeling ability of the model to complex pathological information on the basis of improving the expression complementarity between different modalities, thereby significantly improving the accuracy and robustness of the prognosis prediction result.

[0074] It can be understood that in the specific embodiments of the present application, data related to medical images, medical data, deep learning features, image omics features, etc. are involved. When the embodiments of the present application are applied to specific products or technologies, the user's permission or consent is required, and the collection, use and processing of related data, as well as the construction, training and use of chronic disease prognosis models, encoders and graph neural networks, need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0075] Optionally, in the step of determining the shared features and the unique features in both the deep learning features and the radiomics features, the first mean and the first covariance matrix of the deep learning features can be calculated by a preset first encoder; based on the first mean and the first covariance matrix, the first latent variable corresponding to the deep learning features is determined; the second mean and the second covariance matrix of the radiomics features are calculated by a preset second encoder; based on the second mean and the second covariance matrix, the second latent variable corresponding to the radiomics features is determined; and the shared features and the unique features in both the first latent variable and the second latent variable are determined.

[0076] In the embodiments of the present application, the above-mentioned preset first encoder and preset second encoder can be MLP encoders. The first latent variable and the second latent variable can be obtained by introducing a latent variable modeling mechanism to encode the deep learning features and the radiomics features. Specifically, the deep learning features are first processed by the preset first encoder to obtain a first mean vector and a first covariance matrix; based on the first mean and the first covariance matrix, the first latent variable corresponding to the deep learning features is generated by reparameterization trick from a standard normal distribution.

[0077] Similarly, the radiomics features can be processed by the preset second encoder to obtain a second mean vector and a second covariance matrix, and the second latent variable corresponding to the radiomics features is generated by reparameterization from the mean and the covariance.

[0078] Subsequently, the first latent variable and the second latent variable can be structurally deconstructed by the above-mentioned preset feature disentanglement mechanism to extract the shared features and the unique features in both.

[0079] Optionally, in the step of determining the shared features and the unique features in both the first latent variable and the second latent variable, the first latent variable and the second latent variable can be optimized by a preset mutual information optimization mechanism to obtain a first target latent variable corresponding to the first latent variable and a second target latent variable corresponding to the second latent variable; and the shared features and the unique features in both the first target latent variable and the second target latent variable are determined by a preset disentanglement mechanism.

[0080] In the embodiment of the present application, in order to further improve the discriminative ability of the latent variable in the task and the controllability of the information, a preset mutual information optimization mechanism can be introduced to further optimize the first latent variable and the second latent variable after they are generated.

[0081] Specifically, the first latent variable is optimized by the mutual information optimization mechanism to obtain a first target latent variable; at the same time, the same processing is performed on the second latent variable to obtain a second target latent variable.

[0082] The mutual information optimization mechanism is preferably a target function constructed based on the information bottleneck (IB) principle, which realizes the dual control of the representation ability and the information compression by maximizing the mutual information between the latent variable and the task target and minimizing the redundant information between the latent variable and the original input.

[0083] Subsequently, the first target latent variable and the second target latent variable can be structurally deconstructed based on the preset feature disentanglement mechanism to extract shared features and unique features in them, respectively. The shared features are used to represent common information existing in the two modalities, and the unique features are used to represent specific representations existing only in the corresponding modality.

[0084] Specifically, the preset feature disentanglement mechanism can include the conditions that the shared features across modalities should be as similar as possible and the unique features in each modality should be as unique as possible. To achieve the conditions that the shared features across modalities should be as similar as possible and the unique features in each modality should be as unique as possible, the preset disentanglement mechanism is specifically implemented by the following multi-modal feature disentanglement (MFD) loss function:

[0085]

[0086]

[0087]

[0088] Wherein, the MFD represents the multi-modal feature disentanglement loss function, E1(), E2() represents the encoding of the original input of different modalities into feature representation, f com () represents a subspace mapping function for extracting shared semantic features from the encoded representation, f dist represents a subspace mapping function for extracting modality unique features from the encoded representation, ‖‖2 represents the Euclidean distance (L2 norm), which is used to measure the distance between features, represents the average operation of the deep learning features x d and the radiomics features x r sample. The Lcom denotes the extraction of shared features, and the L dist denotes the extraction of unique features.

[0089] Optionally, in the step of optimizing the first latent variable and the second latent variable through the preset mutual information optimization mechanism to obtain the first target latent variable corresponding to the first latent variable and the second target latent variable corresponding to the second latent variable, the first latent variable can be adjusted to obtain the first target latent variable by taking maximizing the mutual information between the first latent variable and the preset target variable and minimizing the mutual information between the first latent variable and the deep learning feature as an adjustment target; and the second latent variable can be adjusted to obtain the second target latent variable by taking maximizing the mutual information between the second latent variable and the preset target variable and minimizing the mutual information between the second latent variable and the radiomics feature as an adjustment target.

[0090] In the embodiment of the present application, the preset target variable, i.e. the task target, can be understood as a prognosis target for the person to be prognosed. The mutual information optimization mechanism is preferably a target function constructed based on the information bottleneck principle, which realizes the dual control of the representation ability and the information compression by maximizing the mutual information between the latent variable and the task target and minimizing the redundant information between the latent variable and the original input. To maximize the mutual information between the latent variable and the task target and minimize the redundant information between the latent variable and the original input, the following target function can be used:

[0091]

[0092] wherein, L MIB denotes the target function, I(·,·) denotes the mutual information, m∈{d,r} respectively denotes the deep and radiology features, z v denotes the fused visual feature, x m is the input variable, and denotes the original input feature of the mth modality, z m is the intermediate variable, and denotes the hidden representation extracted by the encoder of the mth modality, y is the output label, and β is a hyperparameter, and γ is a weight factor. It can be understood that the first latent variable is the data of one modality, and the second latent variable is the data of another modality. The first latent variable is brought into the input variable to obtain the corresponding output.

[0093] To simplify the optimization of the above L MIB , it can be re-expressed as a variational upper bound:

[0094]

[0095] wherein, the above N is the total number of samples, n is the current sample number, m is the current modality number, y is the input data of the nth sample in the mth modality, n y is the output label of the nth sample, q is the posterior distribution on the latent variable z, N(0, 1) is a normal distribution, z m z is the latent variable corresponding to the mth modality v q is the fused latent variable, q (y n |z m ) is the probability of predicting the label given the modality latent variable, q (y n |z v ) is the probability of predicting the label given the fused latent variable, log q (y n |) is the log-likelihood term used to maximize the prediction accuracy, which is equivalent to the log term in cross-entropy, β is the KL term weight hyperparameter, γ is the fusion representation loss weight, KL is the Kullback-Leibler divergence used to measure the difference between two distributions.

[0096] At the same time, when the objective function is re-expressed as a variational upper bound, it also includes:

[0097]

[0098] where f E is an MLP encoder that can simultaneously predict the mean u and the covariance matrix ∑, p(z|x) is the posterior distribution (true) representing the true posterior distribution of the latent variable z given the input x, but it cannot be directly solved and needs to be approximated using variational approximation. q (z|x) is the variational distribution (approximate posterior), is represented as a Gaussian distribution, θ is a set of parameters, is the mean branch of the encoder, is the covariance branch of the encoder.

[0099] Optionally, in the step of constructing a multi-omics graph based on the target fusion features and medical data, the target fusion features and medical data can be embedded as nodes into a preset empty graph to obtain a multi-omics graph to be processed; in the multi-omics graph to be processed, nodes of the same modality are fully connected, and nodes of different modalities are sparsely connected to obtain a multi-omics graph.

[0100] In the embodiments of the present application, the above node embedding can be embedding the target fusion features and medical data as feature vectors into a preset empty graph to obtain a multi-omics graph to be processed.

[0101] Specifically, the target fusion feature is embedded as a first modality node embedding, and the medical data is embedded as a second modality node embedding, both in the form of a numerical vector and stored as node attributes in the graph structure.

[0102] Secondly, the connection mode between nodes in the graph is constructed based on the modality attribution relationship: for all nodes under the same modality, a complete connection strategy is adopted, that is, an edge is established between any two nodes; for nodes between different modalities, a sparse connection strategy is adopted, and an edge is established only when there is significant semantic correlation between nodes, for example, an edge can be screened based on a cosine similarity or Euclidean distance between feature vectors. Finally, a multi-omics graph structure with stable structure and effective cross-modality expression is constructed.

[0103] Through the embedding and connection strategies described above, the information between nodes within the same modality is fully communicated, and the high correlation structure between modalities is preserved, thereby improving the expression ability of the graph structure in multi-modality semantic modeling and providing stable support for subsequent structure learning of the graph neural network.

[0104] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0105] As shown in Figure 2 , the present embodiment also provides a structural schematic diagram of a chronic disease prognosis model, which specifically comprises:

[0106] a feature extraction layer, configured to extract deep learning features and imageomic features in a medical image;

[0107] a determination module, configured to determine shared features and unique features from the deep learning features and the imageomic features;

[0108] a feature fusion layer, configured to perform fusion processing based on the shared features and the unique features to obtain target fusion features;

[0109] an image construction layer, configured to construct a multi-omics graph based on the target fusion features and medical data;

[0110] a graph neural network layer, configured to perform prognosis processing based on the multi-omics graph to obtain a prognosis result of a person to be prognosed.

[0111] In the present embodiment, as shown in Figure 2 , the feature extraction layer specifically acquires image input and imageomic input, and extracts deep learning features x dand extract the radiomics features x in the radiomics input through feature selection (i.e., a feature selection mechanism) r .

[0112] The determination module extracts the mean u and covariance matrix ∑ corresponding to the deep learning features and the mean u and covariance matrix ∑ corresponding to the radiomics features through E1 (i.e., a preset first encoder) and E2 (i.e., a preset second encoder). The first latent variable is determined through the mean u and covariance matrix ∑ corresponding to the deep learning features, the second latent variable is determined through the mean u and covariance matrix ∑ corresponding to the radiomics features, and the first latent variable and the second latent variable are optimized through an information bottleneck constraint to obtain a first target latent variable z d and a second target latent variable z r of the second latent variable. The shared features and unique features of the first target latent variable and the second target latent variable are determined, wherein the shared features include shared features s d of the deep learning features and shared features s r of the radiomics features, and the unique features include unique features u d of the deep learning features and unique features u r .

[0113] The feature fusion layer fuses the shared features and the unique features through a Transformer Block to obtain a target fusion feature z v .

[0114] The image construction layer embeds the medical data (i.e., the clinical information input) and the target fusion feature into a preset empty graph as nodes to obtain a multi-omics graph to be processed, and performs edge embedding on the multi-omics graph to be processed to obtain the multi-omics graph.

[0115] The graph neural network layer processes the multi-omics graph through a graph neural network GNN to obtain a prognosis result, which can include a good prognosis or a poor prognosis. Alternatively, the prognosis result can be further subdivided into levels, which can be quantified according to the good or poor prognosis to obtain a prognosis level, wherein a higher prognosis level represents a better prognosis, and a lower prognosis level represents a poorer prognosis.

[0116] Optionally, the chronic disease prognosis model is obtained after being trained by a target loss function, the target loss function includes a first sub-loss function and a second sub-loss function, the first sub-loss function is determined through a preset mutual information optimization mechanism, and the second sub-loss function is determined through a preset disentanglement mechanism

[0117] In the embodiments of the present application, the preset mutual information optimization mechanism can maximize the mutual information between the latent variable and the task target, and minimize the redundant information between the latent variable and the original input. To achieve the mutual information optimization mechanism, a first sub-loss function can be set to achieve the mutual information optimization mechanism. The first sub-loss function (i.e., L MIB ) can be specifically:

[0118]

[0119] wherein L MIB represents the first sub-loss function, I(·,·) represents mutual information, m represents depth and radiology features respectively, z v represents the fused visual feature, x m is an input variable, represents the original input feature of the mth modality, z m is an intermediate variable, represents the hidden representation extracted by the encoder of the mth modality, y is an output label, and β is a hyperparameter, and γ is a weight factor. It can be understood that the first latent variable is the data of one modality, and the second latent variable is the data of another modality. The first latent variable is brought into the input variable, and the corresponding output can be obtained.

[0120] Alternatively, the above formula can be re-expressed as a variational upper bound:

[0121]

[0122] wherein N is the total number of samples, n is the current sample number, m is the current modality number, is the input data of the nth sample in the mth modality, y n is the output label of the nth sample, is the hidden variable distribution, generally referred to as the posterior distribution on the hidden variable z, N(0,1) is a normal distribution, z m is the latent variable z v of the mth modality, n is the fused latent variable, qθ(y m |z n ) is the probability of predicting the label given the modality latent variable, qθ(y v |z n ) is the probability of predicting the label given the fused latent variable, log qθ(y E is a log likelihood term used to maximize the prediction accuracy, which is equivalent to the log term in cross entropy, β is a KL term weight hyperparameter, γ is a fusion representation loss weight, KL is a Kullback-Leibler divergence used to measure the difference between two distributions.is a MLP encoder that can predict both the mean u and the covariance matrix ∑ simultaneously, p(z|x) is the posterior distribution (true) representing the true posterior distribution of the latent variable z given the input x, but it is not directly solvable and needs to be approximated using variational approximation. qθ(z|x) is the variational distribution (approximate posterior), is represented as a Gaussian distribution, θ is a set of parameters, is the mean branch of the encoder, is the covariance branch of the encoder.

[0123] The preset disentanglement mechanism can be that the shared features across modalities should be as similar as possible and the unique features in each modality should be as unique as possible. To achieve the above disentanglement mechanism, a second sub-loss function can be set to achieve the above disentanglement mechanism. The second sub-loss function (i.e., L MFD ) can be specifically:

[0124]

[0125]

[0126]

[0127] wherein the L MFD is represented as a multi-modal feature disentanglement loss function (i.e., a second sub-loss function), the E1(), E2() is represented as encoding the original input of different modalities into feature representation, f com () is represented as a subspace mapping function for extracting shared semantic features from the encoded representation, f dist is represented as a subspace mapping function for extracting modality unique features from the encoded representation, ‖‖2 represents the Euclidean distance (L2 norm) for measuring the distance between features, represents the average operation on the deep learning features x d and the radiomics features x r samples. The L com represents the extraction of shared features, and the L dist represents the extraction of unique features.

[0128] After the first sub-loss function (L MIB ) and the second sub-loss function (L MFD ) are determined, the target loss function can be constructed according to the first sub-loss function and the second sub-loss function. It can be further illustrated by the following formula:

[0129] L MDIB = L MiB + γL MFD

[0130] wherein, the above L MDIB is denoted as the above target loss function, and the above γ is denoted as an adjustable weight parameter.

[0131] Specifically, the loss value between the output result of the chronic disease prognosis model to be trained and the actual result can be calculated by the above target loss function, and the parameter adjustment of the chronic disease prognosis model to be trained is performed to minimize the loss value, until the loss value converges at the minimum, or the adjustment is stopped when the preset number of times is reached, and the chronic disease prognosis model is obtained.

[0132] It should be noted that the prognosis result can be the result after the chronic disease of the person to be prognosed is prognosed, and the chronic disease can be any disease with long-term evolution characteristics, such as chronic liver disease, chronic kidney disease, or diabetes, etc. When the chronic disease is chronic liver disease, the medical image can be a medical image related to chronic liver disease, and the medical data can be medical data related to chronic liver disease. When the chronic disease is chronic kidney disease, the medical image can be a medical image related to chronic kidney disease, and the medical data can be medical data related to chronic kidney disease. The chronic liver disease and chronic kidney disease are exemplary, and any disease with long-term evolution characteristics can be used as a chronic disease, and the corresponding medical data, medical image, etc. can be used as a training set to train the chronic disease prognosis model by the training set and the target loss function, and the medical image and medical data of the person to be prognosed are obtained to implement the chronic disease prognosis method provided by the embodiment of the application.

[0133] In an embodiment, a chronic disease prognosis device is provided, which corresponds to the chronic disease prognosis method in the above embodiment. As shown in the figure, the chronic disease prognosis device includes an acquisition module 301, an extraction module 302, a determination module 303, a fusion module 304, a construction module 305, and a prognosis module 306. The functions of each module are described in detail as follows: Figure 3

[0134] The acquisition module 301 is configured to acquire the medical image and medical data of the person to be prognosed.

[0135] The extraction module 302 is configured to extract the deep learning features and the radiomics features from the medical image.

[0136] The determination module 303 is configured to determine the shared features and unique features of the deep learning features and the radiomics features. ​

[0137] a fusion module 304, configured to perform fusion processing based on the shared feature and the unique feature to obtain a target fusion feature;

[0138] a construction module 305, configured to construct a multi-omics graph based on the target fusion feature and the medical data;

[0139] a prognosis module 306, configured to perform prognosis processing based on the multi-omics graph to obtain a prognosis result of the person to be prognosed.

[0140] Optionally, the determination module 303 comprises:

[0141] a first calculation sub-module, configured to calculate a first mean and a first covariance matrix of the deep learning feature by using a preset first encoder;

[0142] a first determination sub-module, configured to determine a first latent variable corresponding to the deep learning feature based on the first mean and the first covariance matrix;

[0143] a second calculation sub-module, configured to calculate a second mean and a second covariance matrix of the image-omics feature by using a preset second encoder;

[0144] a second determination sub-module, configured to determine a second latent variable corresponding to the image-omics feature based on the second mean and the second covariance matrix;

[0145] a third determination sub-module, configured to determine shared features and unique features of the first latent variable and the second latent variable.

[0146] Optionally, the third determination sub-module comprises:

[0147] a first optimization unit, configured to optimize the first latent variable and the second latent variable by using a preset mutual information optimization mechanism to obtain a first target latent variable corresponding to the first latent variable and a second target latent variable corresponding to the second latent variable;

[0148] a first determination unit, configured to determine shared features and unique features of the first target latent variable and the second target latent variable by using a preset disentanglement mechanism.

[0149] Optionally, the first optimization unit comprises:

[0150] The first adjusting unit is configured to adjust the first latent variable to obtain a first target latent variable, by taking maximizing mutual information between the first latent variable and a preset target variable and minimizing mutual information between the first latent variable and the deep learning feature as an adjusting target.

[0151] The second adjusting unit is configured to adjust the second latent variable to obtain a second target latent variable, by taking maximizing mutual information between the second latent variable and a preset target variable and minimizing mutual information between the second latent variable and the radiomics feature as an adjusting target.

[0152] Optionally, the construction module 305 comprises:

[0153] The first embedding submodule is configured to embed the target fusion feature and the medical data as nodes into a preset empty graph to obtain a to-be-processed multi-omics graph.

[0154] The first calculation submodule is configured to perform full connection between nodes of the same modality and sparse connection between nodes of different modalities in the to-be-processed multi-omics graph to obtain the multi-omics graph.

[0155] The above-mentioned various modules in the chronic disease prognosis apparatus can be realized by software, hardware and combinations thereof in whole or in part. The above-mentioned various modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the above-mentioned various modules.

[0156] In an embodiment, a computer device is provided, which can be a terminal device, and an internal structure diagram thereof can be as shown in Figure 4 The computer device comprises a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a readable storage medium. The readable storage medium stores computer readable instructions. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer readable instructions are executed by the processor to implement a chronic disease prognosis method. The readable storage medium provided in the embodiment comprises a non-volatile readable storage medium and a volatile readable storage medium.

[0157] In the embodiments of the present application, a computer device is provided, which comprises a memory, a processor and computer readable instructions stored in the memory and executable on the processor. When the processor executes the computer readable instructions, the steps of the above-mentioned chronic disease prognosis method are implemented.

[0158] In the application embodiment, a readable storage medium is provided, and the readable storage medium stores computer readable instructions. When the computer readable instructions are executed by a processor, the steps of the chronic disease prognosis method are implemented.

[0159] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by computer readable instructions instructing related hardware. The computer readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer readable instructions are executed, the processes of the above-mentioned embodiments of the method can be included. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0160] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified. In actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0161] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for prognosticating chronic diseases, characterized in that, The method includes: Acquire medical images and medical data of individuals awaiting diagnosis; Deep learning features and radiomics features are extracted from the medical images; Among the deep learning features and the radiomics features, the shared features and unique features of the two are identified; Based on the shared features and the unique features, a fusion process is performed to obtain the target fusion feature; Based on the target fusion features and the medical data, a multi-omics learning graph is constructed. Based on the multi-omics graph, prognostic processing is performed to obtain the prognostic results of the individuals to be diagnosed. The step of identifying shared and unique features between the deep learning features and the radiomics features includes: The first mean and the first covariance matrix of the deep learning features are calculated using a preset first encoder. Based on the first mean and the first covariance matrix, the first latent variable corresponding to the deep learning feature is determined; The second mean and second covariance matrix of the image omics features are calculated using a preset second encoder. Based on the second mean and the second covariance matrix, the second latent variable corresponding to the radiomics feature is determined; Among the first latent variable and the second latent variable, the shared features and unique features of the two are determined; The step of determining the shared features and unique features of the first latent variable and the second latent variable includes: By using a preset mutual information optimization mechanism, the first latent variable and the second latent variable are optimized to obtain the first target latent variable corresponding to the first latent variable and the second target latent variable corresponding to the second latent variable. By using a pre-defined deentanglement mechanism, the shared features and unique features of the first target latent variable and the second target latent variable are determined. The step of optimizing the first latent variable and the second latent variable through a preset mutual information optimization mechanism to obtain the first target latent variable corresponding to the first latent variable and the second target latent variable corresponding to the second latent variable includes: The first target latent variable is obtained by adjusting the first latent variable with the goal of maximizing the mutual information between the first latent variable and the preset target variable, and minimizing the mutual information between the first latent variable and the deep learning feature. The second latent variable is adjusted to maximize the mutual information between the second latent variable and the preset target variable, and minimize the mutual information between the second latent variable and the radiomics features, so as to obtain the second target latent variable.

2. The chronic disease prognosis method as described in claim 1, characterized in that, Based on the target fusion features and the medical data, a multi-omics graph is constructed, including: The target fusion features and the medical data are embedded as nodes into a preset empty graph to obtain multiple sets of learning graphs to be processed; In the multi-set learning graph to be processed, nodes of the same mode are fully connected, and nodes of different modes are sparsely connected to obtain the multi-set learning graph.

3. A prognostic model for chronic diseases, characterized in that, The chronic disease prognostic model is used to perform the chronic disease prognostic method as described in any one of claims 1 to 2, wherein the chronic disease prognostic model comprises: The feature extraction layer is used to extract deep learning features and radiomics features from the medical images. The determination module is used to determine the shared features and unique features between the deep learning features and the radiomics features. A feature fusion layer is used to perform fusion processing based on the shared features and the unique features to obtain the target fused features; An image construction layer is used to construct a multi-omics learning map based on the target fusion features and the medical data; A graph neural network layer is used to perform prognostic processing based on the multi-omics graph to obtain the prognostic results of the person to be diagnosed.

4. The chronic disease prognostic model as described in claim 3, characterized in that, The chronic disease prognostic model is obtained after training using a target loss function, which includes a first sub-loss function and a second sub-loss function. The first sub-loss function is determined by a preset mutual information optimization mechanism, and the second sub-loss function is determined by a preset de-entanglement mechanism.

5. A chronic disease prognosis device, characterized in that, include: The acquisition module is used to acquire medical images and medical data of individuals awaiting diagnosis. The extraction module is used to extract deep learning features and radiomics features from the medical images. The determination module is used to determine the shared features and unique features between the deep learning features and the radiomics features. The fusion module is used to perform fusion processing based on the shared features and the unique features to obtain the target fusion features; A construction module is used to construct a multi-omics graph based on the target fusion features and the medical data; The prognosis module is used to perform prognostic processing based on the multi-omics graph to obtain the prognostic results of the person to be diagnosed. The determining module includes: The first calculation submodule is used to calculate the first mean and the first covariance matrix of the deep learning features through a preset first encoder. The first determining submodule is used to determine the first latent variable corresponding to the deep learning feature based on the first mean and the first covariance matrix. The second calculation submodule is used to calculate the second mean and the second covariance matrix of the image omics features through a preset second encoder. The second determination submodule is used to determine the second latent variable corresponding to the radiomics feature based on the second mean and the second covariance matrix; The third determination submodule is used to determine the shared features and unique features of the first latent variable and the second latent variable. The third determining submodule includes: The first optimization unit is used to optimize the first latent variable and the second latent variable through a preset mutual information optimization mechanism to obtain the first target latent variable corresponding to the first latent variable and the second target latent variable corresponding to the second latent variable. The first determining unit is used to determine the shared features and unique features of the first target latent variable and the second target latent variable through a preset deentanglement mechanism. The first optimization unit includes: The first adjustment unit is used to adjust the first latent variable to obtain the first target latent variable by taking the adjustment target as maximizing the mutual information between the first latent variable and the preset target variable and minimizing the mutual information between the first latent variable and the deep learning feature. The second adjustment unit is used to adjust the second latent variable to obtain the second target latent variable by taking maximizing the mutual information between the second latent variable and the preset target variable and minimizing the mutual information between the second latent variable and the radiomics features as the adjustment target.

6. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and running on the processor, characterized in that, When the processor executes the computer-readable instructions, it implements the chronic disease prognosis method as described in any one of claims 1 to 2.

7. A readable storage medium having computer-readable instructions stored thereon, characterized in that, When the computer-readable instructions are executed by a processor, they implement the chronic disease prognosis method as described in any one of claims 1 to 2.

Images