Target model acquisition method, prognosis evaluation value determination method, device, equipment and medium

Abstract

The present disclosure provides a target model acquisition method, a prognosis evaluation value determination method, an apparatus, a device, and a medium. The method comprises: acquiring a plurality of sample groups respectively corresponding to a plurality of sample users, wherein each sample group comprises sample information of multiple modalities; iteratively training a preset model based on the plurality of sample groups to obtain a target model, wherein the target model is used to predict a prognosis evaluation value of a target object; wherein in each iteration training, the preset model is used to extract features from the sample information of multiple modalities in the current sample group, and based on the extracted sample features, a predicted prognosis evaluation value and a consistency expression value are determined; wherein the consistency expression value is used to represent the consistency degree of the sample features corresponding to the same target disease; and the parameters of the preset model are updated based on the predicted prognosis evaluation value, the prognosis evaluation label corresponding to the current sample group, and the consistency expression value.

Description

Technical Field

[0001] This disclosure relates to the field of information processing technology, and in particular to a method for obtaining a target model, a method for determining a prognostic evaluation value, an apparatus, equipment, and a medium. Background Technology

[0002] Gliomas originate from glial cells and are the most common tumors of the central nervous system, accounting for about 50% to 60% of intracranial tumors. The incidence rate is showing an increasing trend year by year. Gliomas patients generally need to undergo prognostic assessment after surgery.

[0003] In prognostic assessment, traditional prognosis is based on factors such as the size of the lesion and the extent of its spread, which has certain limitations. Summary of the Invention

[0004] This disclosure provides a method for obtaining a target model, the method comprising:

[0005] Obtain sample groups corresponding to multiple sample users, wherein the sample groups include sample information of multiple modalities, and the sample information of multiple modalities includes at least two of MRI sample images, clinical sample information and gene sample information;

[0006] Based on multiple sample groups, the preset model is iteratively trained to obtain the target model, which is used to predict the prognostic evaluation value of the target object.

[0007] In each iteration of training, the following steps are performed:

[0008] Using the preset model, feature extraction is performed on sample information of multiple modalities in the current sample group, and based on the extracted sample features, the predicted prognostic assessment value and the consistency expression value are determined; wherein, the consistency expression value is used to characterize the degree of consistency of each sample feature corresponding to the same target disease;

[0009] The parameters of the preset model are updated based on the predicted prognostic evaluation value, the prognostic evaluation label corresponding to the current sample group, and the consistency expression value.

[0010] In an optional example, updating the parameters of the preset model based on the predicted prognostic evaluation value, the prognostic evaluation label corresponding to the current sample group, and the consistency expression value includes:

[0011] Obtain the difference between the predicted prognostic value and the prognostic label;

[0012] The parameters of the preset model are updated with the goal of minimizing the differences and maximizing the consistent expression value.

[0013] In one optional example, updating the parameters of the preset model with the goal of minimizing the differences and maximizing the consistent expression value includes:

[0014] Based on the aforementioned differences and the aforementioned consistent expression values, the following loss function is constructed:

[0015] loss = ∑ i (y′ i -y i ) 2 -consistency;

[0016] Based on the loss value of the loss function, the parameters of the preset model are updated with the goal of minimizing the difference and maximizing the consistent expression value.

[0017] Wherein, loss represents the loss value, y' i Indicates the predicted prognostic evaluation value, y i The prognostic assessment label is indicated by "consistency" (meaning the consistency expression value) and "loss" (meaning the loss value).

[0018] In one optional example, determining the consistency expression value based on the extracted features of each sample includes:

[0019] Transpose each of the sample features to obtain the transposed feature corresponding to each of the sample features;

[0020] For two different sample features, the transpose of one of the sample features is fused with the transpose of the other sample feature to obtain the corresponding fused feature value;

[0021] The consistency expression value is determined based on each of the fusion feature values.

[0022] In one optional example, the sample information for multiple modalities includes multiple sub-sample information. The step of using the preset model to extract features from the sample information for multiple modalities in the current sample group includes:

[0023] For each modality of sample information, feature extraction is performed on multiple sub-sample information in the sample information of that modality to obtain multiple corresponding sub-feature vectors;

[0024] For each modality, multiple sub-feature vectors corresponding to the sample information are fused to obtain the sample features corresponding to that modality.

[0025] In one optional example, the feature fusion of multiple sub-feature vectors corresponding to the sample information of each modality to obtain the sample features corresponding to the sample information of that modality includes:

[0026] Determine the attention value between every two sub-feature vectors, the attention value being used to characterize the tightness between the two sub-feature vectors;

[0027] Based on the attention value, multiple sub-feature vectors are fused to obtain the sample features corresponding to the sample information of the modality.

[0028] In one optional example, fusing multiple sub-feature vectors based on the attention value to obtain sample features corresponding to the sample information of that modality includes:

[0029] For each sub-feature vector, based on the attention value between the sub-feature vector and all other sub-feature vectors, all other sub-feature vectors are fused into the sub-feature vector to obtain the fused sub-vector of the sub-feature vector;

[0030] The multiple fused sub-vectors are then fused again to obtain the sample features corresponding to the sample information of the modality.

[0031] In one optional example, the parameters of the preset model include a first parameter matrix, the MR sample images include multiple slice sample images, and determining the attention value between every two sub-feature vectors includes:

[0032] For each slice sample image included in the MR sample image, based on the current value of the first parameter matrix, an attention value is determined between the sub-feature vectors corresponding to every two slice sample images.

[0033] In one optional example, the parameters of the preset model include a second parameter matrix and a third parameter matrix. The feature fusion of multiple sub-feature vectors corresponding to the sample information of each modality to obtain the sample features corresponding to that modality includes:

[0034] For each clinical subsample information in the clinical sample information, based on the current value of the second parameter matrix, feature fusion is performed on the sub-feature vectors corresponding to each clinical subsample information to obtain the sample features corresponding to the clinical sample information;

[0035] For each gene subsample in the gene sample information, based on the current value of the third parameter matrix, feature fusion is performed on the sub-feature vectors corresponding to each gene subsample to obtain the sample features corresponding to the gene sample information.

[0036] In one optional example, the parameters of the preset model include a parameter set corresponding to the clinical sample information, wherein the clinical sample information includes numerical sub-sample information and non-numerical sub-sample information. The step of using the preset model to extract features from sample information of multiple modalities in the current sample group includes:

[0037] For the clinical sample information in the current sample group, the non-numerical sub-sample information is converted into a first sub-feature vector;

[0038] Based on the current values ​​of each parameter in the parameter set, the numerical subsample information is mapped to the target space to obtain the corresponding second sub-feature vector; the parameters in the parameter set are used to determine the dimension of the target space and the value at each spatial point.

[0039] The first sub-feature vector and the second sub-vector are fused to obtain the sample features corresponding to the clinical sample information.

[0040] In an optional example, the parameter set includes a first parameter vector, a second parameter vector, and a fourth parameter matrix. The step of mapping the numerical subsample information to the target space based on the current values ​​of each parameter in the parameter set to obtain the corresponding second sub-feature vector includes:

[0041] Based on the current value of the first parameter vector, the numerical subsample information is mapped to the first dimension in the target space to obtain the mapping value of the first dimension; wherein, the first parameter vector is used to determine the value of the spatial point of the target space in the first dimension;

[0042] Based on the first dimension mapping value, the current value of the second parameter vector, and the current value of the fourth parameter matrix, the second sub-feature vector is determined; wherein, the second parameter vector is used to determine the value of the spatial point of the target space in the second dimension, and the fourth parameter matrix is ​​used to assign parameters to each spatial location in the first and second dimensions.

[0043] In an optional example, the parameter set further includes multiple third parameter vectors. After determining the second sub-feature vector based on the first dimension mapping value, the current value of the second parameter vector, and the current value of the fourth parameter matrix, the method further includes:

[0044] Based on the second sub-feature vector and the plurality of third parameter vectors, the second sub-feature vector is modified according to the following formula:

[0045]

[0046] Where va is the corrected second sub-feature vector, sa is the second sub-feature vector, and a1, a2 and a3 are the third parameter vectors respectively;

[0047] The first sub-feature vector and the second sub-vector are fused to obtain the sample features corresponding to the clinical sample information, including:

[0048] The first sub-feature vector and the corrected second sub-vector are fused to obtain the sample features corresponding to the clinical sample information.

[0049] In one optional example, the parameters of the preset model include a dimensional parameter matrix corresponding to the sample information for each modality, and the determination of the predicted prognostic evaluation value and consistency expression value based on the extracted sample features includes:

[0050] Based on the sample features corresponding to the sample information of each modality and the dimensional parameter matrix, the dimensional transformation is performed on the sample features corresponding to the sample information of that modality to obtain the transformed sample features.

[0051] Based on the transformed sample features corresponding to sample information from multiple modalities, the predicted prognostic evaluation value and the consistency expression value are determined.

[0052] In one optional example, the preset model includes a fusion module, a prediction branch, a consistency expression branch, and a data processing module corresponding to the sample information of each modality; the step of using the preset model to extract features from the sample information of multiple modalities in the current sample group, and determining the predicted prognosis evaluation value and the consistency expression value based on the extracted sample features, includes:

[0053] For each modality in the current sample group, the data processing module corresponding to the sample information of that modality is used to extract features from the corresponding sample information;

[0054] The fusion module is used to fuse the sample features output by each of the data processing modules.

[0055] Using the prediction branch, the predicted prognostic evaluation value corresponding to the feature output by the fusion module is determined;

[0056] Using the consistency expression branch, the consistency expression value corresponding to the sample features output by each of the data processing modules is determined.

[0057] In an optional example, the gene sample information includes information on at least one of the following genes: isocitrate dehydrogenase, chromosome 1p / 19q combined deletion state, telomerase reverse transcriptase gene promoter, and O6-methylguanine-DNA methyltransferase promoter region methylation.

[0058] The clinical sample information includes at least one of the following: gender, age, histological diagnosis, tumor grade, medication information, and history of malignant tumors.

[0059] This disclosure also provides a method for determining a prognostic assessment value, the method comprising:

[0060] Information on multiple modalities of the subject to be tested is obtained, including MRI images, clinical information, and genetic information.

[0061] The information from the multiple modalities is input into the target model to obtain the prognostic evaluation value of the object under test; wherein the target model is obtained according to the target model acquisition method.

[0062] This disclosure also provides a device for acquiring a target model, the device comprising:

[0063] The sample acquisition module is used to acquire sample groups corresponding to multiple sample users. The sample groups include sample information of multiple modalities, including at least two of MRI sample images, clinical sample information, and gene sample information.

[0064] The training module is used to iteratively train the preset model based on multiple sample groups to obtain the target model, which is used to predict the prognostic evaluation value of the target object.

[0065] In each iteration of training, the following steps are performed:

[0066] Using the preset model, feature extraction is performed on sample information of multiple modalities in the current sample group, and based on the extracted sample features, the predicted prognostic assessment value and the consistency expression value are determined; wherein, the consistency expression value is used to characterize the degree of consistency of each sample feature corresponding to the same target disease;

[0067] The parameters of the preset model are updated based on the predicted prognostic evaluation value, the prognostic evaluation label corresponding to the current sample group, and the consistency expression value.

[0068] This disclosure also provides a prognostic assessment value determination apparatus, the apparatus comprising:

[0069] The information acquisition module is used to acquire information on multiple modalities of the test object, including MRI images, clinical information, and genetic information.

[0070] The input module is used to input the information of the multiple modalities into the target model to obtain the prognostic evaluation value of the object under test;

[0071] The target model is obtained according to the target model acquisition method.

[0072] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the target model telephone acquisition method, or, when executed, implements the prognostic evaluation value determination method.

[0073] This disclosure also provides a computer-readable storage medium storing a computer program that causes a processor to execute the method for obtaining the target model as described, or to implement the method for determining the prognostic evaluation value as described.

[0074] By employing the technical solution of this disclosure embodiment, sample groups corresponding to multiple sample users can be obtained, and a preset model can be iteratively trained based on multiple sample groups to obtain a target model. The target model is used to predict the prognostic evaluation value of the target object. In each iteration of training, the preset model can be used to extract features from the sample information of multiple modalities in the current sample group, and the predicted prognostic evaluation value and consistency expression value can be determined based on the extracted sample features. Then, the parameters of the preset model are updated based on the predicted prognostic evaluation value, the prognostic evaluation label corresponding to the current sample group, and the consistency expression value.

[0075] On the one hand, since the sample set disclosed herein includes sample information of multiple modalities, specifically including at least two of MRI sample images, clinical sample information and genetic sample information, it is possible to use information from different dimensions of sample users as reference factors to predict prognostic assessment values, thereby achieving rapid multi-factor prognostic analysis.

[0076] On the other hand, since the consistency expression value is used as the basis for updating the parameters of the preset model during training, and the consistency expression value is used to characterize the consistency of each sample feature corresponding to the same target disease, the extracted sample features can be understood as a prognostic factor for prognosis prediction. In this way, as training progresses, the preset model can extract the prognostic factors (sample features) related to the target disease in each modality, and gradually discard the prognostic factors that are not related to the target disease in the information of various modalities. This makes the prognostic factors selected by the model clinically important, which can help improve the interpretability of the model. As a result, the output of the target model has high prognostic reference value.

[0077] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the dimensions and shapes of the figures in the drawings do not reflect actual proportions and are only intended to illustrate the content of the present invention. The same or similar reference numerals in the drawings represent the same or similar elements or elements having the same or similar functions.

[0079] Figure 1 A flowchart illustrating the steps of the method for obtaining the target model is shown schematically.

[0080] Figure 2 A schematic diagram of a pre-defined model is shown.

[0081] Figure 3 A schematic diagram of the image data processing module of this disclosure is shown.

[0082] Figure 4 schematically shown Figure 3 A schematic diagram of the ResNet network structure;

[0083] Figure 5 A flowchart illustrating the steps of a method for determining prognostic assessment values ​​is shown schematically.

[0084] Figure 6 A schematic diagram of a target model acquisition device is shown.

[0085] Figure 7 A schematic diagram of a prognostic assessment value determination device is shown.

[0086] Figure 8 A schematic block diagram of the electronic device disclosed herein is shown. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0088] In related technologies, the prognosis of diseases is generally assessed based on lesion size and the presence or absence of enhancement, which has certain limitations. Taking glioma as an example, gliomas originate from glial cells and are the most common tumors of the central nervous system, accounting for approximately 50%–60% of intracranial tumors, with an increasing incidence rate year by year. The World Health Organization classifies gliomas into low-grade (I and II) and high-grade (III and IV) gliomas. High-grade gliomas (HGG) have a poor prognosis. Grade IV glioblastoma has the highest malignancy, with a 10-year survival rate of less than 3% and a median survival of approximately 12–14 months. Previous studies have used tumor location, size, resection extent, and traditional imaging methods as prognostic predictors to assess the prognosis of gliomas, but this approach has certain limitations.

[0089] However, using multivariate analysis for assessment requires researchers to follow up with numerous patients throughout the diagnosis, treatment, and postoperative process, collecting treatment information and patients' physiological indicators, and then analyzing and screening for factors contributing to imaging prognosis. This work is extremely resource-intensive and time-consuming; the manual analysis and screening of prognostic factors alone takes a long time, resulting in low efficiency.

[0090] In view of this, this disclosure proposes a prognostic assessment method. Specifically, a target model is trained using multi-modal sample information as training samples, and the target model is used for prognostic assessment. The multi-modal sample information includes at least two of the following: MRI sample images, clinical sample information, and genetic sample information. This enriches the information sources, allowing for the screening of prognostic factors from multiple dimensions. During model training, the parameters of the preset model are updated using consistency expression value as a factor, thereby enabling the screening of prognostic factors closely related to the target disease from multi-modal information, thus improving the clinical importance of prognostic factors. Consequently, the results output by the target model have high prognostic reference value.

[0091] Reference Figure 1 As shown, a flowchart illustrating the steps of the method for obtaining the target model of this disclosure is presented, as follows: Figure 1 As shown, the specific steps may include:

[0092] Step S101: Obtain sample groups corresponding to multiple sample users. The sample groups include sample information of multiple modalities, including at least two of MRI sample images, clinical sample information, and gene sample information.

[0093] Step S102: Based on multiple sample groups, iteratively train the preset model to obtain the target model, which is used to predict the prognostic evaluation value of the target object;

[0094] In each iteration of training in step S102, the following steps are performed:

[0095] Step S1021: Using a preset model, feature extraction is performed on sample information from multiple modalities in the current sample group. Based on the extracted sample features, the predicted prognostic assessment value and the consistency expression value are determined. The consistency expression value characterizes the degree of consistency between sample information from multiple modalities and the same target disease.

[0096] Step S1022: Update the parameters of the preset model based on the predicted prognostic evaluation value, the prognostic evaluation label corresponding to the current sample group, and the consistency expression value.

[0097] In this embodiment, the sample user refers to a user who clearly knows the prognostic survival and quality of life after being diagnosed with the target disease. The target disease can be any clinically known disease, such as brain tumors, common liver, gallbladder, and lung tumors, as described above, and is not specifically limited here.

[0098] The age and gender of the sample users should be as diverse as possible. The sample information from multiple modalities can include at least two of the following: MRI sample images, clinical sample information, and genetic sample information. Different modalities of sample information can be used to reflect the characteristics of sample users in corresponding dimensions after contracting the target disease. For example, MRI sample information can reflect the morphological characteristics of sample users in organs and tissues after contracting the target disease, clinical sample information can reflect the characteristics of sample users in diagnosis and treatment after contracting the target disease, and genetic sample information can reflect the expression status of certain genes in sample users after contracting the target disease.

[0099] MR sample images can be used to reflect the morphological characteristics of organs and tissues corresponding to a target disease suffered by a sample user. Specifically, MRI sample images can be obtained directly from MR imaging equipment, storage, or any other suitable source. For example, after obtaining the appropriate permissions, MR sample images of the sample user can be retrieved from a medical database. It should be noted that these MR sample images are related to the target disease suffered by the sample user. For example, if the sample user suffers from a glioma, the MR sample image is an MR image of the sample user's brain, which can reflect the morphological characteristics of brain tissue. Similarly, if the sample user suffers from a liver tumor, the MR sample image could be an MR image of the user's abdomen, which can reflect the morphological characteristics of the abdominal cavity tissues.

[0100] Clinical sample information may include information received by the sample user during the diagnosis and treatment process, such as drug information, hospitalization information, treatment plan information, attending physician information, hospital information, etc.

[0101] The gene sample information may include information about genes related to the occurrence and prognosis of the target disease. Specifically, the information for each gene in the gene sample includes the gene name and its expression status. This is because the onset and prognosis of a disease can be reflected in the expression of certain genes. For example, in the case of glioma, taking the TERT (telomerase reverse transcriptase) gene as an example, it is one of the important genes encoding the telomerase complex. The TERT gene has no transcriptional activity in the vast majority of non-tumor cells, but TERT gene mutations exist in 73% of tumors, such as promoter mutations, gene translocations, and DNA amplification. In other words, the expression categories of the above-mentioned genes are related to tumors to a certain extent.

[0102] In this embodiment, at least two of the following—MRI sample images, clinical sample information, and gene sample information—can be used as a sample group for a sample user. In one specific example, the MRI sample image in the sample group is mandatory; that is, any one or both of the clinical sample information and gene sample information can be combined with the MRI sample image to obtain the sample group. Specifically, the sample group may include MRI sample images and clinical sample information, or it may include MRI sample images and gene sample information, or it may include MRI sample images, clinical sample information, and gene sample information.

[0103] In this method, multiple sample groups can be used as training samples to iteratively train the preset model. Specifically, during each training session, multiple sample groups can be input into the preset model in batches, or one sample group can be input at a time.

[0104] In this embodiment, in each iteration of training, a preset model can be used to extract features from the sample information of multiple modalities in the current sample group. The current sample group refers to the sample group that is input into the preset model in this iteration. When multiple sample groups are input into the preset model each time, the current sample group refers to any one of the multiple sample groups input in this iteration.

[0105] The preset model can be used to extract features from the sample information of each modality in the current sample group, thereby obtaining the sample features corresponding to each sample information. Specifically, for MR image samples, the extracted features are the influence omics feature vectors; for clinical sample information, the extracted features are the feature vectors obtained by transforming the clinical information into feature vectors; and for gene sample information, the extracted features are the feature vectors obtained by transforming the gene information into feature vectors.

[0106] In this process, the predicted prognostic evaluation value of the sample object corresponding to the current sample group can be determined based on the sample features extracted from each modality information. In specific implementation, the predicted prognostic evaluation value can be determined based on the fused features after feature fusion of the sample features corresponding to multiple modal information.

[0107] In practice, sample information from different modalities should all contain descriptions of the target disease, and each should contain a large amount of information consistent with the target disease. However, sample information from different modalities also contains information unrelated to the target disease. For example, MR sample images contain images of other normal tissue sites in addition to lesion sites. Images of normal tissue sites may not be relevant in sample information from other modalities, such as clinical sample information. Therefore, the pre-set model should learn to discard this useless information during the training process.

[0108] Therefore, in this embodiment, the preset model can determine the consistency expression value based on the sample features corresponding to multiple modal information, and incorporate the consistency expression value into the construction of the loss function to update the parameters of the preset model, thereby enabling the preset model to continuously enhance the expression degree of the extracted sample features for the target disease based on the consistency expression value.

[0109] In this way, the consistency expression value can be used to characterize the consistency of sample features corresponding to multiple modal information in reflecting the target disease, that is, whether each extracted sample feature consistently expresses the target disease.

[0110] For example, when the consistency expression value is high, the sample features extracted from MR sample images, clinical sample information, and gene sample information are all features used to express the target disease. For MR sample images, the extracted sample features are the features used to express the lesion site of the target disease. For clinical sample information, the extracted sample features are the expression of the diagnosis and treatment plan of the target disease, as well as the expression of the target disease in the patient's age, occupation, etc. For gene sample information, the extracted sample features are used to characterize the expression of the target disease in related genes.

[0111] In this disclosure, the prognostic assessment label represents the actual prognosis of the sample subjects. If it is necessary to predict the prognostic survival period, the prognostic assessment label represents the actual prognostic survival period of the sample subjects, and the predicted prognostic assessment value can be expressed as the prognostic years. If it is necessary to predict the prognostic quality of life, the prognostic assessment label represents the actual prognostic quality of life level of the sample subjects, including high level, low level and medium level, and the predicted prognostic assessment value can be expressed as the prognostic quality of life level.

[0112] The technical solution of this embodiment incorporates consistency expression values, predicted prognostic values, and prognostic labels into the construction of the loss function when updating the parameters of the preset model. As the model trains, on the one hand, based on the consistency expression values, the expression level of extracted sample features to the target disease can be enhanced, ensuring that the prognostic factors used to determine the prognostic value are strongly correlated with the target disease. This increases the clinical importance of prognostic factors, making the prognostic value more valuable for medical reference. On the other hand, the parameters of the preset model can be continuously updated based on the difference between the predicted prognostic value and the prognostic label. As training progresses, the predicted prognostic value can approach the prognostic label more closely, making the prognostic value increasingly closer to the actual prognostic value, thereby further improving the clinical reference value of the prognostic value.

[0113] On the other hand, this disclosure uses sample information from multiple modalities to predict prognosis, thereby leveraging the complementarity between sample information from different modalities as prognostic factors for the target disease, thus enriching the data sources for prognostic factors and improving the medical reference value of prognostic assessment values.

[0114] In an optional implementation, when the target disease is glioma, the gene sample information may include information on at least one of the following genes: isocitrate dehydrogenase IDH, chromosome 1p / 19q combined deletion status, telomerase reverse transcriptase gene promoter, and methylation of the O6-methylguanine-DNA methyltransferase promoter region; the clinical sample information includes at least one of the following clinical information: sex, age, histological diagnosis, tumor grade, medication information, and history of malignant tumors.

[0115] The gene sample information includes the gene name and expression class for each gene, which can vary depending on the gene. Specifically, the IDH gene's expression class is listed as mutant and wild-type; the 1p / 19q gene's expression status includes deletion and non-deletion states; the O6-methylguanine-DNA methyltransferase promoter region MGMT's expression class is listed as methylated and unmethylated; and the telomerase reverse transcriptase gene TERT's expression class includes mutant and wild-type. Specific information for each gene type is shown in Table 1 below.

[0116] Table 1 - Gene Sample Information

[0117]

[0118] Clinical sample information may include one or more of the following: the sample subject's gender, age, histological diagnosis, tumor grade, medication information, and history of malignant tumors. Of course, in practice, it may not be limited to the clinical information mentioned above, and may include too much clinical information, such as the sample subject's occupation and region. Any information that is related to the occurrence and prognosis of the target disease can be used as clinical information.

[0119] In one example, clinical sample information can be presented in the following table:

[0120] Table 2 - Clinical Sample Information

[0121]

[0122] In one embodiment, the sample information of multiple modalities may include MR sample images, clinical sample information and gene sample information. That is, each sample group includes MR sample images, clinical sample information and gene sample information. Thus, feature extraction of the sample information of the three modalities is required in each iteration of training.

[0123] During the training process, each iteration includes the stages of feature extraction, constructing a loss function based on the model's output, and updating parameters.

[0124] Each modality's sample information can include multiple sub-sample information. In the feature extraction stage, features can be extracted from each sub-sample information in the sample information of each modality. Then, the features of multiple sub-sample information in the sample information of a modality are fused to obtain the sample features corresponding to the sample information of that modality.

[0125] The two stages will be described separately below:

[0126] Phase 1: Feature extraction phase.

[0127] As described above, in one embodiment, the sample information of each modality includes multiple sub-sample information. For the sample information of each modality, features are extracted from the multiple sub-sample information in the sample information of that modality to obtain multiple corresponding sub-feature vectors. Then, the multiple sub-feature vectors corresponding to the sample information of each modality are fused to obtain the sample features corresponding to the sample information of that modality.

[0128] The feature extraction for each sub-sample information can be performed as described in the above embodiments. After obtaining the sub-feature vectors corresponding to the sub-sample information, feature fusion can be performed according to the following process to obtain sample features:

[0129] For MR sample images, subsample information can be a slice of the MR sample image. In feature extraction, features can be extracted from each slice of the sample image to obtain the sub-feature vector corresponding to each slice of the sample image. Then, the sub-feature vectors corresponding to each slice of the sample image are fused to obtain the sample features of the MR sample image.

[0130] In the context of clinical sample information, subsample information can be a type of clinical information within the clinical sample information, such as age, gender, and tumor grade. During feature extraction, feature extraction is performed on each type of information separately. Specifically, each type of information is converted into a feature vector to obtain the sub-feature vector corresponding to each type of information. Then, the sub-feature vectors corresponding to various clinical information in the clinical sample information are fused to obtain the sample features of the clinical sample information.

[0131] In the case of gene sample information, subsample information can be information about one type of gene in the gene sample information, such as information about IDH or TERT gene promoter. In this case, the information of each gene can be converted into a sub-feature vector. Then, the sub-feature vectors corresponding to the information of various genes in the gene sample information are fused to obtain the sample features of the gene sample information.

[0132] In one alternative example, when fusing the sub-feature vectors corresponding to the sample information of each modality, the fusion can be performed directly according to the preset weights corresponding to each sub-feature vector. For example, preset weights can be manually set for each slice sample image, and preset weights can also be set for each clinical information and each gene information. These preset weights can characterize the importance of the seed sample information to the prognosis. Thus, feature fusion can be performed based on the importance of each sub-feature vector, thereby fusing the features that are more important for prognostic assessment, which can improve the medical value of prognostic assessment.

[0133] In another example, an attention mechanism can be incorporated into the pre-defined model. For sample information of each modality, the degree of correlation between each sub-feature vector can be determined through this attention mechanism. Then, based on the degree of correlation, each sub-feature vector is fused, thereby incorporating features with a high degree of correlation in the sample information of that modality. This improves the correlation between prognostic factors, making the prognostic factors assessed by the target model closely related, further increasing clinical importance and making the target model clinically interpretable.

[0134] In practice, the attention value between every two sub-feature vectors can be determined. Based on the attention value, multiple sub-feature vectors are fused to obtain the sample features corresponding to the sample information of this modality.

[0135] As mentioned above, the attention value is used to characterize the tightness between two sub-feature vectors.

[0136] In one example, when fusing sub-feature vectors based on the attention value between each pair of sub-feature vectors, the two sub-feature vectors can be fused together to obtain a fused vector. This results in multiple fused vectors, which are then fused together to obtain the sample features. For instance, if the sub-feature vectors include vectors i, j, and k, vectors i and j can be fused to obtain fused vector ij; similarly, fused vectors ik and jk are obtained. These fused vectors ij, ik, and jk are then fused together to obtain the sample features.

[0137] In another example, for each sub-feature vector, based on the attention value between the sub-feature vector and all other sub-feature vectors, all other sub-feature vectors can be fused into the sub-feature vector to obtain the fused sub-vector; multiple fused sub-vectors can be fused again to obtain the sample features corresponding to the sample information of this modality.

[0138] For example, the sub-feature vectors include vector i, vector j, and vector k. During fusion, vectors j and k can be fused into vector i according to the attention values ​​between vector i and vector j, and between vector i and vector k, to obtain the fused sub-vector i'. Similarly, fused sub-vectors j' and k' are obtained. Then, fused sub-vectors i', j', and k' are fused to obtain the sample features.

[0139] For clinical sample information, the attention value can be determined and feature fusion can be performed based on the attention value as follows:

[0140] When determining the attention value between each pair of sub-feature vectors, it can be based on the average attention value between multiple clinical information and the two sub-feature vectors. The average attention value between multiple clinical information can be obtained by referring to the following formula (I).

[0141]

[0142] Among them, v a v represents the sub-feature vector corresponding to age. g v represents the sub-feature vector corresponding to gender. h v represents the sub-feature vector corresponding to the histological diagnosis. hom v represents the sub-feature vector corresponding to a history of malignant tumors. dThe sub-feature vector corresponding to medication information, v gr denoted as the sub-feature vector corresponding to tumor grading information; S represents the average attention value.

[0143] Next, the attention value between each pair of sub-feature vectors can be determined according to the following formula (II):

[0144]

[0145] Where Si represents the attention value between sub-feature vectors vi and vj.

[0146] Accordingly, when performing feature fusion, a sub-feature vector can be fused with all other sub-feature vectors, and then the fused sub-vectors can be fused together. Specifically, taking the fusion of sub-feature vectors corresponding to age information in clinical information as an example, the fused sub-vector corresponding to age information can be determined according to the following formula (III):

[0147]

[0148] In Formula (III), va_att is the fused sub-vector corresponding to the age information.

[0149] For gene sample information, the process of determining the attention value and performing feature fusion based on the attention value can be referred to the above-mentioned clinical sample information, and will not be repeated here.

[0150] For MR sample images, the attention value can be determined and feature fusion performed based on the attention value as follows:

[0151] In one example, during the feature extraction and feature fusion stages, since it is necessary to extract clinically important prognostic factors, that is, to extract clinically important feature vectors, parameter matrices can be set for both the feature extraction and feature fusion stages. The parameter matrices can be continuously updated as the model is trained, thereby extracting the clinically important prognostic factors.

[0152] Specifically, as described above, the MR sample image includes multiple slice sample images, each of which represents a sub-sample information. Specifically, for each slice sample image included in the MR sample image, an attention value can be determined between the sub-feature vectors corresponding to every two slice sample images based on the current value of the first parameter matrix; then, the attention value can be used to fuse the various sub-feature vectors.

[0153] In practice, the attention value between the sub-feature vectors corresponding to every two slice sample images can be determined according to the following formula (iv):

[0154]

[0155] In formula (iv), Q and K are the first parameter matrices, where the values ​​of Q and K can be different. In practice, Q and K can be 512×512 parameter matrices. vi is the sub-feature vector corresponding to the i-th slice sample image, vj is the sub-feature vector corresponding to the j-th slice sample image, and α... i,j It is the attention value between the i-th slice sample image and the j-th slice sample image;

[0156] Next, according to the following formula (V), for each sub-feature vector, based on the attention value between the sub-feature vector and all other sub-feature vectors, all other sub-feature vectors are fused into the sub-feature vector to obtain the fused sub-vector of the sub-feature vector:

[0157]

[0158] In formula (5), SV i 1 It is the fused sub-vector corresponding to the i-th slice sample image, and n represents the total number of slice sample images.

[0159] Then, the fused sub-vectors are fused according to the following formula (vi) to obtain the sample features of the MR sample image:

[0160]

[0161] SV represents the sample features of an MR sample image. i This represents the fused sub-vector corresponding to the i-th slice sample image.

[0162] In a further embodiment, since parameter matrices are set for both the feature extraction and feature fusion stages, these matrices can be continuously updated as the model trains, thereby extracting highly important prognostic factors. Specifically, for clinical sample information and gene sample information, fusion can be further performed based on the set parameter matrices during the fusion of sub-feature vectors.

[0163] Specifically, for each clinical subsample information in the clinical sample information, based on the current value of the second parameter matrix, feature fusion is performed on the sub-feature vectors corresponding to each clinical subsample information to obtain the sample features corresponding to the clinical sample information.

[0164] For each gene subsample in the gene sample information, based on the current value of the third parameter matrix, feature fusion is performed on the sub-feature vectors corresponding to each gene subsample information to obtain the sample features corresponding to the gene sample information.

[0165] Specifically, after obtaining each fused sub-vector corresponding to the sub-feature vector from the above formula (III), feature fusion can be performed on the fused sub-vectors corresponding to the sub-feature vectors corresponding to each clinical sub-sample information based on the current value of the second parameter matrix, to obtain the sample features corresponding to the clinical sample information. Specifically, fusion can be performed according to the following formulas (VII) and (VIII):

[0166]

[0167] S1 = vp T ·(va_att+vg_att+vh_att+vgr_att+vd_att+vhom_att) formula (8);

[0168] Wherein, CV represents the sample features corresponding to the clinical sample information, and vp is the second parameter matrix, which can be a 128×1 parameter vector.

[0169] The fusion of the fused sub-vectors corresponding to each gene sub-sample information can also be performed by referring to the above formulas (VII) and (VIII). The second parameter matrix and the third parameter matrix can both be 128×1 parameter vectors. The parameters in the second parameter matrix and the third parameter matrix are updated as the preset model is updated, that is, they are updated according to the loss value of the loss function.

[0170] In a further example, clinical sample information includes numerical and non-numerical sub-sample information. During the feature extraction stage, based on the parameter set set for the preset model, the numerical sub-sample information can be mapped to a vector space according to the model's parameter set. During training, the parameters in the parameter set are continuously updated, allowing the numerical clinical information of different sample objects to be mapped within a spatial range, thereby extracting comprehensive prognostic factors from the numerical clinical information. For example, for different patients of the same age, after patient A's sample group is input into the preset model for training, the parameters in the updated parameter set are updated. Then, when patient B's sample group is input into the preset model for training, the age is vector-mapped based on the updated parameter set. Thus, different patients of the same age can have different corresponding feature vectors during training, but rather they will vary within a certain spatial range. Therefore, the impact of age on prognosis can be obtained in prognostic assessment.

[0171] In practice, for clinical sample information, in each iteration of training, the non-numerical sub-sample information in the current sample group can be converted into the first sub-feature vector; and based on the current value of each parameter in the parameter set, the numerical sub-sample information can be mapped to the target space to obtain the corresponding second sub-feature vector; the first sub-feature vector and the second sub-vector are fused to obtain the sample features corresponding to the clinical sample information.

[0172] In this embodiment, the parameters in the parameter set are used to determine the dimension of the target space and the value at each spatial point.

[0173] Non-numerical subsample information can refer to subsample information in string format or text type. For example, gender "male" is text type subsample information, and tumor grade is string type subsample information. For non-numerical subsample information, the feature vectors corresponding to these subsample information can be pre-set. Then, for the clinical sample information in the current sample group, the first sub-feature vector corresponding to the non-numerical subsample information in the current sample group can be obtained by looking up the table. The table includes feature vectors corresponding to various non-numerical subsample information, which can be understood as a fixed feature vector.

[0174] Among them, numerical subsample information can be numerical type subsample information. For example, the age "62" is numerical type subsample information. For this type of subsample information, the numerical subsample information can be mapped to the target space based on the current value of each parameter in the parameter set. The target space can be a vector space, which can be a two-dimensional space, including multiple values ​​in the first dimension and multiple values ​​in the second dimension. That is to say, a numerical value can be distributed to various positions in the target space to obtain the second sub-feature vector corresponding to the numerical value.

[0175] In this way, each parameter in the parameter set can be understood as a weight value that distributes the value to various locations in the target space.

[0176] The process of fusing the first sub-feature vector and the second sub-vector to obtain the sample features corresponding to the clinical sample information can be described in the example above. For example, the first sub-feature vector and the second sub-vector together constitute multiple sub-feature vectors, which can be fused in the manner described above for fusing multiple sub-feature vectors.

[0177] Specifically, the parameter set includes a first parameter vector, a second parameter vector, and a fourth parameter matrix. When mapping numerical subsample information to the target space based on the current values ​​of each parameter in the parameter set to obtain the corresponding second sub-feature vector, the numerical subsample information can be mapped to the first dimension in the target space based on the current value of the first parameter vector to obtain the mapping value of the first dimension. Based on the mapping value of the first dimension, the current value of the second parameter vector, and the current value of the fourth parameter matrix, the second sub-feature vector is determined.

[0178] The first parameter vector is used to determine the value of the spatial point in the first dimension of the target space; the second parameter vector is used to determine the value of the spatial point in the second dimension of the target space; and the fourth parameter matrix is ​​used to assign parameters to each spatial location in the first and second dimensions.

[0179] The first parameter vector can be understood as the weight value that distributes the numerical value to each position in the first dimension of the target space; the second parameter vector can be understood as the weight value that distributes the numerical value to each position in the second dimension of the target space; and the fourth parameter matrix can be understood as the weight value that distributes the numerical value to each position in the target space.

[0180] Specifically, according to the following formula (ix), based on the current value of the first parameter vector, the numerical subsample information can be mapped to the first dimension in the target space to obtain the mapping value of the first dimension:

[0181] temp_a = sigmoid(w × a) (Formula 9)

[0182] In formula (ix), w is the first parameter vector, which in practice can be a parameter vector of 128×1 dimensions, and a is the numerical subsample information, such as the age value "62".

[0183] Next, the second sub-feature vector can be determined according to the following formula (x), based on the first dimension mapping value, the current value of the second parameter vector, and the current value of the fourth parameter matrix:

[0184] sa=W×temp_a+b Formula (10)

[0185] In formula (x), b is the second parameter vector, which can be a parameter vector of 128×1 dimensions in practice; W is the fourth parameter matrix, which can be a parameter matrix of 128 dimensions in practice; and Sa is the second sub-feature vector.

[0186] Of course, in another embodiment, the parameter set also includes multiple third parameter vectors. After determining the second sub-feature vector, the second sub-feature vector can be modified. Specifically, the modification can be performed according to the following formula (XI):

[0187]

[0188] Where va is the corrected second sub-feature vector, sa is the second sub-feature vector, and a1, a2 and a3 are the third parameter vectors. Specifically, these three third parameter vectors can all be 128*1 parameter vectors, and differences are allowed between the three third parameter vectors.

[0189] Accordingly, the first sub-feature vector and the corrected second sub-vector can be fused to obtain the sample features corresponding to the clinical sample information.

[0190] In one optional example, when fusing sample features corresponding to sample information from multiple modalities, the individual sample features can be mapped to the same space before fusion. Specifically, the preset model parameters include a dimensionality parameter matrix corresponding to the sample information of each modality. Thus, based on the sample features and the corresponding dimensionality parameter matrix of the sample information for each modality, the dimensionality of the sample features corresponding to that modality can be transformed to obtain the transformed sample features. Based on the transformed sample features corresponding to the sample information from multiple modalities, the predicted prognostic evaluation value and the consistency expression value are determined.

[0191] Each dimension parameter matrix is ​​used to adjust the dimension of the sample features corresponding to the sample information of that modality. Specifically, the dimension of the sample features corresponding to the sample information of each modality can be adjusted according to the following formulas (12) to (14):

[0192] Formula (XII) = PV1 = M1·SV

[0193] Formula XIII: PV² = M²·CV

[0194] PV3 = M3·GV (Formula XIV)

[0195] Wherein, PV1 is the transformed sample feature corresponding to the MR sample image, PV2 is the transformed sample feature corresponding to the clinical sample information, and PV3 is the transformed sample feature corresponding to the gene sample information.

[0196] Wherein, SV is the sample feature corresponding to the MR sample image, CV is the sample feature corresponding to the clinical sample information, and GV is the sample feature corresponding to the gene sample information.

[0197] Where M1 is the dimensional parameter matrix corresponding to the MR sample image, which can be a 64×512 parameter matrix; M2 is the dimensional parameter matrix corresponding to the clinical sample information, which can be a 64×128 parameter matrix; and M3 is the dimensional parameter matrix corresponding to the gene sample information, which can be a 64×128 parameter matrix.

[0198] Phase 2: The parameter update phase.

[0199] In this disclosure, the parameters of the preset model need to be updated based on the predicted prognostic assessment value, the prognostic assessment label corresponding to the current sample group, and the consistency expression value. In specific implementation, the difference between the predicted prognostic assessment value and the prognostic assessment label can be obtained; the parameters of the preset model are updated with the goal of minimizing the difference and maximizing the consistency expression value.

[0200] The difference between the predicted prognostic value and the prognostic label can reflect the distance between the prognostic value predicted by the preset model and the actual prognostic value. The consistency expression value can characterize the degree of consistency of each sample feature corresponding to the same target disease. During the training process, the training objective can be to minimize the difference between the predicted prognostic value and the prognostic label and to maximize the degree of consistency.

[0201] In this way, the parameters of the preset model can be updated with the goal of minimizing differences and maximizing consistent expression values.

[0202] In practical implementation, when updating the parameters of the preset model with the goal of minimizing the difference and maximizing the consistency expression value, the loss function shown in the following formula (XV) can be constructed based on the difference and the consistency expression value:

[0203] loss = ∑ i (y′ i -y i ) 2 -consistency formula (15)

[0204] Based on the loss value of the loss function, the parameters of the preset model are updated with the goal of minimizing the difference and maximizing the consistent expression value; wherein, the loss represents the loss value, y' i Indicates the predicted prognostic evaluation value, y i The prognostic assessment label and consistency represent the consistency expression value.

[0205] As can be seen from the above loss function, the training objective is to minimize the loss value. In order to minimize the loss value, it is necessary to minimize the difference and maximize the consistency expression value. It should be noted that the consistency expression value disclosed herein can be a value between 0 and 1.

[0206] In one example, the process of determining the consistent expression value can be as follows:

[0207] First, each sample feature is transposed to obtain the transposed feature corresponding to each sample feature; then, for two different sample features, the transposed feature corresponding to one sample feature is fused with the transposed feature corresponding to the other sample feature to obtain the corresponding fused feature value; based on each fused feature value, the consistency expression value is determined.

[0208] In one embodiment, the transpose features corresponding to each sample feature can be determined based on the following formula (xvii):

[0209] PV T ·PV=1 Formula (XVI);

[0210] In formula (xvii), PV represents the sample feature. T It is a transposed feature, and the dot product between the two is 1; that is, by transposing, the sample features can be normalized.

[0211] Next, the transpose of one sample feature and the transpose of another sample feature can be fused according to the following formula (XVII) to obtain the corresponding fused feature value:

[0212]

[0213] In formula (18), PV1 is the transpose of sample feature 1, PV2 is sample feature 2, and the dot product between the two results in a feature between 0 and 1.

[0214] Taking MR sample information, clinical sample information, and gene sample information as an example, the consistency expression value can be determined according to formula (XVIII):

[0215]

[0216] Wherein, PV1 represents the sample features of the MR sample image, PV2 represents the sample features of the clinical sample information, and PV3 represents the sample features of the genetic sample information. These are the transpose features corresponding to the sample features of the MR sample image. These are the transposed features corresponding to the sample features of clinical sample information. It is the transposed feature corresponding to the sample features of gene sample information.

[0217] Reference Figure 2 As shown, a schematic diagram of a pre-defined model of this disclosure is illustrated. Figure 2 As shown, it may include a fusion module, a prediction branch, a consistency expression branch, and a data processing module corresponding to the sample information of each modality.

[0218] Specifically, when using a preset model to extract features from sample information of multiple modalities in the current sample group, and determining the predicted prognostic evaluation value and consistency expression value based on the extracted sample features, the process can involve extracting features from the sample information of each modality in the current sample group using the data processing module corresponding to that modality's sample information; fusing the sample features output by each data processing module using a fusion module; then, using a prediction branch to determine the predicted prognostic evaluation value corresponding to the features output by the fusion module; and finally, using a consistency expression branch to determine the consistency expression value corresponding to the sample features output by each data processing module.

[0219] In other words, the sample information of each modality disclosed herein is input into the corresponding data processing module, which extracts features from the sample information of that modality. Then, the extracted sample features are input into the fusion module, which fuses the various sample features. Finally, the fused sample features are input into the prediction module, which determines the predicted prognosis value based on the fused sample features.

[0220] Below, in conjunction with Figure 2 The preset model shown serves as an example to illustrate the method for obtaining the target model of this disclosure:

[0221] S1. Prepare training samples. The training samples include multiple sample groups corresponding to each sample user. Each sample group includes the sample user's MR sample image, clinical sample information, and gene sample information.

[0222] S2, input multiple sample groups into the preset model, wherein MR sample images in the sample groups are input into the image data processing module, clinical sample information is input into the clinical data processing module, and gene sample information is input into the gene data processing module, so as to perform feature extraction of sample information of multiple modalities and obtain sample features corresponding to sample information of each modality.

[0223] Specifically, the image data processing module is used to extract features from multiple slice sub-sample images in the MR sample image. After extraction, the sub-feature vectors corresponding to each slice sub-sample image are fused to obtain the sample features corresponding to the MR sample image.

[0224] Reference Figure 3 and Figure 4 As shown, Figure 3 A schematic diagram of the image data processing module is shown. Figure 4 It shows Figure 3 A schematic diagram of the ResNet network structure.

[0225] like Figure 3As shown, it includes multiple slice sample images, such as Slice1-Slice n. Each slice sample image is input into the corresponding ResNet network, and the ResNet network extracts features from the slice sample image.

[0226] Among them, such as Figure 4 As shown, in this example, each input slice sample image is 256×256 in size. It first passes through a 7*7 convolutional kernel with a stride of 2 and a 3*3 max pooling step with a stride of 2. This transforms the 256*256 input slice sample image into a 64*64 feature map, effectively reducing the storage size required. Then, it sequentially enters multiple ResNet_Block and downsampling modules. Specifically, it sequentially enters network layer 1 consisting of 3 ResNet_Blocks and a downsampling module; network layer 2 consisting of 3 ResNet_Blocks and 1 downsampling module; network layer 3 consisting of 5 ResNet_Blocks and 1 downsampling module; network layer 4 consisting of 2 ResNet_Blocks; followed by an average pooling layer. Finally, it outputs a sub-feature vector for each slice sample image, which can be a 512×1 vector.

[0227] Next, each sub-feature vector enters the self-attention layer, which determines the attention value between every two sub-feature vectors. Then, the attention value between every two sub-feature vectors is input to the feature representation layer. Based on the attention value, the feature representation layer obtains the sample features corresponding to the MR sample image according to the above formulas (iv) and (v).

[0228] Among them, such as Figure 4 As shown on the right, each ResNet_Block consists of, in sequence, a 3×3 convolution with a stride of 1, Batch Norm regularization, ReLU activation, another 3×3 convolution with a stride of 1, and Batch Norm regularization. The downsampling module has a similar structure to ResNet_Block, but uses a 3×3 convolution with a stride of 2, Batch Norm regularization, ReLU activation, a 3×3 convolution with a stride of 1, Batch Norm regularization, and downsampling (a 1×1 convolution with a stride of 2 and Batch Norm regularization).

[0229] Specifically, the clinical data processing module is used to perform feature transformation on multiple types of clinical information in the clinical sample information to obtain the sample features corresponding to the clinical sample information.

[0230] This example uses gender, age, histological diagnosis, tumor grade, medication information, and history of malignant tumors as clinical sample information, and then maps this information into sub-feature vectors. Character-type sub-sample information such as gender, histological diagnosis, tumor grade, medication information, and history of malignant tumors can be represented by sub-feature vectors using vg, vh, vgr, vd, and vhom. Specifically, a lookup table method can be used to complete the vector mapping of character-type sub-sample information.

[0231] Finally, formula (VII) is used to fuse the various sub-feature vectors to obtain the sample features corresponding to the clinical sample information.

[0232] For numerical subsample information such as age, the sub-feature vector corresponding to the numerical subsample information can be obtained by formulas (ix), (x), and (xi).

[0233] Specifically, the gene data processing module is used to perform feature transformation on information of multiple genes in the gene sample information to obtain sample features corresponding to the gene sample information.

[0234] Similar to the character-type subsample information in clinical sample information, a lookup table method can be used to complete the vector mapping of information for each gene, obtain the sub-feature vector corresponding to the information of each gene, and then use formula (VII) to fuse the sub-feature vectors to obtain the sample features corresponding to the gene sample information.

[0235] S3, input the sample features corresponding to the sample information of multiple modalities into the fusion module. The fusion module can use formulas (12) to (14) to fuse the sample features of each sample, and obtain the consistency expression value corresponding to the sample group input into the preset model based on formulas (16) to (18).

[0236] S4. Input the fused sample features output by the fusion module into the prediction module, and the prediction module determines the predicted prognosis value based on the fused sample features.

[0237] S5. Based on the consistent expression value output by the fusion module, the predicted prognostic evaluation value output by the prediction module, and the prognostic evaluation label corresponding to the sample group input into the preset model, construct the loss function shown in formula (xv) to update the parameters of the preset model with the goal of minimizing the difference and maximizing the consistent expression value.

[0238] When updating the parameters of the preset model, the first parameter matrix, the second parameter matrix, the third parameter matrix, the fourth parameter matrix, as well as the first parameter vector, the second parameter vector, and the three third parameter vectors in the parameter set can be updated synchronously. This allows the accuracy of the data processing module, the fusion module, and the prediction module of the three modalities to be affected simultaneously in one training session based on two optimization objectives.

[0239] S6. The preset model after multiple updates, or when the difference between the predicted prognostic assessment value and the prognostic assessment label is less than the preset difference, and the consistency expression value is higher than or equal to the preset expression value, is stopped. The preset model at the time of stopping training is used as the target model, and the patient's prognostic assessment value can be predicted using the target model.

[0240] Accordingly, refer to Figure 5 As shown, a flowchart illustrating the steps of a method for determining prognostic assessment values ​​according to this disclosure is presented, as follows: Figure 5 As shown, the specific steps may include:

[0241] Step S501: Obtain information on multiple modalities of the subject to be tested, including MRI images, clinical information, and genetic information;

[0242] Step S502: Input the information of the multiple modalities into the target model to obtain the prognostic evaluation value of the object to be tested; wherein, the target model is obtained by the target model acquisition method described in the above embodiment.

[0243] In this embodiment, after obtaining the target model, the prognostic assessment value of the patient can be predicted using the target model. In practice, information on multiple modalities of the subject to be tested can be obtained. The information on multiple modalities of the subject to be tested can be consistent with the modalities used in training the preset model. For example, if the sample group used in training the preset model includes MR sample images, clinical sample information and gene sample information, then the information on multiple modalities of the subject to be tested can also include the MR images, clinical information and gene information of the subject to be tested.

[0244] Among them, MR images for multiple modalities of the subject to be tested can include multiple slice images, and clinical information can also include multiple sub-clinical information. Similarly, genetic information can also include information on multiple genes.

[0245] Because the target model uses the consistency expression value as the basis for updating the parameters of the preset model during training, and the consistency expression value is used to characterize the degree of consistency of the features of each sample corresponding to the same target disease, as the training progresses, the preset model can extract prognostic factors (sample features) related to the target disease and gradually discard prognostic factors that are not related to the target disease from the information of various modalities. This makes the prognostic factors selected by the model clinically important, which can help improve the interpretability of the model. As a result, the output of the target model has high prognostic reference value.

[0246] Of course, in some alternative examples, since the target model includes a data processing module corresponding to the sample information of each modality, and a fusion module connected to multiple data processing modules, and since the data processing module and the fusion module extract sample features that are strongly correlated with the expression of the target disease from the sample information of multiple modalities, in one application, after obtaining the target model, the data processing module and the fusion module in the target model can be extracted separately as a feature extraction model. This feature extraction model can be used to extract features that are strongly correlated with the target disease from the information of multiple modalities, and thus can be independently applied to the screening of prognostic factors in the prognostic process.

[0247] The technical solution adopted in this disclosure has the following advantages:

[0248] First, by using consistent expression values, information from different modalities can be mapped to the space of important expression information of the target disease, making the features extracted from different modalities more similar in this space. This can improve the complementarity of information from different modalities, reduce the influence of noise (unimportant information), and enhance the clinical importance of the target disease. As a result, the target model has medical interpretability, and its predicted prognostic assessment value has high medical reference value.

[0249] Second, information from different modalities can complement each other, thereby enriching the data sources for prognostic factors and interpreting the expression of the target disease from multiple dimensions, thus further improving the medical reference value of the prognostic assessment value.

[0250] Third, by combining information within the same modality through a self-attention mechanism, the nonlinear representation capability of the target model can be enhanced.

[0251] Fourth, it eliminates the need for users to manually screen prognostic factors multiple times, improving the efficiency of prognostic factor determination and reducing labor costs.

[0252] Based on the same inventive concept, this disclosure also provides a device for acquiring a target model, referring to... Figure 6 As shown, a schematic diagram of the structure of the target model acquisition device is illustrated. Figure 6As shown, it can specifically include the following modules:

[0253] The sample acquisition module 601 is used to acquire sample groups corresponding to multiple sample users. The sample groups include sample information of multiple modalities, including at least two of nuclear magnetic resonance (MRI) sample images, clinical sample information, and gene sample information.

[0254] Training module 602 is used to iteratively train a preset model based on multiple sample groups to obtain the target model, which is used to predict the prognostic evaluation value of the target object; wherein, the following steps are performed in each iteration of training:

[0255] Using the preset model, feature extraction is performed on sample information of multiple modalities in the current sample group, and based on the extracted sample features, the predicted prognostic assessment value and the consistency expression value are determined; wherein, the consistency expression value is used to characterize the degree of consistency of each sample feature corresponding to the same target disease;

[0256] The parameters of the preset model are updated based on the predicted prognostic evaluation value, the prognostic evaluation label corresponding to the current sample group, and the consistency expression value.

[0257] Optionally, the training module 602 includes a parameter update unit, which includes:

[0258] The difference determination subunit is used to obtain the difference between the predicted prognostic assessment value and the prognostic assessment label;

[0259] The parameter update subunit is used to update the parameters of the preset model with the goal of minimizing the differences and maximizing the consistent expression value.

[0260] Optionally, the parameter updating subunit is specifically used for:

[0261] Based on the aforementioned differences and the aforementioned consistent expression values, the following loss function is constructed:

[0262] loss = ∑ i (y′ i -y i ) 2 -consistency

[0263] Based on the loss value of the loss function, the parameters of the preset model are updated with the goal of minimizing the difference and maximizing the consistent expression value.

[0264] Wherein, loss represents the loss value, predictive prognostic evaluation value, prognostic evaluation label, and consistency represents the consistency expression value.

[0265] Optionally, the step of determining the consistency expression value based on the extracted features of each sample includes:

[0266] Transpose each of the sample features to obtain the transposed feature corresponding to each of the sample features;

[0267] For two different sample features, the transpose of one of the sample features is fused with the transpose of the other sample feature to obtain the corresponding fused feature value;

[0268] The consistency expression value is determined based on each of the fusion feature values.

[0269] Optionally, the sample information for each modality includes multiple sub-sample information. The step of using the preset model to extract features from the sample information of multiple modalities in the current sample group includes the following steps:

[0270] For each modality of sample information, feature extraction is performed on multiple sub-sample information in the sample information of that modality to obtain multiple corresponding sub-feature vectors;

[0271] For each modality, multiple sub-feature vectors corresponding to the sample information are fused to obtain the sample features corresponding to that modality.

[0272] Optionally, the step of fusing multiple sub-feature vectors corresponding to the sample information of each modality to obtain the sample features corresponding to the sample information of that modality includes:

[0273] Determine the attention value between every two sub-feature vectors, the attention value being used to characterize the tightness between the two sub-feature vectors;

[0274] Based on the attention value, multiple sub-feature vectors are fused to obtain the sample features corresponding to the sample information of the modality.

[0275] Optionally, the step of fusing multiple sub-feature vectors based on the attention value to obtain the sample features corresponding to the sample information of the modality includes:

[0276] For each sub-feature vector, based on the attention value between the sub-feature vector and all other sub-feature vectors, all other sub-feature vectors are fused into the sub-feature vector to obtain the fused sub-vector of the sub-feature vector;

[0277] The multiple fused sub-vectors are then fused again to obtain the sample features corresponding to the sample information of the modality.

[0278] Optionally, the parameters of the preset model include a first parameter matrix, the MR sample images include multiple slice sample images, and the step of determining the attention value between every two sub-feature vectors includes:

[0279] For each slice sample image included in the MR sample image, based on the current value of the first parameter matrix, an attention value is determined between the sub-feature vectors corresponding to every two slice sample images.

[0280] Optionally, the parameters of the preset model include a second parameter matrix and a third parameter matrix. The step of fusing multiple sub-feature vectors corresponding to the sample information of each modality to obtain the sample features corresponding to that modality includes:

[0281] For each clinical subsample information in the clinical sample information, based on the current value of the second parameter matrix, feature fusion is performed on the sub-feature vectors corresponding to each clinical subsample information to obtain the sample features corresponding to the clinical sample information;

[0282] For each gene subsample in the gene sample information, based on the current value of the third parameter matrix, feature fusion is performed on the sub-feature vectors corresponding to each gene subsample to obtain the sample features corresponding to the gene sample information.

[0283] Optionally, the parameters of the preset model include a parameter set corresponding to the clinical sample information, wherein the clinical sample information includes numerical sub-sample information and non-numerical sub-sample information, and the step of using the preset model to extract features from sample information of multiple modalities in the current sample group includes:

[0284] For the clinical sample information in the current sample group, the non-numerical sub-sample information is converted into a first sub-feature vector;

[0285] Based on the current values ​​of each parameter in the parameter set, the numerical subsample information is mapped to the target space to obtain the corresponding second sub-feature vector; the parameters in the parameter set are used to determine the dimension of the target space and the value at each spatial point.

[0286] The first sub-feature vector and the second sub-vector are fused to obtain the sample features corresponding to the clinical sample information.

[0287] Optionally, the parameter set includes a first parameter vector, a second parameter vector, and a fourth parameter matrix. The step of mapping the numerical subsample information to the target space based on the current values ​​of each parameter in the parameter set to obtain the corresponding second sub-feature vector includes:

[0288] Based on the current value of the first parameter vector, the numerical subsample information is mapped to the first dimension in the target space to obtain the mapping value of the first dimension; wherein, the first parameter vector is used to determine the value of the spatial point of the target space in the first dimension;

[0289] Based on the first dimension mapping value, the current value of the second parameter vector, and the current value of the fourth parameter matrix, the second sub-feature vector is determined; wherein, the second parameter vector is used to determine the value of the spatial point of the target space in the second dimension, and the fourth parameter matrix is ​​used to assign parameters to each spatial location in the first and second dimensions.

[0290] Optionally, the parameter set further includes multiple third parameter vectors, and the device further includes:

[0291] The correction module is used to correct the second sub-feature vector based on the second sub-feature vector and multiple third parameter vectors according to the following formula:

[0292]

[0293] Where va is the corrected second sub-feature vector, sa is the second sub-feature vector, and a1, a2 and a3 are the third parameter vectors respectively;

[0294] The step of fusing the first sub-feature vector and the second sub-vector to obtain the sample features corresponding to the clinical sample information includes:

[0295] The first sub-feature vector and the corrected second sub-vector are fused to obtain the sample features corresponding to the clinical sample information.

[0296] Optionally, the parameters of the preset model include a dimensional parameter matrix corresponding to the sample information of each modality, and the step of determining the predicted prognostic evaluation value and the consistency expression value based on the extracted sample features includes:

[0297] Based on the sample features corresponding to the sample information of each modality and the dimensional parameter matrix, the dimensional transformation is performed on the sample features corresponding to the sample information of that modality to obtain the transformed sample features.

[0298] Based on the transformed sample features corresponding to sample information from multiple modalities, the predicted prognostic evaluation value and the consistency expression value are determined.

[0299] Optionally, the preset model includes a fusion module, a prediction branch, a consistency expression branch, and a data processing module corresponding to the sample information of each modality; the step of using the preset model to extract features from the sample information of multiple modalities in the current sample group, and determining the predicted prognosis evaluation value and the consistency expression value based on the extracted sample features, includes:

[0300] For each modality in the current sample group, the data processing module corresponding to the sample information of that modality is used to extract features from the corresponding sample information;

[0301] The fusion module is used to fuse the sample features output by each of the data processing modules.

[0302] Using the prediction branch, the predicted prognostic evaluation value corresponding to the feature output by the fusion module is determined;

[0303] Using the consistency expression branch, the consistency expression value corresponding to the sample features output by each of the data processing modules is determined.

[0304] Optionally, the gene sample information includes information on at least one of the following genes: isocitrate dehydrogenase, chromosome 1p / 19q combined deletion state, telomerase reverse transcriptase gene promoter, and O6-methylguanine-DNA methyltransferase promoter region methylation.

[0305] The clinical sample information includes at least one of the following: gender, age, histological diagnosis, tumor grade, medication information, and history of malignant tumors.

[0306] Reference Figure 7 As shown, a schematic diagram of a prognostic assessment value determination device is illustrated. Figure 7 As shown, the device includes:

[0307] The information acquisition module 701 is used to acquire information on multiple modalities of the test object, including MRI images, clinical information, and genetic information.

[0308] The input module 702 is used to input the information of the multiple modalities into the target model to obtain the prognostic evaluation value of the object to be tested; wherein the target model is obtained according to the target model acquisition method.

[0309] Based on the same inventive concept, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing, implements the target model acquisition method, or implements the prognostic evaluation value determination method.

[0310] Reference Figure 8 As shown, a structural block diagram of an electronic device 800 according to an embodiment of the present disclosure is illustrated. Figure 8 As shown in the embodiment of the present invention, an electronic device 800 is provided, which can be used to execute a target model acquisition method or a prognostic evaluation value determination method.

[0311] The electronic device 800 may include a memory 801, a processor 802, and a computer program stored in the memory and executable on the processor, wherein the processor 802 is configured to perform the image processing method described above.

[0312] like Figure 8 As shown, in one embodiment, the electronic device 800 may completely include an input device 803, an output device 804, and a data acquisition device 805. When executing the image processing method of this embodiment, the data acquisition device 805 can acquire information of multiple modalities. Then, the input device 803 can acquire the information of multiple modalities from the data acquisition device 805. The information of multiple modalities can be processed by the processor 802. Specifically, the processing can execute the above-described method for acquiring the target model and the above-described method for determining the prognostic evaluation value. The output device 804 can output the target model or the prognostic evaluation value result output by the target model.

[0313] Of course, in one embodiment, memory 801 may include volatile memory and non-volatile memory. Volatile memory can be understood as random access memory used to store and save data. Non-volatile memory refers to computer memory whose stored data will not disappear when the current is turned off. Of course, the computer program for the method of obtaining the target model or the method of determining the prognostic evaluation value disclosed herein may be stored in volatile memory and non-volatile memory, or exist in either one.

[0314] Based on the same inventive concept, this disclosure also provides a computer-readable storage medium storing a computer program that causes a processor to execute the target model acquisition method, or to implement the prognostic evaluation value determination method during execution.

[0315] Based on the same inventive concept, this disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the target model acquisition method, or, when executed, implements the prognostic evaluation value determination method.

[0316] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0317] Finally, it should be noted that, unless otherwise defined, the terms "first," "second," and similar terms used herein do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Terms such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0318] The above provides a detailed description of the target model acquisition method, prognostic evaluation value determination method, apparatus, equipment, and medium provided by this disclosure. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

[0319] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0320] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0321] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0322] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0323] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This disclosure can be implemented by means of hardware comprising a plurality of different elements and by means of a suitably programmed computer. In a unit claim enumerating a plurality of means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0324] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A method for obtaining a target model, characterized in that, The method includes: Obtain sample groups corresponding to multiple sample users, wherein the sample groups include sample information of multiple modalities, and the sample information of multiple modalities includes at least two of MRI sample images, clinical sample information and gene sample information; Based on multiple sample groups, the preset model is iteratively trained to obtain the target model, which is used to predict the prognostic evaluation value of the target object. In each iteration of training, the following steps are performed: Using the preset model, feature extraction is performed on sample information of multiple modalities in the current sample group, and based on the extracted sample features, the predicted prognostic assessment value and the consistency expression value are determined; wherein, the consistency expression value is used to characterize the degree of consistency of each sample feature corresponding to the same target disease; Based on the predicted prognostic evaluation value, the prognostic evaluation label corresponding to the current sample group, and the consistency expression value, the parameters of the preset model are updated; The step of updating the parameters of the preset model based on the predicted prognostic evaluation value, the prognostic evaluation label corresponding to the current sample group, and the consistency expression value includes: Obtain the difference between the predicted prognostic value and the prognostic label; The parameters of the preset model are updated with the goal of minimizing the differences and maximizing the consistent expression value. The step of updating the parameters of the preset model with the goal of minimizing the differences and maximizing the consistent expression value includes: Based on the differences and the consistent expression values, the following loss function is constructed: -consistency ; Based on the loss value of the loss function, the parameters of the preset model are updated with the goal of minimizing the difference and maximizing the consistent expression value. Wherein, loss represents the loss value, Indicates the predicted prognostic assessment value, The prognostic assessment label is indicated by consistency, which represents the consistency expression value.

2. The method according to claim 1, characterized in that, The determination of consistency expression values ​​based on the extracted features of each sample includes: Transpose each of the sample features to obtain the transposed feature corresponding to each of the sample features; For two different sample features, the transpose of one of the sample features is fused with the transpose of the other sample feature to obtain the corresponding fused feature value; The consistency expression value is determined based on each of the fusion feature values.

3. The method according to claim 1, characterized in that, Each modality's sample information includes multiple sub-sample information. The step of using the preset model to extract features from the sample information of multiple modalities in the current sample group includes: For each modality of sample information, feature extraction is performed on multiple sub-sample information in the sample information of that modality to obtain multiple corresponding sub-feature vectors; For each modality, multiple sub-feature vectors corresponding to the sample information are fused to obtain the sample features corresponding to that modality.

4. The method according to claim 3, characterized in that, The step of fusing multiple sub-feature vectors corresponding to the sample information of each modality to obtain the sample features corresponding to the sample information of that modality includes: Determine the attention value between every two sub-feature vectors, the attention value being used to characterize the tightness between the two sub-feature vectors; Based on the attention value, multiple sub-feature vectors are fused to obtain the sample features corresponding to the sample information of the modality.

5. The method according to claim 4, characterized in that, The process of fusing multiple sub-feature vectors based on the attention value to obtain sample features corresponding to the sample information of the modality includes: For each sub-feature vector, based on the attention value between the sub-feature vector and all other sub-feature vectors, all other sub-feature vectors are fused into the sub-feature vector to obtain the fused sub-vector of the sub-feature vector; The multiple fused sub-vectors are then fused again to obtain the sample features corresponding to the sample information of the modality.

6. The method according to claim 4, characterized in that, The parameters of the preset model include a first parameter matrix, the MR sample images include multiple slice sample images, and determining the attention value between every two sub-feature vectors includes: For each slice sample image included in the MR sample image, based on the current value of the first parameter matrix, an attention value is determined between the sub-feature vectors corresponding to every two slice sample images.

7. The method according to any one of claims 3-6, characterized in that, The parameters of the preset model include a second parameter matrix and a third parameter matrix. The feature fusion of multiple sub-feature vectors corresponding to the sample information of each modality to obtain the sample features corresponding to that modality includes: For each clinical subsample information in the clinical sample information, based on the current value of the second parameter matrix, feature fusion is performed on the sub-feature vectors corresponding to each clinical subsample information to obtain the sample features corresponding to the clinical sample information; For each gene subsample in the gene sample information, based on the current value of the third parameter matrix, feature fusion is performed on the sub-feature vectors corresponding to each gene subsample to obtain the sample features corresponding to the gene sample information.

8. The method according to any one of claims 1-6, characterized in that, The parameters of the preset model include a parameter set corresponding to the clinical sample information, which includes numerical sub-sample information and non-numerical sub-sample information. The step of using the preset model to extract features from sample information of multiple modalities in the current sample group includes: For the clinical sample information in the current sample group, the non-numerical sub-sample information is converted into a first sub-feature vector; Based on the current values ​​of each parameter in the parameter set, the numerical subsample information is mapped to the target space to obtain the corresponding second sub-feature vector; the parameters in the parameter set are used to determine the dimension of the target space and the value at each spatial point. The first sub-feature vector and the second sub-feature vector are fused to obtain the sample features corresponding to the clinical sample information.

9. The method according to claim 8, characterized in that, The parameter set includes a first parameter vector, a second parameter vector, and a fourth parameter matrix. The step of mapping the numerical sub-sample information to the target space based on the current values ​​of each parameter in the parameter set to obtain the corresponding second sub-feature vector includes: Based on the current value of the first parameter vector, the numerical subsample information is mapped to the first dimension in the target space to obtain the mapping value of the first dimension; wherein, the first parameter vector is used to determine the value of the spatial point of the target space in the first dimension; Based on the first dimension mapping value, the current value of the second parameter vector, and the current value of the fourth parameter matrix, the second sub-feature vector is determined; wherein, the second parameter vector is used to determine the value of the spatial point of the target space in the second dimension, and the fourth parameter matrix is ​​used to assign parameters to each spatial location in the first and second dimensions.

10. The method according to claim 9, characterized in that, The parameter set further includes multiple third parameter vectors. After determining the second sub-feature vector based on the first dimension mapping value, the current value of the second parameter vector, and the current value of the fourth parameter matrix, the method further includes: Based on the second sub-feature vector and the plurality of third parameter vectors, the second sub-feature vector is modified according to the following formula: ； Where va is the corrected second sub-feature vector, sa is the second sub-feature vector, and a1, a2 and a3 are the third parameter vectors respectively; The first sub-feature vector and the second sub-feature vector are fused to obtain the sample features corresponding to the clinical sample information, including: The first sub-feature vector and the corrected second sub-feature vector are fused to obtain the sample features corresponding to the clinical sample information.

11. The method according to claim 1, characterized in that, The parameters of the preset model include a dimensional parameter matrix corresponding to the sample information of each modality. The determination of the predicted prognostic evaluation value and the consistency expression value based on the extracted sample features includes: Based on the sample features corresponding to the sample information of each modality and the dimensional parameter matrix, the dimensional transformation is performed on the sample features corresponding to the sample information of that modality to obtain the transformed sample features. Based on the transformed sample features corresponding to sample information from multiple modalities, the predicted prognostic evaluation value and the consistency expression value are determined.

12. The method according to claim 1, characterized in that, The preset model includes a fusion module, a prediction branch, a consistency expression branch, and a data processing module corresponding to the sample information of each modality. The step of using the preset model to extract features from the sample information of multiple modalities in the current sample group, and determining the predicted prognosis evaluation value and the consistency expression value based on the extracted sample features, includes: For each modality in the current sample group, the data processing module corresponding to the sample information of that modality is used to extract features from the corresponding sample information; The fusion module is used to fuse the sample features output by each of the data processing modules. Using the prediction branch, the predicted prognostic evaluation value corresponding to the feature output by the fusion module is determined; Using the consistency expression branch, the consistency expression value corresponding to the sample features output by each of the data processing modules is determined.

13. A method for determining prognostic assessment values, characterized in that, The method includes: Information on multiple modalities of the subject to be tested is obtained, including MRI images, clinical information, and genetic information. The information from the various modalities is input into the target model to obtain the prognostic evaluation value of the object under test; The target model is obtained by the method according to any one of claims 1-12.

14. A device for acquiring a target model, characterized in that, The device includes: The sample acquisition module is used to acquire sample groups corresponding to multiple sample users. The sample groups include sample information of multiple modalities, including at least two of MRI sample images, clinical sample information, and gene sample information. The training module is used to iteratively train the preset model based on multiple sample groups to obtain the target model, which is used to predict the prognostic evaluation value of the target object. In each iteration of training, the following steps are performed: Using the preset model, feature extraction is performed on sample information of multiple modalities in the current sample group, and based on the extracted sample features, the predicted prognostic assessment value and the consistency expression value are determined; wherein, the consistency expression value is used to characterize the degree of consistency of each sample feature corresponding to the same target disease; Based on the predicted prognostic evaluation value, the prognostic evaluation label corresponding to the current sample group, and the consistency expression value, the parameters of the preset model are updated; The training module includes a parameter update unit, which includes: The difference determination subunit is used to obtain the difference between the predicted prognostic assessment value and the prognostic assessment label; The parameter update subunit is used to update the parameters of the preset model with the goal of minimizing the difference and maximizing the consistent expression value. The parameter update subunit is used for: Based on the differences and the consistent expression values, the following loss function is constructed: -consistency Based on the loss value of the loss function, the parameters of the preset model are updated with the goal of minimizing the difference and maximizing the consistent expression value. Wherein, loss represents the loss value, Indicates the predicted prognostic assessment value, The prognostic assessment label is indicated by consistency, which represents the consistency expression value.

15. A device for determining prognostic assessment values, characterized in that, The device includes: The information acquisition module is used to acquire information on multiple modalities of the test object, including MRI images, clinical information, and genetic information. The input module is used to input the information of the multiple modalities into the target model to obtain the prognostic evaluation value of the object under test; The target model is obtained by the method according to any one of claims 1-12.

16. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the target model telephone acquisition method as described in any one of claims 1-12, or implements the prognostic evaluation value determination method as described in claim 13.

17. A computer-readable storage medium, characterized in that, The stored computer program causes the processor to execute the method for obtaining the target model as described in any one of claims 1-12, or to implement the method for determining the prognostic evaluation value as described in claim 13.

Images