Disease prediction model training method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a disease prediction model training method and device, electronic equipment and a medium. The method comprises the steps that a model training sample is acquired, the model training sample is multi-modal data formed by a fundus image of a user and non-invasive user data, associated with a preset disease, of the user, and the model training sample is marked with a real disease category; training a to-be-trained disease prediction model based on the multi-modal data to obtain a predicted disease category output by the to-be-trained disease prediction model; calculating a loss value of the to-be-trained disease prediction model based on the real disease category, the predicted disease category and a weight corresponding to the model training sample; and under the condition that the loss value is within a preset range, taking the trained to-be-trained disease prediction model as a final disease prediction model for predicting the preset disease. According to the embodiment of the invention, the efficiency and accuracy of preset disease prediction can be improved.

Description

Technical Field

[0001] The present application relates to the field of equipment control technology, and in particular to a disease prediction model training method, device, electronic equipment and medium. Background Art

[0002] Hepatobiliary diseases, gout and kidney stones are currently common and serious diseases. Hepatobiliary diseases include diseases that are mainly treated by internal medicine and diseases that are mainly treated by surgery. Related diseases include both hereditary diseases and infectious diseases, and there may also be space-occupying lesions. Gout is closely related to lifestyle habits. Its main manifestation is high uric acid levels. As a metabolic disease, gout can affect multiple systems and cause a variety of complications. Some studies have shown that elevated uric acid levels can increase the risk of cardiovascular disease, including changes in heart structure, myocardial function and vascular structure and function. Kidney stones are a common disease in nephrology. Widely prevalent worldwide, the incidence of kidney stones has increased from 3.8% (1970) to 8.8% (2010).

[0003] At present, when predicting diseases such as hepatobiliary diseases, gout and kidney stones, doctors and other professionals are usually required to make predictions based on medical imaging data, which results in low disease prediction efficiency. In addition, doctors are unable to effectively predict the early stages of hepatobiliary diseases, gout and kidney stones, resulting in a relatively low accuracy rate in disease prediction. Summary of the invention

[0004] The embodiments of the present application provide a disease prediction model training method, device, electronic device and medium to solve the problems of prediction efficiency and accuracy of hepatobiliary diseases, gout and kidney stones in the prior art.

[0005] In order to solve the above technical problems, the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a disease prediction model training method, the method comprising:

[0007] Acquire a model training sample, where the model training sample is multimodal data formed by a fundus image of a user and non-invasive user data of the user associated with a preset disease, and the model training sample is annotated with a real disease category;

[0008] Training the disease prediction model to be trained based on the multimodal data, to obtain a predicted disease category output by the disease prediction model to be trained;

[0009] Based on the actual disease category, the predicted disease category and the weights corresponding to the model training samples, a loss value of the disease prediction model to be trained is calculated;

[0010] When the loss value is within a preset range, the trained disease prediction model to be trained is used as the final disease prediction model for predicting the preset disease.

[0011] Optionally, the training of the disease prediction model to be trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained includes:

[0012] The disease prediction model to be trained is trained using a five-fold cross-validation method and the multimodal data to obtain a predicted disease category output by the disease prediction model to be trained.

[0013] Optionally, the disease prediction model to be trained includes: N base learners and a regression meta-learner, wherein the N base learners are connected in parallel and are all connected to the regression meta-learner, and N is a positive integer greater than or equal to 2.

[0014] The step of training the disease prediction model to be trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained includes:

[0015] Calling N base learners to process the multimodal data respectively, and obtaining initial feature information associated with the preset disease corresponding to the multimodal data output by the N base learners;

[0016] Processing the initial feature information based on a weighted voting method to obtain target feature information;

[0017] The regression meta-learner is called to process the target feature information, and output the predicted disease category.

[0018] Optionally, the disease prediction model to be trained further includes: a self-attention mechanism layer,

[0019] The calling of the N base learners to process the multimodal data respectively to obtain initial feature information associated with the preset disease corresponding to the multimodal data output by the N base learners includes:

[0020] Calling the self-attention mechanism layer to perform attention mechanism learning on the fundus image to obtain an attention feature map corresponding to the fundus image;

[0021] N base learners are called to process the attention feature map and the non-invasive user data respectively, so as to obtain initial feature information associated with the preset disease output by the N base learners.

[0022] Optionally, the calculating the loss value of the disease prediction model to be trained based on the real disease category, the predicted disease category and the weights corresponding to the model training samples includes:

[0023] The weight of the model training samples of the target category is calculated based on the total number of samples, the total number of categories and the number of samples of the target category in this model training;

[0024] Based on the weight, the true disease category and the predicted disease category, a loss value of the disease prediction model to be trained is calculated.

[0025] Optionally, the non-invasive user data includes: daily alcohol consumption, user taste data, body mass index, daily meat consumption and daily cereal food consumption.

[0026] Optionally, the preset diseases include: any one of liver and gallbladder diseases, kidney stones, and gout.

[0027] Optionally, after using the trained disease prediction model to be trained as the final disease prediction model for predicting the preset disease, the method further includes:

[0028] Acquire target multimodal data of a target user, wherein the target multimodal data is data formed by a target fundus image of the target user and non-invasive data of the target user associated with a preset disease;

[0029] Inputting the target multimodal data into the disease prediction model;

[0030] Calling N base learners to process the target multimodal data respectively to obtain feature information associated with the disease output by the N base learners;

[0031] Processing the characteristic information based on a weighted voting method to obtain disease-related characteristic information;

[0032] The regression meta-learner is called to process the disease-related feature information to obtain a prediction result of whether the target user suffers from a preset disease.

[0033] In a second aspect, an embodiment of the present application provides a disease prediction model training device, the device comprising:

[0034] A training sample acquisition module, used to acquire a model training sample, wherein the model training sample is multimodal data formed by a fundus image of a user and non-invasive user data of the user associated with a preset disease, and the model training sample is annotated with a real disease category;

[0035] A prediction category acquisition module, used to train the disease prediction model to be trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained;

[0036] A loss value calculation module, used to calculate the loss value of the disease prediction model to be trained based on the real disease category, the predicted disease category and the weights corresponding to the model training samples;

[0037] The disease prediction model acquisition module is used to use the trained disease prediction model to be trained as the final disease prediction model for predicting the preset disease when the loss value is within a preset range.

[0038] Optionally, the predicted category acquisition module includes:

[0039] The prediction category acquisition unit is used to train the disease prediction model to be trained using a five-fold cross-validation method and the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained.

[0040] Optionally, the disease prediction model to be trained includes: N base learners and a regression meta-learner, wherein the N base learners are connected in parallel and are all connected to the regression meta-learner, and N is a positive integer greater than or equal to 2.

[0041] The prediction category acquisition module includes:

[0042] An initial feature acquisition unit, used for calling the N base learners to process the multimodal data respectively, to obtain initial feature information associated with the preset disease corresponding to the multimodal data output by the N base learners;

[0043] A target feature acquisition unit, used for processing the initial feature information based on a weighted voting method to obtain target feature information;

[0044] A predicted category output unit is used to call the regression meta-learner to process the target feature information and output the predicted disease category.

[0045] Optionally, the disease prediction model to be trained further includes: a self-attention mechanism layer,

[0046] The initial feature acquisition unit comprises:

[0047] A feature map acquisition subunit is used to call the self-attention mechanism layer to perform attention mechanism learning on the fundus image to obtain an attention feature map corresponding to the fundus image;

[0048] The initial feature acquisition subunit is used to call N of the base learners to process the attention feature map and the non-invasive user data respectively, so as to obtain the initial feature information associated with the preset disease output by the N base learners.

[0049] Optionally, the loss value calculation module includes:

[0050] A sample weight calculation unit, used to calculate the weight of the model training samples of the target category according to the total number of samples, the total number of categories and the number of samples of the target category in this model training;

[0051] A loss value calculation unit is used to calculate the loss value of the disease prediction model to be trained based on the weight, the real disease category and the predicted disease category.

[0052] Optionally, the non-invasive user data includes: daily alcohol consumption, user taste data, body mass index, daily meat consumption and daily cereal food consumption.

[0053] Optionally, the preset diseases include: any one of liver and gallbladder diseases, kidney stones, and gout.

[0054] Optionally, the device further comprises:

[0055] A target data acquisition module, used to acquire target multimodal data of a target user, wherein the target multimodal data is data formed by a target fundus image of the target user and non-invasive data of the target user associated with a preset disease;

[0056] A target data input module, used for inputting the target multimodal data into the disease prediction model;

[0057] A feature information acquisition module, used for calling N base learners to process the target multimodal data respectively, and obtaining feature information associated with the disease output by the N base learners;

[0058] A correlation feature acquisition module, used to process the feature information based on a weighted voting method to obtain disease correlation feature information;

[0059] The prediction result acquisition module is used to call the regression meta-learner to process the disease-related feature information to obtain a prediction result of whether the target user suffers from a preset disease.

[0060] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0061] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements any of the above-mentioned disease prediction model training methods when executed by the processor.

[0062] In a fourth aspect, an embodiment of the present application provides a readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any of the disease prediction model training methods described above.

[0063] In an embodiment of the present application, by obtaining a model training sample, the model training sample is multimodal data formed by the user's fundus image and the user's non-invasive user data associated with a preset disease, and the model training sample is annotated with a real disease category. The disease prediction model to be trained is trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained. Based on the weights corresponding to the real disease category, the predicted disease category and the model training sample, the loss value of the disease prediction model to be trained is calculated. When the loss value is within a preset range, the trained disease prediction model to be trained is used as the final disease prediction model for predicting the preset disease. The embodiment of the present application obtains a disease prediction model by training multimodal data formed by combining the user's fundus image and non-invasive data associated with a preset disease (such as hepatobiliary disease, kidney stones, gout, etc.). The disease prediction model is used to predict the preset disease, which can improve the efficiency and accuracy of disease prediction compared to manual prediction.

[0064] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A flowchart of the steps of a disease prediction model training method provided in an embodiment of the present application;

[0067] Figure 2 A flowchart of the steps of a hepatobiliary disease prediction model training method provided in an embodiment of the present application;

[0068] Figure 3A schematic diagram of a relevant index and the Pearson coefficient of hepatobiliary disease provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of a five-fold cross-validation process provided in an embodiment of the present application;

[0070] Figure 5 A schematic diagram of a prediction process provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of a multimodal model provided in an embodiment of the present application;

[0072] Figure 7 A flowchart of the steps of a gout prediction model training method provided in an embodiment of the present application;

[0073] Figure 8 A schematic diagram of a gout prediction model training process provided in an embodiment of the present application;

[0074] Fig. 9 A schematic diagram of the AUC result on a test set provided in an embodiment of the present application;

[0075] Fig.10 A flowchart of the steps of a kidney stone prediction model training method provided in an embodiment of the present application;

[0076] Fig.11 A schematic diagram of a self-attention mechanism layer provided in an embodiment of the present application;

[0077] Fig.12 A schematic diagram of a specific implementation process provided for an embodiment of the present application;

[0078] Fig.13 A schematic diagram of the AUC result on another test set provided in an embodiment of the present application;

[0079] Fig.14 A schematic diagram of the structure of a disease prediction model training device provided in an embodiment of the present application;

[0080] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0082] Reference Figure 1 , shows a flowchart of the steps of a disease prediction model training method provided by an embodiment of the present application. Figure 1 As shown, the disease prediction model training method may include: step 101, step 102, step 103 and step 104.

[0083] Step 101: Obtain a model training sample, where the model training sample is multimodal data formed by a fundus image of a user and non-invasive user data of the user associated with a preset disease, and the model training sample is annotated with a real disease category.

[0084] The embodiments of the present application can be applied to scenarios in which a disease prediction model is trained by combining multimodal data formed by fundus images and non-invasive user data associated with preset diseases.

[0085] Model training samples refer to samples used to train a disease prediction model. In this example, the model training samples may be multimodal data formed by the user's fundus images and the user's non-invasive user data associated with a preset disease.

[0086] In this example, the preset diseases may be, but are not limited to, hepatobiliary diseases, kidney stones, and gout.

[0087] In the specific implementation, fundus images can be used as one of the model training samples. However, pure fundus images have limited features related to the preset diseases, and the prediction ability of using only fundus images as model input is poor. Other non-invasive meta data are additionally integrated to construct multimodal data input.

[0088] The non-invasive meta data included may include: daily meat consumption, daily cereal consumption, and BMI. Therefore, the non-invasive indicators finally included are daily alcohol consumption, user taste data, body mass index (BMI, BMI = weight / height squared (international unit kg / ㎡)), daily meat consumption, and daily cereal consumption.

[0089] When training a disease prediction model, the user's fundus image and the user's non-invasive user data associated with a preset disease may be acquired to form multimodal data as a model training sample.

[0090] After the model training samples are obtained, step 102 is executed.

[0091] Step 102: training the disease prediction model to be trained based on the multimodal data to obtain a predicted disease category output by the disease prediction model to be trained.

[0092] After obtaining the model training samples, the disease prediction model to be trained can be trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained.

[0093] In a specific implementation, a five-fold cross-validation method and multimodal data may be used to train the disease prediction model to be trained, and the predicted disease category output by the disease prediction model to be trained may be obtained.

[0094] The five-fold cross validation process can be as follows Figure 4 As shown, the following steps may be specifically included:

[0095] 1. Divide the data set into training set and test set;

[0096] 2. Divide the training set into 6 parts;

[0097] 3. Use one of the six sets as the validation set each time, and the other five sets as the training set;

[0098] 4. After 5 trainings, five different models can be obtained;

[0099] 5. Select the best hyperparameters;

[0100] 6. Use the best hyperparameter model, and then use all 5 points of data as the retrained hyperparameter model to obtain the final model;

[0101] 7. Use the test set to test the final model effect.

[0102] The embodiment of the present application adopts a five-fold cross-validation model training method, which can reduce overfitting to a certain extent and obtain as much effective information as possible from limited data.

[0103] In this embodiment, the disease prediction model to be trained includes: N base learners and a regression meta-learner, wherein the N base learners are connected in parallel and are all connected to the regression meta-learner, and N is a positive integer greater than or equal to 2.

[0104] The model training process can be described in detail in combination with the following specific implementation methods.

[0105] In a specific implementation of the present application, the above step 102 may include:

[0106] Sub-step A1: calling N base learners to process the multimodal data respectively, and obtaining initial feature information associated with the preset disease corresponding to the multimodal data output by the N base learners.

[0107] In this embodiment, after the model training samples are obtained, the model training samples can be input into the disease prediction model to be trained.

[0108] Furthermore, N base learners can be called to process the multimodal data respectively to obtain the initial feature information associated with the preset disease corresponding to the multimodal data output by the N base learners. In this example, the multimodal data input into the N base learners are exactly the same. The N base learners are N different models. By processing the multimodal data through multiple models, more accurate initial feature information associated with the preset disease can be obtained.

[0109] It can be understood that the more base learners there are, the higher the accuracy of the initial feature information associated with the preset disease. Therefore, in order to improve the accuracy of the initial feature information associated with the preset disease, in this embodiment, the value of N is preferably 3 or more.

[0110] After obtaining the initial feature information associated with the preset disease corresponding to the multimodal data output by the N base learners, sub-step A2 is performed.

[0111] Sub-step A2: Processing the initial feature information based on a weighted voting method to obtain target feature information.

[0112] After obtaining the initial feature information associated with the preset disease corresponding to the multimodal data output by N base learners, the weighted voting method can be used to process the initial feature information to obtain the target feature information. For example, when the number of base learners is 3, the outputs of 3 base learners can be obtained. If two output results are close and the other is far away, the two output results are selected and weighted to obtain the final output result, which is the target feature information.

[0113] It can be understood that the above examples are only examples listed for a better understanding of the technical solutions provided by the embodiments of the present application, and are not intended to be the sole limitation to the embodiments.

[0114] After the initial feature information is processed based on the weighted voting method to obtain the target feature information, sub-step A3 is performed.

[0115] Sub-step A3: calling the regression meta-learner to process the target feature information and output the predicted disease category.

[0116] After obtaining the target feature information, the regression meta-learner can be called to process the target feature information to output the predicted disease category.

[0117] In this embodiment, the disease prediction model to be trained can be an integrated learning that combines the prediction results of multiple models to obtain a prediction result that is better and more robust than any one model. For example, the number of base learners can be set to 3, and the three base learners are VGG-16, Inception-V3, and ResNet-50. When training the model, a five-fold cross-validation method can be used. First, the data set is divided into a training set and a test set. The three base learners of VGG-16, Inception-V3, and ResNet-50 are trained using the training set, and then the base learner is used to predict the test set. The result output of the base learner is processed using a weighted voting method, and the processed output value is used as the input value of the next stage of training. The final label is used as the output value for training the Logistic regression meta-learner, and the training set in the meta-learner is divided into a training set and a test set according to the five-fold cross-validation method. Five-fold cross-validation is performed on each base learner. Since the training data used twice are different, overfitting can be prevented to a certain extent. And since patient data is very scarce and precious, using the Stacking method will get more robust results when the total amount of data is not large. Figure 5 As shown in the figure, when the model is trained, the input samples are multimodal data: fundus images and meta data. Dataset 1, Dataset 2, and Dataset 3 (the data of the three data sets are the same) are respectively input to base learner 1: VGG-16, base learner 2: Inception-V3, and base learner 3: ResNet-50. The outputs of the three base learners are processed by weighted voting to obtain the final output value, which can be used as the input to the Logistic Regression Meta Learner for the next stage of training. The final label is used as the output value to train the Logistic Regression Meta Learner. The Logistic Regression Meta Learner outputs the disease prediction results.

[0118] In this embodiment, a self-attention mechanism layer can also be set in the disease prediction model to be trained, and the self-attention mechanism layer can highlight the distinguishing areas in the feature map while suppressing irrelevant information. The processing process of the self-attention mechanism layer can be described in detail in combination with the following specific implementation method.

[0119] In another specific implementation of the present application, the above sub-step A1 may include:

[0120] Sub-step B1: calling the self-attention mechanism layer to perform attention mechanism learning on the fundus image to obtain an attention feature map corresponding to the fundus image.

[0121] In this embodiment, the main idea of ​​the attention mechanism (Self-Attention, SA) is to highlight the distinguishing regions in the feature map while suppressing irrelevant information. Usually, the attention mechanism is embedded in the convolutional neural network to generate an attention feature map by exploring a learnable attention weight map covering the original feature map. Most of the existing research belongs to the self-attention mechanism, that is, the attention map is inferred only based on the feature map itself. The self-attention module receives the feature map F∈R C×W×H , output self-attention feature map F SA ∈R C×W×H , whose shape is similar to F∈R C×W×H The same, where C, W, H represent the channel, width and height of the feature map. To generate the self-attention feature map, the input feature map F is first input into three learnable 1×1 convolution operations to generate three vectors (query vector F q ∈R C×W×H , the key vector F k ∈R C×W×H , value vector F v ∈R C×W×H ). By calculating F q With all F k The dot product of self-attention feature F is used to generate the attention weight. SA It is by F q and F k The obtained attention weight map F v The weighted sum of the total values ​​of is obtained:

[0122]

[0123] in,

[0124] SA is the self-attention module (i.e., the self-attention mechanism layer). The self-attention mechanism layer can be Fig.11 See Fig.11 , shows a schematic diagram of a self-attention module provided in an embodiment of the present application. Fig.11 As shown in the figure, in the self-attention module, after the image features pass through three 1×1 convolutional layers, the obtained features are fused to obtain the attention features.

[0125] After the model training samples are input into the disease prediction model to be trained, the self-attention mechanism layer can be called to perform attention mechanism learning on the fundus image to obtain the attention feature map corresponding to the fundus image.

[0126] After obtaining the attention feature map, execute sub-step B2.

[0127] Sub-step B2: calling N base learners to process the attention feature map and the non-invasive user data respectively, and obtaining initial feature information associated with the preset disease output by the N base learners.

[0128] After obtaining the attention feature map, N base learners can be used to process the attention feature map and non-invasive user data respectively to obtain the initial feature information associated with the preset disease output by the N base learners. In this example, the multimodal data input into the N base learners are exactly the same. The N base learners are N different models. By processing the multimodal data through multiple models, more accurate initial feature information associated with the preset disease can be obtained.

[0129] After the disease prediction model to be trained is trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained, step 103 is performed.

[0130] Step 103: Based on the actual disease category, the predicted disease category and the weights corresponding to the model training samples, the loss value of the disease prediction model to be trained is calculated.

[0131] After the disease prediction model to be trained is trained based on multimodal data to obtain the predicted disease category output by the disease prediction model to be trained, the loss value of the disease prediction model to be trained can be calculated based on the weights corresponding to the real disease category, the predicted disease category and the model training samples.

[0132] In this example, in order to solve the problem of model performance degradation caused by unbalanced collected samples in actual scenarios, a weighted cross entropy loss function is introduced, that is, the weight of the loss function is increased for small sample labels. The calculation process of the loss value can be described in detail in combination with the following specific implementation method.

[0133] In another specific implementation of the present application, the above step 103 may include:

[0134] Sub-step C1: According to the total number of samples, the total number of categories and the number of samples of the target category in this model training, the weight of the model training samples of the target category is calculated.

[0135] In this embodiment, the total number of samples for this model training (i.e., the total number of model training samples used in this model training), the total number of categories (i.e., the total number of real disease categories annotated by the model training samples used in this model training), and the number of samples of the target category can be obtained. Then, the weight of the model training samples of the target category can be calculated based on the total number of samples for this model training, the total number of categories, and the number of samples of the target category.

[0136] Sub-step C2: Based on the weight, the actual disease category and the predicted disease category, calculate the loss value of the disease prediction model to be trained.

[0137] After calculating the weights of the model training samples of the target category, the loss value of the disease prediction model to be trained can be calculated based on the calculated weights, the annotated real disease categories, and the predicted disease categories. The specific calculation formula is as follows:

[0138]

[0139] In the above formula, N represents the batch size, y i (n) represents the label value of sample n in the i-th category (if sample n belongs to this category, then y i (n) is 1, otherwise 0), C is the total number of categories, p i (n) represents the predicted probability that sample n is the i-th category, w c Represents the weight of the sample whose actual label is c.

[0140] The weight can be calculated as shown in the following formula:

[0141]

[0142] Among them, N sample Represents the total number of samples, n_classes is the total number of categories, N c Represents the total number of samples in the cth category.

[0143] If the distribution of the training set and the test set are not so consistent, the stacking ensemble learning model will have certain problems. Retraining with the labels trained by the initial model using the real labels will cause the model to overfit the training set, resulting in a certain decrease in the generalization ability or effect of the model on the test set. The problem of poor model generalization ability caused by overfitting can be solved by introducing five-fold cross-validation. The process of five-fold cross-validation is described above and will not be repeated in this embodiment.

[0144] After the loss value of the disease prediction model to be trained is calculated based on the actual disease category, the predicted disease category and the weights corresponding to the model training samples, step 104 is executed.

[0145] Step 104: When the loss value is within a preset range, the trained disease prediction model to be trained is used as the final disease prediction model for predicting the preset disease.

[0146] After calculating the loss value of the disease prediction model to be trained based on the real disease category, the predicted disease category and the weights corresponding to the model training samples, it can be determined whether the loss value is within a preset range.

[0147] If the loss value is within the preset range, it means that the disease prediction model to be trained has converged. At this time, the trained disease prediction model to be trained can be used as the final disease prediction model.

[0148] After the disease prediction model is trained, it can be used for subsequent preset disease prediction scenarios, such as Figure 5 As shown, when the model prediction process is performed, the fundus image and clinical data can be input, input into the prediction network (ie, the disease prediction model), and finally the prediction result is output. The model prediction process can be described in detail in combination with the following specific implementation methods.

[0149] In another specific implementation of the present application, after the above step 104, the following may also be included:

[0150] Step D1: Acquire target multimodal data of a target user, wherein the target multimodal data is data formed by a target fundus image of the target user and non-invasive data of the target user associated with a preset disease.

[0151] In this embodiment, when predicting a preset disease for a target user, target multimodal data of the target user can be obtained, wherein the target multimodal data can be data formed by a target fundus image of the target user and non-invasive data of the target user associated with the preset disease.

[0152] After the target multimodal data of the target user is acquired, step D2 is performed.

[0153] Step D2: Inputting the target multimodal data into the disease prediction model.

[0154] After the target multimodal data of the target user is acquired, the target multimodal data may be input into the disease prediction model and step D3 is performed.

[0155] Step D3: calling N base learners to process the target multimodal data respectively, and obtaining feature information associated with the disease output by the N base learners.

[0156] After the target multimodal data is input into the preset disease prediction model, N base learners can be called to process the target multimodal data respectively to obtain feature information associated with the preset disease output by the N base learners.

[0157] Step D4: Process the characteristic information based on the weighted voting method to obtain disease-related characteristic information.

[0158] After obtaining the characteristic information associated with the preset disease output by the N base learners, the characteristic information may be processed based on a weighted voting method to obtain the characteristic information associated with the preset disease.

[0159] Step D5: calling the regression meta-learner to process the disease-related feature information to obtain a prediction result of whether the target user suffers from a preset disease.

[0160] Finally, the regression meta-learner can be called to process the preset disease-related feature information to obtain a prediction result of whether the target user has the preset disease. Specifically, the regression meta-learner can process the preset disease-related feature information to normalize and obtain values ​​of 0 and 1, where 1 indicates that the target user has the preset disease, and 0 indicates that the target user does not have the preset disease.

[0161] In specific implementations, since eye changes are only related to a few preset diseases, and since these changes are non-specific and have a low detection rate, it is difficult to perform early screening for preset diseases using only fundus image information and traditional deep learning models. The present application embodiment proposes a new early prediction method for preset diseases, which incorporates non-invasive multimodal information and integrated learning technology. This method uses simple, convenient, non-invasive and low-cost fundus images and text information, which can greatly reduce diagnostic time and costs while avoiding invasive diagnostic methods. Therefore, the method proposed in the present application embodiment can perform large-scale early screening for preset diseases more accurately, conveniently and efficiently, and has strong practical value.

[0162] The disease prediction model training method provided in the embodiment of the present application obtains model training samples, and the model training samples are multimodal data formed by the user's fundus image and the user's non-invasive user data associated with the preset disease, and the model training samples are annotated with real disease categories. The disease prediction model to be trained is trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained. Based on the weights corresponding to the real disease category, the predicted disease category and the model training samples, the loss value of the disease prediction model to be trained is calculated. When the loss value is within the preset range, the trained disease prediction model to be trained is used as the final disease prediction model for predicting the preset disease. The embodiment of the present application obtains a disease prediction model by training multimodal data formed by combining the user's fundus image and non-invasive data associated with the preset disease (such as hepatobiliary disease, kidney stones, gout, etc.). The disease prediction model is used to predict the preset disease, which can improve the efficiency and accuracy of disease prediction compared to manual prediction.

[0163] In this embodiment, the preset diseases may include any one of liver and gallbladder diseases, kidney stones and gout. In a specific implementation, the disease prediction model can be used to predict one of the diseases. The training process of the three disease prediction models is described in detail below in conjunction with the following embodiments.

[0164] First, the training of hepatobiliary disease prediction models can be combined with Figure 2 This is described in detail as follows.

[0165] Reference Figure 2 , shows a flowchart of the steps of a hepatobiliary disease prediction model training method provided by an embodiment of the present application. Figure 2 As shown, the training method of the hepatobiliary disease prediction model may include: step 201, step 202, step 203 and step 204.

[0166] Step 201: Obtain a model training sample, where the model training sample is multimodal data formed by a user's fundus image and the user's non-invasive user data associated with hepatobiliary diseases, and the model training sample is annotated with a real hepatobiliary disease category.

[0167] The embodiments of the present application can be applied to a scenario in which a hepatobiliary disease prediction model is trained by combining fundus images and non-invasive data associated with hepatobiliary diseases.

[0168] Model training samples refer to samples used to train the hepatobiliary disease prediction model. In this example, the model training samples can be multimodal data formed by the user's fundus image and the user's non-invasive user data associated with hepatobiliary diseases. Each model training sample is labeled with a real hepatobiliary disease category, such as chronic hepatitis, hepatitis C antibodies, type B surface antigen, fatty liver, gallstones, etc.

[0169] In the specific implementation, fundus images can be used as one of the model training samples, but pure fundus images have limited features related to hepatobiliary diseases. The prediction ability of using only fundus images as model input is poor. In addition, other non-invasive meta data are incorporated to construct multimodal data input. Non-invasive meta data is obtained by screening through the Pearson correlation coefficient with the disease. Figure 3 The relationships between the relevant indices shown and the Pearson coefficient for hepatobiliary diseases.

[0170] When training a hepatobiliary disease prediction model, the user's fundus images and the user's non-invasive user data associated with the hepatobiliary disease may be obtained to form a model training sample.

[0171] After the model training samples are obtained, step 202 is executed.

[0172] Step 202: training the hepatobiliary disease prediction model to be trained based on the multimodal data to obtain a predicted hepatobiliary disease category output by the hepatobiliary disease prediction model to be trained.

[0173] After obtaining the model training samples, the hepatobiliary disease prediction model to be trained can be trained based on the multimodal data to obtain the predicted hepatobiliary disease category output by the hepatobiliary disease prediction model to be trained. Figure 6 As shown, the fundus image and meta data are converted into multimodal data and input into the Stacking learning model.

[0174] In a specific implementation, a five-fold cross-validation method and multimodal data may be used to train the hepatobiliary disease prediction model to be trained, and the predicted hepatobiliary disease category output by the hepatobiliary disease prediction model to be trained may be obtained.

[0175] In this example, the hepatobiliary disease prediction model to be trained may include: N (N is a positive integer greater than or equal to 2) basis learners and regression meta-learners. The training process of the model can refer to the description of the above embodiment, and this embodiment will not be repeated here.

[0176] After the hepatobiliary disease prediction model to be trained is trained based on the multimodal data to obtain the predicted hepatobiliary disease category output by the hepatobiliary disease prediction model to be trained, step 203 is performed.

[0177] Step 203: Based on the actual hepatobiliary disease category, the predicted hepatobiliary disease category and the weights corresponding to the model training samples, the loss value of the hepatobiliary disease prediction model to be trained is calculated.

[0178] After the hepatobiliary disease prediction model to be trained is trained based on multimodal data to obtain the predicted hepatobiliary disease category output by the hepatobiliary disease prediction model to be trained, the loss value of the hepatobiliary disease prediction model to be trained can be calculated based on the actual hepatobiliary disease category, the predicted hepatobiliary disease category and the weights corresponding to the model training samples.

[0179] In this example, in order to solve the problem of model performance degradation caused by unbalanced collected samples in actual scenarios, a weighted cross entropy loss function is introduced, that is, the weight of the loss function is increased for small sample labels. The calculation process of the loss value can refer to the description in the above embodiment, and this embodiment will not be repeated here.

[0180] After the loss value of the hepatobiliary disease prediction model to be trained is calculated based on the actual hepatobiliary disease category, the predicted hepatobiliary disease category and the weights corresponding to the model training samples, step 204 is executed.

[0181] Step 204: When the loss value is within a preset range, the trained hepatobiliary disease prediction model to be trained is used as the final hepatobiliary disease prediction model.

[0182] After calculating the loss value of the hepatobiliary disease prediction model to be trained based on the actual hepatobiliary disease category, the predicted hepatobiliary disease category and the weights corresponding to the model training samples, it can be determined whether the loss value is within a preset range.

[0183] If the loss value is within the preset range, it means that the hepatobiliary disease prediction model to be trained has converged. At this time, the trained hepatobiliary disease prediction model to be trained can be used as the final hepatobiliary disease prediction model.

[0184] After the hepatobiliary disease prediction model is trained, the hepatobiliary disease prediction model can be used for subsequent hepatobiliary disease prediction scenarios, such as Figure 5 As shown in the figure, when the model prediction process is performed, the fundus image and clinical data can be input into the prediction network (i.e., the hepatobiliary disease prediction model), and the prediction result is finally output. Figure 6 As shown in the figure, multimodal data can be transformed into disease prediction results after passing through the learning model.

[0185] The training method of the hepatobiliary disease prediction model provided in the embodiment of the present application obtains the hepatobiliary disease prediction model by training multimodal data formed by combining the user's fundus images and non-invasive data associated with hepatobiliary diseases, which can improve the accuracy of hepatobiliary disease prediction and reduce the difficulty of model training.

[0186] Second, the training of gout prediction model can be combined with Figure 7 This is described in detail as follows.

[0187] Reference Figure 7 , shows a flowchart of the steps of a gout prediction model training method provided in an embodiment of the present application, such as Figure 7 As shown, the gout prediction model training method may include: step 701, step 702, step 703 and step 704.

[0188] Step 701: Acquire a model training sample, where the model training sample is multimodal data formed by a fundus image of a user and non-invasive user data of the user associated with gout.

[0189] The embodiments of the present application can be applied to a scenario in which a gout prediction model is trained by combining fundus images and non-invasive data associated with gout.

[0190] The model training samples refer to samples used to train a gout prediction model. In this example, the model training samples may be multimodal data formed by the user's fundus images and the user's non-invasive user data associated with gout.

[0191] In the specific implementation, fundus images can be used as one of the model training samples. However, pure fundus images have limited features related to gout, and the prediction ability of using only fundus images as model input is poor. Other non-invasive meta data are additionally integrated to construct multimodal data input.

[0192] When training a gout prediction model, the user's fundus images and the user's non-invasive user data associated with gout can be obtained to form a model training sample.

[0193] After the model training samples are obtained, step 702 is executed.

[0194] Step 702: training a gout prediction model to be trained based on the multimodal data to obtain a predicted probability of the user suffering from gout output by the gout prediction model to be trained.

[0195] After obtaining the model training samples, the gout prediction model to be trained can be trained based on the multimodal data to obtain the predicted probability of the user suffering from gout output by the gout prediction model to be trained.

[0196] In a specific implementation, a five-fold cross validation method and multimodal data can be used to train the gout prediction model to be trained, and the predicted probability of the user suffering from gout output by the gout prediction model to be trained can be obtained. The five-fold cross validation process can be as described in the above embodiment, and this embodiment will not be repeated here.

[0197] The embodiment of the present application adopts a five-fold cross-validation model training method, which can reduce overfitting to a certain extent and obtain as much effective information as possible from limited data.

[0198] In this example, the gout prediction model to be trained may include: N (N is a positive integer greater than or equal to 2) basis learners and regression meta-learners. The training process of the model can refer to the description of the above embodiment, and this embodiment will not be repeated here.

[0199] After the gout prediction model to be trained is trained based on the multimodal data to obtain the predicted probability of the user suffering from gout output by the gout prediction model to be trained, step 703 is performed.

[0200] Step 703: Based on the prediction probability, the hard-to-distinguish sample weights corresponding to the model training samples, and the category weights of the model training samples, the loss value of the gout prediction model to be trained is calculated.

[0201] After training the gout prediction model to be trained based on multimodal data and obtaining the predicted probability of the user suffering from gout output by the gout prediction model to be trained, the loss value of the gout prediction model to be trained can be calculated based on the predicted probability, the hard-to-distinguish sample weights corresponding to the model training samples, and the category weights of the model training samples.

[0202] In this embodiment, in order to focus the model's attention on rare categories when the data categories are unbalanced, because these categories have few samples and are difficult to classify, FocalLoss is used in this embodiment. The calculation process of the loss value can be described in detail in conjunction with the following specific implementation method.

[0203] In a specific implementation of the present application, the above step 703 may include:

[0204] Sub-step E1: Determine the category weights and the hard-to-distinguish sample weights of the model training samples used in the current model training according to the number of positive and negative samples of the model training samples used in the current model training.

[0205] In this embodiment, when calculating the loss value of the gout prediction model to be trained, the number of positive and negative samples of the model training samples used in this model training can be obtained, where the multimodal data of users suffering from gout are positive samples, and conversely, the multimodal data of users not suffering from gout are negative samples.

[0206] After obtaining the number of positive and negative samples of the model training samples used in this model training, the category weights and hard-to-distinguish sample weights of the model training samples used in this model training can be determined based on the number. The category weights refer to the weights corresponding to the positive sample category and the negative sample category. The hard-to-distinguish sample weights refer to the weights corresponding to the difference in the number of positive and negative samples.

[0207] After determining the category weights and the hard-to-distinguish sample weights of the model training samples used in the current model training according to the number of positive and negative samples of the model training samples used in the current model training, sub-step E2 is performed.

[0208] Sub-step E2: Based on the category weight, the hard-to-classify sample weight and the prediction probability, calculate the loss value of the gout prediction model to be trained.

[0209] After determining the category weights and hard-to-distinguish sample weights of the model training samples used in this model training according to the number of positive and negative samples of the model training samples used in this model training, the loss value of the gout prediction model to be trained can be calculated based on the category weights, hard-to-distinguish sample weights and prediction probabilities. The calculation formula is as follows:

[0210] FL(p)=-a(1-p)γ log(p)

[0211] In the above formula, a is the category weight, which is used to balance the imbalance between positive and negative samples. If there are more negative samples, the weight a given to the negative samples will be smaller, thus reducing the impact of negative samples.

[0212] γ is the weight of the hard-to-separate samples, which is used to weigh the hard-to-separate samples and the easy-to-separate samples, and plays a smoothing role. In the experiment, a=0.25 is used. γ =2 works best.

[0213] After the loss value of the gout prediction model to be trained is calculated based on the prediction probability, the hard-to-distinguish sample weights corresponding to the model training samples, and the category weights of the model training samples, step 704 is executed.

[0214] Step 704: When the loss value is within a preset range, the trained gout prediction model to be trained is used as the final gout prediction model.

[0215] After calculating the loss value of the gout prediction model to be trained based on the prediction probability, the hard-to-distinguish sample weights corresponding to the model training samples, and the category weights of the model training samples, it can be determined whether the loss value is within a preset range.

[0216] If the loss value is within the preset range, it means that the gout prediction model to be trained has converged. At this time, the trained gout prediction model to be trained can be used as the final gout prediction model.

[0217] After the gout prediction model is trained, the gout prediction model can be used for subsequent gout prediction scenarios. The model prediction process can refer to the description of the above embodiment, and this embodiment will not be repeated here.

[0218] The embodiment of the present application incorporates non-invasive multimodal information and integrated learning technology. The method uses simple, convenient, non-invasive and low-cost fundus images and text information, which can greatly reduce the diagnosis time and cost, while avoiding invasive diagnosis methods. Therefore, the method proposed in this embodiment can be more accurate, convenient and efficient for large-scale early screening of gout, and has strong practical value.

[0219] In order to verify the effect of the gout prediction model obtained by combining multimodal data training, the following experiments were conducted: Figure 8 As shown:

[0220] 1. Data Collection

[0221] The data set uses the physical examination questionnaire, physical examination information form, fundus Figure 3 Partial composition;

[0222] 2. Data labeling

[0223] Doctors diagnose whether patients have gout based on physical examination data and record it, and then perform data cleaning and matching.

[0224] 3. Image preprocessing

[0225] The patient's fundus image and the corresponding mark combination constitute the data set of the present invention. In the model training stage, the data is divided into a training set and a test set according to a ratio of 8:2, wherein the specific data is shown in Table 1:

[0226] Table 1: Experimental data distribution

[0227] Training set Test Set Total number of pictures 648 164 Total number of people 324 82 Total number of patients 162 41 Control group 162 41 Number of gout patient pictures 324 82

[0228] 4. Model training

[0229] The stacking ensemble learning method is constructed using three base learners: VGG-16, Inception-V3, and ResNet-50. The meta-learner uses Logistic regression, and a five-fold cross validation is performed during the training of each base learner. The input image size of the model is 224x224, the batch size is 16, the stochastic gradient descent algorithm is used, and the initial learning rate is 0.0001.

[0230] 5. Experimental results prediction

[0231] For the experimental results, AUC, ACC, SEN, and SPE are used as evaluation indicators. In the experiment, the training set is used for training, and the model with the best results on the training set is selected and tested on the test set. The average AUC on the test set is 0.76, and the average SEN is 0.74. The AUC curve is shown in Figure 9.

[0232] Based on the above experimental analysis, it can be seen that the gout prediction model trained by the method provided in this embodiment has a good prediction effect.

[0233] The gout prediction model training method provided in the embodiment of the present application can reduce the difficulty of data collection by using non-invasive user data to train the gout prediction model. At the same time, the gout prediction model obtained by multimodal data training combining fundus images and non-invasive data can improve the accuracy of gout prediction.

[0234] Third, the training of kidney stone prediction model can be combined with Fig.10 This is described in detail as follows.

[0235] Reference Fig.10, shows a flowchart of the steps of a kidney stone prediction model training method provided in an embodiment of the present application, such as Fig.10 As shown, the kidney stone prediction model training method may include: step 1001, step 1002, step 1003 and step 1004.

[0236] Step 1001: Obtain a model training sample, where the model training sample is multimodal data formed by a fundus image of a user and non-invasive user data of the user associated with kidney stones, and the model training sample is annotated with real data on whether the user suffers from kidney stones.

[0237] The embodiments of the present application can be applied to a scenario in which a kidney stone prediction model is trained by combining fundus images and non-invasive data associated with kidney stones.

[0238] The model training samples refer to samples used to train the kidney stone prediction model. In this example, the model training samples may be multimodal data formed by the user's fundus image and the user's non-invasive user data associated with kidney stones. Each model training sample is annotated with real data of whether the user suffers from kidney stones.

[0239] In the specific implementation, fundus images can be used as one of the model training samples. However, pure fundus images have limited features related to kidney stones, and the prediction ability of using only fundus images as model input is poor. Other non-invasive meta data are additionally integrated to construct multimodal data input.

[0240] When training a kidney stone prediction model, the user's fundus images and the user's non-invasive user data associated with kidney stones may be acquired to form a model training sample.

[0241] After obtaining the model training samples, execute step 1002.

[0242] Step 1002: training a kidney stone prediction model to be trained based on the multimodal data, so that the kidney stone prediction model to be trained outputs a predicted probability that the user suffers from kidney stones.

[0243] After obtaining the model training samples, the kidney stone prediction model to be trained can be trained based on the multimodal data to obtain the predicted probability of the user suffering from kidney stones output by the kidney stone prediction model to be trained.

[0244] In a specific implementation, a five-fold cross validation method and multimodal data can be used to train the kidney stone prediction model to be trained, and the predicted probability of the user suffering from kidney stones output by the kidney stone prediction model to be trained is obtained. The five-fold cross validation process can be as described in the first embodiment above, and this embodiment will not be repeated here.

[0245] The embodiment of the present application adopts a five-fold cross-validation model training method, which can reduce overfitting to a certain extent and obtain as much effective information as possible from limited data.

[0246] In this example, the kidney stone prediction model to be trained may include: a self-attention mechanism layer, N (N is a positive integer greater than or equal to 2) base learners and a regression meta-learner. The self-attention mechanism layer can be as follows Fig.11 See Fig.11 , shows a schematic diagram of a self-attention module provided in an embodiment of the present application. Fig.11 As shown, in the self-attention module, after the image features pass through three 1×1 convolutional layers, the obtained features are fused to obtain the attention features. The training process of the model can be combined with the description of the above embodiment, and this embodiment will not be repeated here.

[0247] After the kidney stone prediction model to be trained is trained based on the multimodal data and the kidney stone prediction model to be trained outputs a predicted probability of the user having kidney stones, step 1003 is performed.

[0248] Step 1003: Based on the predicted probability and the real data, the loss value of the kidney stone prediction model to be trained is calculated.

[0249] After the kidney stone prediction model to be trained is trained based on the multimodal data, and the predicted probability of the user suffering from kidney stones is obtained from the kidney stone prediction model to be trained, the loss value of the kidney stone prediction model to be trained can be calculated based on the predicted probability and the real data. In this example, the calculated loss value can be a cross entropy loss value or other types of loss values. Specifically, the calculation method and type of the loss value can be determined according to business needs, and this embodiment does not limit this.

[0250] After the loss value of the kidney stone prediction model to be trained is calculated based on the predicted probability and the real data, step 1004 is executed.

[0251] Step 1004: When the loss value is within a preset range, the trained kidney stone prediction model to be trained is used as the final kidney stone prediction model.

[0252] After the loss value of the kidney stone prediction model to be trained is calculated based on the predicted probability and the real data, it can be determined whether the loss value is within a preset range.

[0253] If the loss value is within the preset range, it means that the kidney stone prediction model to be trained has converged. At this time, the trained kidney stone prediction model to be trained can be used as the final kidney stone prediction model.

[0254] After the kidney stone prediction model is trained, the kidney stone prediction model can be used for subsequent kidney stone prediction scenarios. The model prediction process can be combined with the description in the above embodiment, and this embodiment will not be repeated here.

[0255] The above process can be combined with Fig.12 This is described in detail as follows.

[0256] like Fig.12 As shown in the figure, data can be collected through hospital outpatient clinics. The collected data include: fundus images and physical examination / questionnaire data. Then the doctor diagnoses the patient data and cleans the data. The doctor double-checks the data to obtain the final multimodal data set to obtain model training samples. The multimodal data set can be divided into three sub-data sets (the data of the three sub-data sets are the same). After being processed by the self-attention mechanism layer, they are input into three base learners respectively. The outputs of the three base learners are processed by the weighted voting method and input into the meta-learner. The meta-learner can output the prediction result of whether the user has kidney stones.

[0257] The embodiment of the present application incorporates non-invasive multimodal information and integrated learning technology. The method uses simple, convenient, non-invasive and low-cost fundus images and text information, which can greatly reduce the diagnosis time and cost, while avoiding invasive diagnosis methods. Therefore, the method proposed in this embodiment can perform large-scale early screening for kidney stones more accurately, conveniently and efficiently, and has strong practical value.

[0258] In order to verify the effect of the kidney stone prediction model trained with multimodal data, the following experiments were conducted:

[0259] 1. Data Collection

[0260] The dataset consists of questionnaires, physical examination information forms, fundus examination data from October 2009 to December 2020 in xxx Hospital. Figure 3 Part composition.

[0261] 2. Data labeling

[0262] The doctor diagnoses whether the patient has kidney stones based on the physical examination data and records it, and then performs data cleaning and matching.

[0263] 3. Image preprocessing

[0264] The patient's fundus image and the corresponding mark combination constitute the data set of the present invention. In the model training stage, the data is divided into a training set and a test set according to a ratio of 8:2, wherein the specific data is shown in Table 1:

[0265] Table 1: Experimental data distribution

[0266]

[0267]

[0268] 4. Model training

[0269] The stacking ensemble learning method is constructed using three base learners: VGG-16, Inception-V3, and ResNet-50. The meta-learner uses Logistic regression, and a five-fold cross validation is performed during the training of each base learner. The input image size of the model is 224x224, the batch size is 16, the stochastic gradient descent algorithm is used, and the initial learning rate is 0.0001.

[0270] 5. Experimental results evaluation

[0271] For the experimental results, AUC, ACC, SEN, and SPE are used as evaluation indicators. In the experiment, the training is done on the training set, and the model with the best results on the training set is selected and tested on the test set. The average AUC on the test set is 0.75, and the average SEN is 0.72. The AUC curve is as follows: Fig.13 shown.

[0272] Based on the above experimental analysis, it can be seen that the kidney stone prediction model trained by the method provided in this embodiment has a good prediction effect.

[0273] The kidney stone prediction model training method provided in the embodiment of the present application can reduce the difficulty of data acquisition and thus reduce the difficulty of model training by using the user's non-invasive data associated with kidney stones as model training data. At the same time, the multimodal data formed by combining the fundus image and the user's non-invasive user data associated with kidney stones as training samples to train the kidney stone prediction model can improve the accuracy of kidney stone prediction.

[0274] Reference Fig.14 , shows a schematic diagram of the structure of a disease prediction model training device provided in an embodiment of the present application, such as Fig.14 As shown, the disease prediction model training device 1400 may include the following modules:

[0275] The training sample acquisition module 1410 is used to acquire a model training sample, wherein the model training sample is multimodal data formed by a fundus image of a user and non-invasive user data of the user associated with a preset disease, and the model training sample is annotated with a real disease category;

[0276] A prediction category acquisition module 1420 is used to train the disease prediction model to be trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained;

[0277] A loss value calculation module 1430 is used to calculate the loss value of the disease prediction model to be trained based on the real disease category, the predicted disease category and the weights corresponding to the model training samples;

[0278] The disease prediction model acquisition module 1440 is used to use the trained disease prediction model to be trained as the final disease prediction model for predicting the preset disease when the loss value is within a preset range.

[0279] Optionally, the predicted category acquisition module includes:

[0280] The prediction category acquisition unit is used to train the disease prediction model to be trained using a five-fold cross-validation method and the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained.

[0281] Optionally, the disease prediction model to be trained includes: N base learners and a regression meta-learner, wherein the N base learners are connected in parallel and are all connected to the regression meta-learner, and N is a positive integer greater than or equal to 2.

[0282] The prediction category acquisition module includes:

[0283] An initial feature acquisition unit, used for calling the N base learners to process the multimodal data respectively, to obtain initial feature information associated with the preset disease corresponding to the multimodal data output by the N base learners;

[0284] A target feature acquisition unit, used for processing the initial feature information based on a weighted voting method to obtain target feature information;

[0285] A predicted category output unit is used to call the regression meta-learner to process the target feature information and output the predicted disease category.

[0286] Optionally, the disease prediction model to be trained further includes: a self-attention mechanism layer,

[0287] The initial feature acquisition unit comprises:

[0288] A feature map acquisition subunit is used to call the self-attention mechanism layer to perform attention mechanism learning on the fundus image to obtain an attention feature map corresponding to the fundus image;

[0289] The initial feature acquisition subunit is used to call N of the base learners to process the attention feature map and the non-invasive user data respectively, so as to obtain the initial feature information associated with the preset disease output by the N base learners.

[0290] Optionally, the loss value calculation module includes:

[0291] A sample weight calculation unit, used to calculate the weight of the model training samples of the target category according to the total number of samples, the total number of categories and the number of samples of the target category in this model training;

[0292] A loss value calculation unit is used to calculate the loss value of the disease prediction model to be trained based on the weight, the real disease category and the predicted disease category.

[0293] Optionally, the non-invasive user data includes: daily alcohol consumption, user taste data, body mass index, daily meat consumption and daily cereal food consumption.

[0294] Optionally, the preset diseases include: any one of liver and gallbladder diseases, kidney stones, and gout.

[0295] Optionally, the device further comprises:

[0296] A target data acquisition module, used to acquire target multimodal data of a target user, wherein the target multimodal data is data formed by a target fundus image of the target user and non-invasive data of the target user associated with a preset disease;

[0297] A target data input module, used for inputting the target multimodal data into the disease prediction model;

[0298] A feature information acquisition module, used for calling N base learners to process the target multimodal data respectively, and obtaining feature information associated with the disease output by the N base learners;

[0299] A correlation feature acquisition module, used to process the feature information based on a weighted voting method to obtain disease correlation feature information;

[0300] The prediction result acquisition module is used to call the regression meta-learner to process the disease-related feature information to obtain a prediction result of whether the target user suffers from a preset disease.

[0301] The disease prediction model training device provided in the embodiment of the present application obtains model training samples, and the model training samples are multimodal data formed by the user's fundus image and the user's non-invasive user data associated with the preset disease, and the model training samples are annotated with real disease categories. The disease prediction model to be trained is trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained. Based on the weights corresponding to the real disease category, the predicted disease category and the model training samples, the loss value of the disease prediction model to be trained is calculated. When the loss value is within the preset range, the trained disease prediction model to be trained is used as the final disease prediction model for predicting the preset disease. The embodiment of the present application obtains a disease prediction model by training multimodal data formed by combining the user's fundus image and non-invasive data associated with the preset disease (such as hepatobiliary disease, kidney stones, gout, etc.). The disease prediction model is used to predict the preset disease, which can improve the efficiency and accuracy of disease prediction compared to manual prediction.

[0302] In addition, an embodiment of the present application also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above-mentioned disease prediction model training method when executed by the processor.

[0303] Fig.15 FIG. 1 is a schematic diagram showing the structure of an electronic device 1500 according to an embodiment of the present invention. Fig.15 As shown, the electronic device 1500 includes a central processing unit (CPU) 1501, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1502 or computer program instructions loaded from a storage unit 1508 to a random access memory (RAM) 1503. In the RAM 1503, various programs and data required for the operation of the electronic device 1500 can also be stored. The CPU 1501, the ROM 1502, and the RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0304] Multiple components in the electronic device 1500 are connected to the I / O interface 1505, including: an input unit 1506, such as a keyboard, a mouse, a microphone, etc.; an output unit 1507, such as various types of displays, speakers, etc.; a storage unit 1508, such as a disk, an optical disk, etc.; and a communication unit 1509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1509 allows the electronic device 1500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0305] The various processes and processing described above may be performed by the processing unit 1501. For example, the method of any of the above embodiments may be implemented as a computer software program, which is tangibly contained in a computer-readable medium, such as the storage unit 1508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1500 via the ROM 1502 and / or the communication unit 1509. When the computer program is loaded into the RAM 1503 and executed by the CPU 1501, one or more actions in the method described above may be performed.

[0306] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the training method embodiment of the above-mentioned disease prediction model is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is such as a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0307] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0308] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0309] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0310] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0311] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0312] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0313] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0314] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0315] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0316] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A disease prediction model training method, characterized in that: The method comprises: Acquire a model training sample, where the model training sample is multimodal data formed by a fundus image of a user and non-invasive user data of the user associated with a preset disease, and the model training sample is annotated with a real disease category; Training the disease prediction model to be trained based on the multimodal data, to obtain a predicted disease category output by the disease prediction model to be trained; Based on the real disease category, the predicted disease category and the weights corresponding to the model training samples, a loss value of the disease prediction model to be trained is calculated; When the loss value is within a preset range, the trained disease prediction model to be trained is used as the final disease prediction model for predicting the preset disease.

2. The method according to claim 1, characterized in that The step of training the disease prediction model to be trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained includes: The disease prediction model to be trained is trained using a five-fold cross-validation method and the multimodal data to obtain a predicted disease category output by the disease prediction model to be trained.

3. The method according to claim 1, characterized in that The disease prediction model to be trained includes: N base learners and a regression meta-learner, wherein the N base learners are connected in parallel and are all connected to the regression meta-learner, and N is a positive integer greater than or equal to 2. The step of training the disease prediction model to be trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained includes: Calling N base learners to process the multimodal data respectively, and obtaining initial feature information associated with the preset disease corresponding to the multimodal data output by the N base learners; Processing the initial feature information based on a weighted voting method to obtain target feature information; The regression meta-learner is called to process the target feature information, and output the predicted disease category.

4. The method according to claim 3, characterized in that The disease prediction model to be trained also includes: a self-attention mechanism layer, The calling of the N base learners to process the multimodal data respectively to obtain initial feature information associated with the preset disease corresponding to the multimodal data output by the N base learners includes: Calling the self-attention mechanism layer to perform attention mechanism learning on the fundus image to obtain an attention feature map corresponding to the fundus image; N base learners are called to process the attention feature map and the non-invasive user data respectively, so as to obtain initial feature information associated with the preset disease output by the N base learners.

5. The method according to claim 1, characterized in that The calculating the loss value of the disease prediction model to be trained based on the real disease category, the predicted disease category and the weights corresponding to the model training samples includes: The weight of the model training samples of the target category is calculated based on the total number of samples, the total number of categories and the number of samples of the target category in this model training; Based on the weight, the true disease category and the predicted disease category, a loss value of the disease prediction model to be trained is calculated.

6. The method according to claim 1, characterized in that The non-invasive user data includes: daily alcohol consumption, user taste data, body mass index, daily meat consumption and daily cereal food consumption.

7. The method according to any one of claims 1 to 6, characterized in that The preset diseases include: any one of liver and gallbladder diseases, kidney stones, and gout.

8. The method according to any one of claims 1 to 3, characterized in that After the trained disease prediction model to be trained is used as the final disease prediction model for predicting the preset disease, the method further includes: Acquire target multimodal data of a target user, wherein the target multimodal data is data formed by a target fundus image of the target user and non-invasive data of the target user associated with a preset disease; Inputting the target multimodal data into the disease prediction model; Calling N base learners to process the target multimodal data respectively to obtain feature information associated with the disease output by the N base learners; Processing the characteristic information based on a weighted voting method to obtain disease-related characteristic information; The regression meta-learner is called to process the disease-related feature information to obtain a prediction result of whether the target user suffers from a preset disease.

9. A disease prediction model training device, characterized in that: The device comprises: A training sample acquisition module, used to acquire a model training sample, wherein the model training sample is multimodal data formed by a fundus image of a user and non-invasive user data of the user associated with a preset disease, and the model training sample is annotated with a real disease category; A prediction category acquisition module, used to train the disease prediction model to be trained based on the multimodal data to obtain the predicted disease category output by the disease prediction model to be trained; A loss value calculation module, used to calculate the loss value of the disease prediction model to be trained based on the real disease category, the predicted disease category and the weights corresponding to the model training samples; The disease prediction model acquisition module is used to use the trained disease prediction model to be trained as the final disease prediction model for predicting the preset disease when the loss value is within a preset range.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the disease prediction model training method as described in any one of claims 1 to 8.

11. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the disease prediction model training method described in any one of claims 1 to 8.

Images