Method and system for determining disease occurrence probability based on multimodal assisted learning

Abstract

This invention discloses a method and system for determining the probability of disease occurrence based on multimodal assisted learning, relating to the field of artificial intelligence technology. In this method, real-time medical data of the patient is acquired, including one or more of structured data features, medical text features, and medical image features. The real-time medical data is preprocessed to obtain medical data to be analyzed. This medical data is then input into a trained disease occurrence probability prediction model to obtain a first disease occurrence probability value. The disease occurrence probability prediction model is trained based on a model training dataset and a model training auxiliary task. When the first disease occurrence probability value is greater than a preset disease occurrence threshold, a warning message is output to the physician's terminal. Implementing the technical solution provided in this application can improve the accuracy of predicting the probability of disease occurrence.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a method and system for determining the probability of disease occurrence based on multimodal assisted learning. Background Art

[0002] At present, it is becoming increasingly important to detect the probability of a patient's potential disease as early as possible during hospitalization. Early detection of a disease that a patient may have and timely treatment can effectively increase the probability of the patient's disease being cured.

[0003] Traditional prediction of disease probability is to build a prediction model and analyze the medical data of clinical tests of patients to determine the probability of potential disease occurrence in patients. However, when actually predicting the probability of disease occurrence, the medical data of clinical tests of patients are often single-modal data and are not comprehensive, resulting in low accuracy of disease probability obtained by traditional prediction models when predicting this type of patients.

[0004] Therefore, the inventors believe that there is an urgent need for a method and system for determining the probability of disease occurrence based on multimodal assisted learning. Summary of the invention

[0005] The present application provides a method and system for determining the probability of disease occurrence based on multimodal assisted learning, which can improve the accuracy of predicting the probability value of disease occurrence.

[0006] In a first aspect, the present application provides a method for determining the probability of disease occurrence based on multimodal assisted learning, the method comprising: acquiring real-time medical data of a patient, the real-time medical data comprising one or more of structured data features, medical text features, and medical image features;

[0007] Preprocessing the real-time medical data to obtain medical data to be analyzed;

[0008] Inputting the medical data to be analyzed into a trained disease occurrence probability prediction model to obtain a first disease occurrence probability value, wherein the disease occurrence probability prediction model is trained based on a model training data set and a model training auxiliary task, wherein the model training data set includes preprocessed historical medical data of multiple sample patients and patient labels of multiple sample patients, and the model training auxiliary task is constructed by the feature types contained in the preprocessed historical medical data of multiple sample patients and prior knowledge corresponding to the sample patients, wherein the prior knowledge is obtained based on the patient labels, and the historical medical data includes one or more of structured data features, medical text features, and medical image features;

[0009] When the first disease occurrence probability value is greater than a preset disease occurrence threshold, an early warning prompt message is output to the physician terminal.

[0010] Optionally, before acquiring the real-time medical data of the patient, the method further includes:

[0011] Acquire historical medical data of multiple sample patients and patient labels of multiple sample patients;

[0012] Based on the patient signature, determining a predicted endpoint;

[0013] Acquiring prior knowledge corresponding to the predicted endpoint;

[0014] Constructing a basic data set based on the historical medical data and the patient labels;

[0015] The basic data set is preprocessed to obtain a model training data set.

[0016] Optionally, the disease occurrence probability prediction model is trained based on a model training data set and a model training auxiliary task, specifically including:

[0017] Acquire the model training data set; the model training data set includes preprocessed historical medical data x of multiple sample patients and patient labels y of multiple sample patients; wherein the preprocessed historical medical data x of multiple sample patients includes one or more of structured features x1, medical text features x2, and medical image features x3; the patient labels y∈{0,1} of the multiple sample patients, 0 represents no disease, and 1 represents disease;

[0018] Inputting the model training data set into the initial disease occurrence probability prediction model to obtain the main task loss value;

[0019] Determine the auxiliary task of model training based on the feature types only contained in the preprocessed historical medical data x of the plurality of sample patients and the focus characteristics corresponding to the prior knowledge;

[0020] Training the auxiliary task according to the model obtains an auxiliary task data set X and an auxiliary task label set Y, and obtains an auxiliary task loss value based on the auxiliary task data set X and the auxiliary task label set Y;

[0021] Adding the main task loss value and the auxiliary task loss value to obtain a total loss value;

[0022] Based on the total loss value, the model parameters of the initial disease occurrence probability prediction model are updated using a back propagation algorithm, and the initial disease occurrence probability prediction model is trained for a preset number of rounds to obtain the trained disease occurrence probability prediction model.

[0023] Optionally, the step of inputting the model training data set into the initial disease occurrence probability prediction model to obtain the main task loss value specifically includes:

[0024] Inputting the model training data set into an initial disease occurrence probability prediction model; the initial disease occurrence probability prediction model includes a bidirectional Transformer structure model and a unidirectional Transformer structure model;

[0025] The bidirectional Transformer structure model is selected as an encoder to encode the non-time series data in the structured feature x1, the medical text feature x2, and the medical image feature x3, and the unidirectional Transformer structure model is selected as an encoder to encode the time series data in the structured feature x1, the medical text feature x2, and the medical image feature x3, and the data encoded by the bidirectional Transformer structure model and the unidirectional Transformer structure model are combined to output the second disease occurrence probability value;

[0026] Obtaining a cross entropy loss value based on the second disease occurrence probability value and the patient labels y of the multiple sample patients;

[0027] The cross entropy loss value is used as the main task loss value.

[0028] Optionally, the attention characteristics corresponding to the prior knowledge include a first related characteristic, a second related characteristic, a third related characteristic, a fourth related characteristic, a fifth related characteristic, a sixth related characteristic, and a seventh related characteristic; the determining of the model training auxiliary task based on the feature types only contained in the preprocessed historical medical data x of the plurality of sample patients and the attention characteristics corresponding to the prior knowledge specifically includes:

[0029] When the preprocessed historical medical data of the plurality of sample patients only contain the structural feature x1, and the characteristic of interest corresponding to the prior knowledge is the first relevant characteristic, determining the model training auxiliary task to be the first feature reconstruction auxiliary task;

[0030] When the preprocessed historical medical data of the plurality of sample patients only contain the structural feature x1, and the characteristic of interest corresponding to the prior knowledge is the second related characteristic, determining the model training auxiliary task to be the feature trend prediction auxiliary task;

[0031] When the preprocessed historical medical data of the plurality of sample patients only contain the feature of the medical text feature x2, and the prior knowledge corresponds to the third related characteristic, determining the model training auxiliary task to be the feature position identification auxiliary task;

[0032] When the preprocessed historical medical data of the plurality of sample patients only contain the medical image feature x3, and the characteristic of interest corresponding to the prior knowledge is the fourth related characteristic, determining that the model training auxiliary task is the second feature reconstruction auxiliary task;

[0033] When the preprocessed historical medical data of the plurality of sample patients only contain the structural feature x1 and the medical text feature x2, and the characteristic of interest corresponding to the prior knowledge is the fifth related characteristic, determining that the model training auxiliary task is the characteristic correlation judgment auxiliary task;

[0034] When the preprocessed historical medical data of the plurality of sample patients only contain the medical text feature x2 and the medical image feature x3, and the characteristic of interest corresponding to the prior knowledge is the sixth related characteristic, determining that the model training auxiliary task is the first feature transformation generation auxiliary task;

[0035] When the preprocessed historical medical data of the multiple sample patients only contain the structural feature x1 and the medical image feature x3, and the focus characteristic corresponding to the prior knowledge is the seventh related characteristic, the model training auxiliary task is determined to be the second feature transformation generation auxiliary task.

[0036] Optionally, the first related characteristic is to focus on the value of a specific test indicator, the second related characteristic is to focus on the changing trend of a specific test indicator, the third related characteristic is to focus on a preset part in the text, the fourth related characteristic is to focus on a preset part in the image result, the fifth related characteristic is to focus on whether the diagnosis and treatment conclusion is related to the specific test indicator, the sixth related characteristic is to focus on whether the medical imaging feature can obtain a specific conclusion description, and the seventh related characteristic is to focus on the size or area of ​​the medical imaging feature.

[0037] Optionally, the auxiliary task data set X includes a first reconstructed feature x1', a second reconstructed feature x3' and the medical text feature x2; the auxiliary task label set Y includes a first newly created label, a second newly created label, a third newly created label, a fourth newly created label, a fifth newly created label, a sixth newly created label, a seventh newly created label and an eighth newly created label; the auxiliary task data set X and the auxiliary task label set Y are obtained according to the model training auxiliary task, and the auxiliary task loss value is obtained based on the auxiliary task data set X and the auxiliary task label set Y, specifically including:

[0038] Reconstructing the auxiliary task based on the first feature, obtaining the first reconstructed feature x1', the first newly created label y1, and the second newly created label y2, and obtaining a first loss value based on the first reconstructed feature x1', the first newly created label y1, and the second newly created label y2;

[0039] Based on the feature trend prediction auxiliary task, obtaining the third newly created label y3, and obtaining a second loss value based on the third newly created label y3;

[0040] Based on the feature position recognition auxiliary task, obtaining the fourth new label y4, and obtaining a third loss value based on the fourth new label y4;

[0041] Reconstructing the auxiliary task based on the second feature to obtain the second reconstructed feature x3' and the fifth newly created label y5, and obtaining a fourth loss value based on the second reconstructed feature x3' and the fifth newly created label y5;

[0042] Determine the auxiliary task based on the feature correlation, obtain the sixth new label y6, and obtain the fifth loss value based on the sixth new label y6;

[0043] Generate an auxiliary task based on the first feature transformation to obtain the seventh new label y7, and obtain a sixth loss value based on the seventh new label y7;

[0044] Generate an auxiliary task based on the second feature transformation to obtain the eighth new label y8, and obtain a seventh loss value based on the eighth new label y8;

[0045] The first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, the sixth loss value and the seventh loss value are added to obtain the auxiliary task loss value.

[0046] Optionally, reconstructing the auxiliary task based on the first feature to obtain the first reconstructed feature x1', the first newly created label y1, and the second newly created label y2, and obtaining a first loss value based on the first reconstructed feature x1', the first newly created label y1, and the second newly created label y2, specifically includes:

[0047] Based on the first feature reconstruction auxiliary task, the specific test index x in all the structured features x1 is 1'Add random noise disturbance to obtain the first reconstructed feature x1', as well as the first new label y1 and the second new label y2 corresponding to the first reconstructed feature x1'; where y1∈{0,1}, 0 represents the feature without adding random noise disturbance, and 1 represents the feature with adding random noise disturbance; y2 is a continuous value, indicating the value of the feature before adding random noise disturbance;

[0048] Calculate the function loss value of the first newly created label y1 based on the cross entropy loss function to obtain a first reconstruction loss value;

[0049] Calculate the function loss value of the second newly created label y2 based on the mean square error loss function to obtain a second reconstruction loss value;

[0050] The first reconstruction loss value is added to the second reconstruction loss value to obtain a first loss value.

[0051] Optionally, obtaining the third new label y3 based on the feature trend prediction auxiliary task, and obtaining the second loss value based on the third new label y3 specifically includes:

[0052] Based on the feature trend prediction auxiliary task, determine the time series data in the structured feature x1; the time series data includes n time steps, where n is a positive integer greater than 1;

[0053] Based on the specific assay indicators corresponding to the first m time steps among the n time steps, predict the specific assay indicators corresponding to the m+1th time step;

[0054] The specific test index corresponding to the m+1th time step is used as the third newly created label y3;

[0055] The function loss value of the third newly created label y3 is calculated based on the mean square error loss function to obtain a second loss value.

[0056] Optionally, obtaining the fourth new label y4 based on the feature position recognition auxiliary task, and obtaining the third loss value based on the fourth new label y4 specifically includes:

[0057] Based on the feature position recognition auxiliary task, construct the fourth new label y4; wherein y4∈{0,1,2}, 0 represents a non-starting or ending position, 1 represents a starting position, and 2 represents an ending position;

[0058] The function loss value of the fourth new label y4 is calculated based on the cross entropy loss function to obtain the third loss value.

[0059] Optionally, reconstructing the auxiliary task based on the second feature to obtain the second reconstructed feature x3' and the fifth newly created label y5, and obtaining a fourth loss value based on the second reconstructed feature x3' and the fifth newly created label y5 specifically includes:

[0060] Based on the second feature reconstruction auxiliary task, a preset proportion of pixels in the target area corresponding to the preset target area in the medical image feature x3 are masked to obtain the second reconstructed feature x3' and the pixel values ​​of the masked replaced part;

[0061] Using the pixel value of the mask replacement part as the fifth newly created label y5;

[0062] The function loss value of the fifth new label y5 is calculated based on the mean square error loss function to obtain the fourth loss value.

[0063] Optionally, the determining the auxiliary task based on the feature correlation to obtain the sixth new label y6, and obtaining the fifth loss value based on the sixth new label y6, specifically includes:

[0064] Based on the auxiliary task of judging the correlation of the characteristics, and according to the list of diagnosis and treatment conclusions corresponding to the prior knowledge, the sixth newly created label y6 is constructed, where y6∈{0,1}, 0 represents a non-relevant indicator, and 1 represents a relevant indicator;

[0065] The function loss value of the sixth new label y6 is calculated based on the cross entropy loss function to obtain the fifth loss value.

[0066] Optionally, the generating an auxiliary task based on the first feature transformation to obtain the seventh new label y7, and obtaining a sixth loss value based on the seventh new label y7 specifically includes:

[0067] Using the medical text feature x2 as the seventh newly created label y7;

[0068] The function loss value of the seventh new label y7 is calculated based on the cross entropy loss function to obtain the sixth loss value.

[0069] Optionally, the generating an auxiliary task based on the second feature transformation to obtain the eighth new label y8, and obtaining a seventh loss value based on the eighth new label y8 specifically includes:

[0070] Using the structured feature x1 as the eighth newly created label y8;

[0071] The function loss value of the eighth new label y8 is calculated based on the mean square error loss function to obtain the seventh loss value.

[0072] In a second aspect of the present application, a disease occurrence probability determination system based on multimodal assisted learning is provided, the system comprising: a first acquisition module, a data preprocessing module, a processing module and an early warning module; the first acquisition module is used to acquire real-time medical data of a patient, the real-time medical data comprising one or more of structured data features, medical text features and medical image features; the data preprocessing module is used to preprocess the real-time medical data to obtain medical data to be analyzed; the processing module is used to input the medical data to be analyzed into a trained disease occurrence probability prediction model to obtain a first disease occurrence probability value, the disease occurrence probability prediction model is trained based on a model training data set and a model training auxiliary task, the model training data set comprises preprocessed historical medical data of a plurality of sample patients and patient labels of a plurality of sample patients, the model training auxiliary task is constructed by the feature types contained in the preprocessed historical medical data of the plurality of sample patients and the prior knowledge corresponding to the sample patients, the prior knowledge is obtained based on the patient label, the historical medical data comprises one or more of structured data features, medical text features and medical image features; the early warning module is used to output early warning prompt information to the physician terminal when the first disease occurrence probability value is greater than a preset disease occurrence threshold.

[0073] Optionally, the system also includes: a second acquisition module, a prediction module, a model building module and an auxiliary task building module; the second acquisition module is used to acquire historical medical data of multiple sample patients and patient labels of multiple sample patients; the prediction module is used to determine the prediction endpoint based on the patient label; the second acquisition module is also used to acquire prior knowledge corresponding to the prediction endpoint; the model building module is used to build a basic data set based on the historical medical data and the patient label; the preprocessing module is used to acquire the model training data set; the model training data set includes preprocessed historical medical data x of multiple sample patients and patient labels y of multiple sample patients; wherein the historical medical data x of the multiple sample patients includes one or more of structured features x1, medical text features x2 and medical imaging features x3; the patient labels y∈{0,1} of the multiple sample patients, 0 represents no disease, 1 represents disease; the model The construction module is further used to input the model training data set into the initial disease occurrence probability prediction model to obtain the main task loss value; the auxiliary task construction module is used to determine the model training auxiliary task based on the feature types only contained in the preprocessed historical medical data x of multiple sample patients and the focus characteristics corresponding to the prior knowledge; the auxiliary task construction module is further used to obtain the auxiliary task data set X and the auxiliary task label set Y according to the model training auxiliary task, and obtain the auxiliary task loss value based on the auxiliary task data set X and the auxiliary task label set Y; the model construction module is further used to add the main task loss value and the auxiliary task loss value to obtain the total loss value; the model construction module is further used to update the model parameters of the initial disease occurrence probability prediction model based on the total loss value using the back propagation algorithm, and train the initial disease occurrence probability prediction model for a preset number of rounds to obtain the trained disease occurrence probability prediction model.

[0074] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any method described in the first aspect of the present application.

[0075] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, a computer program executes the method as described in any one of the first aspect of the present application.

[0076] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0077] By combining structured data, medical text and image features, the disease occurrence probability prediction model can use multimodal data to make up for the problem of insufficient information when judging single modal information, analyze and understand disease-related information from multiple perspectives, and thus improve the accuracy of disease identification and prediction. By adding model training auxiliary tasks on the basis of the model training main tasks corresponding to the disease occurrence probability prediction model, the disease occurrence probability prediction model can deeply dig out the key information in each type of data through the model training main tasks, and introduce prior knowledge into the disease occurrence probability prediction model through model training auxiliary tasks, guide model training, and define the focus of model learning. The implementation of different auxiliary tasks helps to effectively integrate medical data of different modalities, provide a more comprehensive perspective for disease occurrence probability prediction, and thus improve the accuracy of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is one of the flow charts of a method for determining the probability of disease occurrence based on multimodal assisted learning provided in an embodiment of the present application;

[0079] Figure 2 This is the second flow chart of a method for determining the probability of disease occurrence based on multimodal assisted learning provided in an embodiment of the present application;

[0080] Figure 3 This is a flowchart of a method for determining the probability of disease occurrence based on multimodal assisted learning provided in an embodiment of the present application;

[0081] Figure 4 This is a fourth flow chart of a method for determining the probability of disease occurrence based on multimodal assisted learning provided in an embodiment of the present application;

[0082] Figure 5 It is a structural schematic diagram of a disease occurrence probability determination system based on multimodal assisted learning disclosed in an embodiment of the present application;

[0083] Figure 6 It is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application.

[0084] Explanation of the accompanying drawings: 1. First acquisition module; 2. Data preprocessing module; 3. Processing module; 4. Early warning module; 5. Second acquisition module; 6. Prediction module; 7. Model building module; 8. Auxiliary task building module; 600. Electronic device; 601. Processor; 602. Communication bus; 603. User interface; 604. Network interface; 605. Memory. DETAILED DESCRIPTION

[0085] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0086] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0087] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0088] This application provides a method for determining the probability of disease occurrence based on multimodal assisted learning, referring to Figure 1 , Figure 1 This is one of the flow charts of a method for determining the probability of disease occurrence based on multimodal assisted learning disclosed in an embodiment of the present application. The method for determining the probability of disease occurrence based on multimodal assisted learning is applied to a system for determining the probability of disease occurrence based on multimodal assisted learning. The method includes steps S11 to S14, which are as follows:

[0089] Step S11: Acquire the patient's real-time medical data, where the real-time medical data includes one or more of structured data features, medical text features, and medical image features.

[0090] Specifically, in the present technical solution, the disease occurrence probability determination system based on multimodal assisted learning obtains the patient's real-time medical data. That is, the disease occurrence probability determination system based on multimodal assisted learning first collects the patient's initial real-time medical data, wherein the initial real-time medical data is multimodal data, and the multimodal data includes three modal data: structured data, medical text, and medical image. Structured data is data information such as test results or measurement indicators; medical text is text information such as diagnosis results issued by doctors; medical images are image information such as ultrasound images and CT images. Features are extracted from structured data, medical text, and medical images to obtain structured data features x1, medical text features x2, and medical image features x3, and the structured data features x1, medical text features x2, and medical image features x3 are combined into real-time medical data.

[0091] Among them, the structured data feature x1, the medical text feature x2, and the medical image feature x3 all contain several sub-features. For example, the sub-features contained in the structured data feature x1 include but are not limited to blood test data features and urine test data features; the sub-features contained in the medical text feature x2 include but are not limited to current medical history features, ultrasound test result features, and CT test result features; the sub-features contained in the medical image feature x3 include but are not limited to chest X-ray image features and cardiac ultrasound image features.

[0092] Step S12: pre-process the real-time medical data to obtain medical data to be analyzed.

[0093] Specifically, in the present technical solution, a disease occurrence probability determination system based on multimodal assisted learning preprocesses real-time medical data to obtain medical data to be analyzed.

[0094] The specific method for preprocessing real-time medical data is: preprocessing the structured data feature x1 using one or both of the fitting result filling and mode filling methods; performing text preprocessing on the medical text feature x2; and performing standardization on the medical image feature x3.

[0095] Filling with fitting results refers to training a regression model based on existing data, with the prediction target being the missing data column and the input being other data columns. For example, suppose the data is divided into 3 features, A, B, and C, of ​​which feature B is missing. First, take data in which A, B, and C are not missing to train the regression model, take A and C as input, and obtain the predicted value of the regression model as the filling value of B. Mode filling refers to using the value with the most occurrences in the data as the filling value. For example, for any feature, there is 1 null value in 10 data, 8 values ​​in the other 9 data are 0, and 1 value is 4, then the filling value is 0. In the present technical solution, for example, in a blood test, three structured data sub-features of red blood cells, white blood cells, and platelets are included, of which the platelet feature is missing. Using the method of filling with fitting results, first use the data with complete features of red blood cells, white blood cells, and platelets to train the regression model, take the features of red blood cells and white blood cells in the data to be completed as input values, and obtain the filling value of platelets. The medical text feature x2 is processed by text noise removal, that is, garbled characters and irregular symbols in the text are removed, such as half-width / full-width conversion, / n / t and other line breaks; the medical image feature x3 is processed by standardization, that is, the pixel values ​​in medical images such as X-rays, color ultrasound and other images are processed using the formula (pixel value-pixel value mean) / pixel value standard deviation.

[0096] Step S13: input the medical data to be analyzed into the trained disease occurrence probability prediction model to obtain a first disease occurrence probability value. The disease occurrence probability prediction model is trained based on a model training data set and a model training auxiliary task. The model training data set includes pre-processed historical medical data of multiple sample patients and patient labels of multiple sample patients. The model training auxiliary task is constructed by the feature types contained in the pre-processed historical medical data of multiple sample patients and the prior knowledge corresponding to the sample patients. The prior knowledge is obtained based on the patient labels. The historical medical data includes one or more of structured data features, medical text features, and medical imaging features.

[0097] Specifically, in the present technical solution, the multimodal assisted learning disease occurrence probability determination system inputs the medical data to be analyzed into the trained disease occurrence probability prediction model to obtain a first disease occurrence probability value.

[0098] The disease occurrence probability prediction model is trained based on historical medical data, patient labels, and model training auxiliary tasks. The model training auxiliary tasks are used to introduce prior knowledge and guide the training direction of the disease occurrence probability prediction model. The model training auxiliary tasks are constructed by the feature types contained in the historical medical data of multiple sample patients that have been preprocessed and the prior knowledge corresponding to the sample patients, where the prior knowledge is the knowledge collected and created by professional technicians in this field from patient labels.

[0099] The specific method of constructing a disease occurrence probability prediction model will be described in detail in subsequent embodiments. Similarly, the specific method of creating prior knowledge will be described in detail in subsequent embodiments, so it will not be described in detail here.

[0100] Step S14: When the first disease occurrence probability value is greater than a preset disease occurrence threshold, output warning prompt information to the physician terminal.

[0101] Specifically, in this technical solution, when the disease probability determination system based on multimodal assisted learning determines that the first disease probability value is greater than the preset disease occurrence threshold, the warning prompt information is output to the physician terminal. Among them, the physician terminal includes but is not limited to electronic devices such as computers, mobile phones, and tablets used by physicians. It should be noted that the output warning prompt information is only used to provide reference opinions for physicians, and physicians cannot make the final diagnosis decision based on the warning prompt information provided by the system. The warning prompt information is intended to assist doctors in identifying potential risks more quickly, rather than replacing the professional judgment of doctors.

[0102] Before step S11, it is also necessary to set a preliminary step for building a model training data set. The preliminary step is to prepare and process historical medical data for use in building and training a disease occurrence probability prediction model.

[0103] Therefore, in a possible implementation manner, before step S11, the method further includes the following steps:

[0104] Historical medical data of multiple sample patients and patient labels of multiple sample patients are obtained.

[0105] Specifically, in the present technical solution, the disease occurrence probability determination system based on multimodal assisted learning will collect historical medical data of multiple sample patients and patient labels corresponding to each sample patient; these historical medical data include but are not limited to medical history, medical records, laboratory test results and other data. Similarly, historical medical data include one or more of structured data features, medical text features and medical imaging features. The patient label of the sample patient is obtained according to the clinical standard corresponding to the disease to be predicted, and the patient label y∈{0,1}, 0 represents no disease, and 1 represents disease. For example, according to the clinical standard corresponding to acute kidney injury, the result of whether the sample patient suffers from acute kidney injury is obtained, and the result of whether the sample patient suffers from acute kidney injury is used as the patient label.

[0106] Based on the patient labels, predictive endpoints were determined.

[0107] Specifically, in this technical solution, the prediction endpoint will be determined based on the patient label, which will help guide the direction and focus of the model training.

[0108] For example, if the patient label is whether he / she suffers from acute kidney injury, the predicted endpoint is AKI detection; if the patient label is whether he / she suffers from intravascular embolic disease, the predicted endpoint is warfarin dosage prediction; if the patient label is whether he / she suffers from renal vascular hypertension, the predicted endpoint is renal vascular hypertension prediction. The predicted endpoint is a method for professionals in this field to determine the predicted disease or indicator based on the patient label.

[0109] Acquire prior knowledge corresponding to the predicted endpoint.

[0110] Specifically, in the present technical solution, in the aforementioned embodiments, the prior knowledge is the knowledge created by professional and technical personnel in this field from patient labels. Furthermore, professional and technical personnel in this field also need to first determine the predicted endpoint of the disease and obtain the corresponding prior knowledge based on the predicted endpoint of the disease and the patient labels.

[0111] For example: when the prediction endpoint is AKI prediction, the prior knowledge corresponding to the prediction endpoint can be: the patient's postoperative creatinine value; when the prediction endpoint is warfarin dosage prediction, the prior knowledge corresponding to the prediction endpoint can be the patient's last or last few INR measurement results and medication information; when the prediction endpoint is renal vascular hypertension prediction, the prior knowledge corresponding to the prediction endpoint can be: the conclusion of renal CT.

[0112] Build a basic dataset based on historical medical data and patient labels.

[0113] Specifically, in this technical solution, the multimodal assisted learning disease probability determination system constructs a basic data set based on historical medical data and patient labels. The constructed basic data set will serve as the initial source for model training, and it is necessary to ensure the diversity and representativeness of the basic data set so that the disease probability prediction model can learn different types of cases and situations.

[0114] The basic data set is preprocessed to obtain the model training data set.

[0115] Specifically, in this technical solution, the basic data set is preprocessed to obtain the model training data set. The method for preprocessing the basic data set here is consistent with the method for preprocessing the real-time medical data in the above embodiment, so it will not be described in detail here.

[0116] Since the disease occurrence probability prediction model is trained based on the model training data set and the model training auxiliary tasks, the disease occurrence probability prediction model can be trained specifically in the following way, that is, referring to Figure 2 The method shown in. Wherein, Figure 2This is the second flow chart of a method for determining the probability of disease occurrence based on multimodal assisted learning disclosed in an embodiment of the present application.

[0117] In a possible implementation, the method for training a disease occurrence probability prediction model specifically includes steps S21-S26:

[0118] Step S21: Obtain a model training data set; the model training data set includes preprocessed historical medical data x of multiple sample patients and patient labels y of multiple sample patients; wherein the preprocessed historical medical data x of multiple sample patients includes one or more of structured features x1, medical text features x2, and medical imaging features x3; the patient labels y∈{0,1} of multiple sample patients, 0 represents no disease, and 1 represents disease.

[0119] Specifically, in this technical solution, the multimodal assisted learning disease occurrence probability determination system will obtain the model training data set obtained in the above-mentioned embodiment. Further, the model training data set includes preprocessed historical medical data x of multiple sample patients and patient labels y of multiple sample patients; the preprocessed historical medical data of each sample patient includes one or more of structured features x1, medical text features x2, and medical imaging features x3; the patient labels y∈{0,1} of multiple sample patients, 0 represents no disease, 1 represents disease.

[0120] Step S22: Input the model training data set into the initial disease occurrence probability prediction model to obtain the main task loss value.

[0121] Specifically, in this technical solution, the multimodal assisted learning disease probability determination system inputs the model training data set into the initial disease probability prediction model for training. The training process is the main task, and the main task loss value is obtained. The main task loss value is a key indicator to measure the accuracy of model prediction and is used to optimize the disease probability prediction model.

[0122] In the subsequent embodiments, the specific method of inputting the model training data set into the initial disease occurrence probability prediction model to obtain the main task loss value will be described in detail.

[0123] Step S23: Determine the auxiliary tasks for model training based on the feature types only contained in the preprocessed historical medical data x of multiple sample patients and the focus characteristics corresponding to the prior knowledge.

[0124] Specifically, in this technical solution, the disease occurrence probability determination system of multimodal assisted learning determines the auxiliary tasks of model training based on the feature types and the attention characteristics corresponding to the prior knowledge only contained in the historical medical data x of multiple sample patients that have been preprocessed. These auxiliary tasks are to introduce prior knowledge and enhance the learning ability of the model, especially the prediction ability in specific aspects.

[0125] In subsequent embodiments, a specific method for determining the auxiliary task of model training based on the feature types only contained in the preprocessed historical medical data x of multiple sample patients and the focus characteristics corresponding to the prior knowledge will be described in detail.

[0126] Step S24: obtaining an auxiliary task dataset X and an auxiliary task label set Y according to the model training auxiliary task, and obtaining an auxiliary task loss value based on the auxiliary task dataset X and the auxiliary task label set Y.

[0127] Specifically, in this technical solution, the disease occurrence probability determination system of multimodal assisted learning will obtain the auxiliary task data set X and the auxiliary task label set Y according to the model training auxiliary task, and obtain the auxiliary task loss value based on the auxiliary task data set X and the auxiliary task label set Y. The calculated loss value of the auxiliary task helps to evaluate the performance of the model on the auxiliary task, so as to further optimize the disease occurrence probability prediction model.

[0128] In subsequent embodiments, a specific method for obtaining an auxiliary task dataset X and an auxiliary task label set Y according to model training auxiliary tasks, and obtaining an auxiliary task loss value based on the auxiliary task dataset X and the auxiliary task label set Y will be described in detail.

[0129] Step S25: Add the main task loss value and the auxiliary task loss value to obtain the total loss value.

[0130] Specifically, in this technical solution, the disease occurrence probability determination system of multimodal assisted learning adds the main task loss value and the auxiliary task loss value to obtain the total loss value of the model. This total loss value is a key indicator for evaluating model performance and is used to guide model optimization. When the total loss value is less than the preset threshold, the model stops updating the parameter value and obtains the trained disease occurrence probability prediction model.

[0131] Step S26: Based on the total loss value, the model parameters of the initial disease occurrence probability prediction model are updated using the back propagation algorithm, and the initial disease occurrence probability prediction model is trained for a preset number of rounds to obtain a trained disease occurrence probability prediction model.

[0132] Specifically, in the present technical solution, the multimodal assisted learning disease probability determination system uses a back propagation algorithm to update the parameters of the initial disease probability prediction model based on the total loss value. The use of the back propagation algorithm is the key to optimizing the model parameters, and the initial disease probability prediction model is trained for a preset number of rounds, which ensures that the model can continuously learn from the loss value and adjust the parameters, thereby continuously improving the accuracy of the prediction. Among them, the preset number of rounds will be set by professional and technical personnel based on historical experience data according to the requirements for the accuracy of the model prediction. Therefore, the preset number of rounds is not specifically limited in this application.

[0133] In terms of steps S21-S26, the disease occurrence probability determination system of multimodal assisted learning is based on the pre-processed historical medical data of multiple sample patients, the patient labels of multiple sample patients, the auxiliary task data set, the auxiliary task label set, the main task loss value and the auxiliary task loss value, to construct a trained disease occurrence probability prediction model, which comprehensively considers the results of the main task and the auxiliary task, wherein the main task allows the model to use structured, textual, image and other multimodal medical data, and through effective fusion and association of multimodal data, to mine valuable information, and make up for the lack of information in the judgment of single modal information, so as to achieve the purpose of improving the accuracy of the disease occurrence probability prediction model. Among them, the auxiliary task introduces prior knowledge into the model, guides the model training direction, and allows the model to learn data from different angles. Each auxiliary task focuses on different aspects of the data, such as feature trends, location recognition or feature reconstruction, so that the disease occurrence probability prediction model can better understand and process various types of medical data, thereby constructing a disease occurrence probability prediction model with higher accuracy and robustness, that is, a trained disease occurrence probability prediction model.

[0134] In a possible implementation, step S22 specifically includes the following steps:

[0135] The model training data set is input into the initial disease occurrence probability prediction model; the initial disease occurrence probability prediction model includes a bidirectional Transformer structure model and a unidirectional Transformer structure model.

[0136] Specifically, in this technical solution, the multimodal assisted learning disease probability determination system inputs the model training data set into the initial disease probability prediction model. The initial disease probability prediction model includes a bidirectional Transformer structure model and a unidirectional Transformer structure model. Both models are based on the Transformer architecture, which is very effective in processing sequence data, especially in the field of natural language processing.

[0137] A bidirectional Transformer structure model is selected as the encoder to encode the non-time series data in the structured feature x1, medical text feature x2 and medical image feature x3, and a unidirectional Transformer structure model is selected as the encoder to encode the time series data in the structured feature x1, medical text feature x2 and medical image feature x3 to output the probability value of the second disease occurrence.

[0138] Specifically, in this technical solution, the multimodal assisted learning disease probability determination system uses a bidirectional Transformer structure model as an encoder to encode non-time series data in structured features, medical text features, and medical image features. Since the bidirectional Transformer structure model can simultaneously consider the elements before and after each element in the sequence, the bidirectional Transformer structure model is suitable for understanding complex relationships in non-time series data.

[0139] The multimodal assisted learning disease probability determination system will also use the unidirectional Transformer structure model as an encoder to encode the time series data in the structured features, medical text features, and medical image features, and combine the data encoded by the bidirectional Transformer structure model and the unidirectional Transformer structure model to output the second disease probability value. Since the unidirectional Transformer structure model can only consider the elements before (or after, depending on the specific implementation) the elements in the sequence, the unidirectional Transformer structure model is suitable for time series data analysis.

[0140] Based on the second disease occurrence probability value and the patient labels y of the multiple sample patients, a cross entropy loss value is obtained.

[0141] Specifically, in the present technical solution, the multimodal assisted learning disease occurrence probability determination system obtains a cross entropy loss value based on the second disease occurrence probability value and patient labels of multiple sample patients.

[0142] Cross-Entropy Loss is a loss function commonly used in binary classification problems. It measures the difference between the probability distribution of model predictions and the probability distribution of actual labels.

[0143] For a binary classification problem, the cross entropy loss can be expressed as:

[0144] ; where N is the number of samples, y i is the patient label of the i-th sample patient, p i is the predicted probability value corresponding to the patient label of the i-th sample patient.

[0145] These probability values ​​(denoted as p) are the probabilities of disease occurrence predicted by the disease occurrence probability prediction model based on input features (such as structured features, medical text features, and medical image features). For each sample patient's patient label, the multimodal assisted learning disease occurrence probability determination system will calculate the corresponding cross entropy loss, add the cross entropy losses of all samples, and divide by the total number of samples N to obtain the overall average cross entropy loss. If the model's predicted probability is close to the actual label (that is, for diseased samples, the predicted probability is close to 1, and for non-diseased samples, the predicted probability is close to 0), the cross entropy loss will be small. On the contrary, if the model's predicted probability is significantly different from the actual label, the loss will be larger.

[0146] The cross entropy loss value is used as the main task loss value.

[0147] Specifically, in the present technical solution, the multimodal assisted learning disease occurrence probability determination system regards the calculated cross entropy loss value as the loss value of the main task, and uses it for subsequent optimization and adjustment of the disease occurrence probability prediction model.

[0148] In one possible implementation, refer to Figure 3 , Figure 3 This is the third flow chart of a method for determining the probability of disease occurrence based on multimodal assisted learning disclosed in an embodiment of the present application. Since the characteristics of interest corresponding to the prior knowledge include the first relevant characteristic, the second relevant characteristic, the third relevant characteristic, the fourth relevant characteristic, the fifth relevant characteristic, the sixth relevant characteristic and the seventh relevant characteristic, step S23 specifically includes steps S231-S237:

[0149] Step S231: When the preprocessed historical medical data of multiple sample patients only contain the structural feature x1, and the focus feature corresponding to the prior knowledge is the first relevant feature, the model training auxiliary task is determined to be the first feature reconstruction auxiliary task.

[0150] Specifically, in the present technical solution, when the historical medical data of multiple sample patients that have been preprocessed only contain structured features, that is, the historical medical data of multiple sample patients that have been preprocessed only contain structured features such as age, blood pressure readings, laboratory test results, and the characteristic of interest corresponding to the prior knowledge is the first relevant characteristic, the disease occurrence probability determination system of multimodal assisted learning can determine the model training auxiliary task as the first feature reconstruction auxiliary task. By performing the first feature reconstruction auxiliary task, feature reconstruction usually involves dimensionality reduction technology, such as principal component analysis (PCA) or autoencoder, which can extract the most informative features from a large amount of data, remove redundant and irrelevant information, so that the disease occurrence probability prediction model can more effectively understand and explain the complex patterns and associations between structured features, thereby enhancing the performance of the main prediction task (disease occurrence probability prediction).

[0151] Step S232: when the preprocessed historical medical data of multiple sample patients only contain the structural feature x1, and the characteristic of interest corresponding to the prior knowledge is the second related characteristic, determining the model training auxiliary task as the feature trend prediction auxiliary task;

[0152] Specifically, in this technical solution, when the preprocessed historical medical data of multiple sample patients only contains the structured feature x1, and the characteristic of interest corresponding to the prior knowledge is the second related characteristic, the multimodal assisted learning disease probability determination system can determine the model training auxiliary task as the feature trend prediction auxiliary task. By performing the feature trend prediction auxiliary task, the disease probability prediction model can more effectively analyze and predict time series trends or patterns in structured data.

[0153] Step S233: When the preprocessed historical medical data of multiple sample patients only contain the feature of the medical text feature x2 and the third related characteristic corresponding to the prior knowledge, the model training auxiliary task is determined to be the feature position recognition auxiliary task.

[0154] When the preprocessed historical medical data of multiple sample patients only contain the feature of medical text feature x2, that is, the preprocessed historical medical data of multiple sample patients only contain key information points or specific medical terms, and the feature of interest corresponding to the prior knowledge is the third related feature, the multimodal assisted learning disease probability determination system can determine the model training auxiliary task as the feature position identification auxiliary task. By performing the feature position identification auxiliary task, the disease occurrence probability prediction model can more effectively identify the specific location and context of symptom descriptions, diagnostic information or treatment plans in medical texts to provide deeper insights into the patient's health status.

[0155] Step S234: When the preprocessed historical medical data of multiple sample patients only contain the medical imaging feature x3, and the characteristic of interest corresponding to the prior knowledge is the fourth related characteristic, the model training auxiliary task is determined to be the second feature reconstruction auxiliary task.

[0156] Specifically, in this technical solution, when the historical medical data of multiple sample patients that have been preprocessed only contain features that are medical imaging features, that is, the historical medical data of multiple sample patients that have been preprocessed only contain various medical imaging materials, such as X-rays, CT scans, MRI, etc., and the characteristic of interest corresponding to the prior knowledge is the fourth related characteristic, the disease occurrence probability determination system of multimodal assisted learning can determine the model training auxiliary task as the second feature reconstruction auxiliary task. Image data usually contains rich information, but not all information is directly related or helpful for disease diagnosis. Because feature reconstruction can reveal hidden patterns or structures in the image, such as the shape and size of the tumor, or the texture features of the tissue, these may not be observed by the naked eye. And because medical imaging data is usually high-dimensional, direct analysis may be very complicated. Feature reconstruction, such as by using deep learning techniques such as convolutional neural networks (CNNs), can effectively reduce the data dimension while retaining key information. Therefore, by performing the second feature reconstruction auxiliary task, the disease occurrence probability prediction model can more effectively understand and interpret medical imaging features to identify key structures or abnormalities in the image.

[0157] Step S235: When the preprocessed historical medical data of multiple sample patients only contain structural features x1 and medical text features x2, and the focus feature corresponding to the prior knowledge is the fifth related feature, the model training auxiliary task is determined to be a feature correlation judgment auxiliary task.

[0158] Specifically, in this technical solution, when the historical medical data of multiple sample patients that have been preprocessed only contain structural features and medical text features, and the feature of interest corresponding to the prior knowledge is the fifth related feature, the multimodal assisted learning disease probability determination system can determine the model training auxiliary task as the feature correlation judgment auxiliary task. By performing the correlation judgment auxiliary task, the disease occurrence probability prediction model can more effectively understand the relationship between the two types of data, structured features and medical text features, so as to deeply understand complex medical situations. For example, if the main task is to predict the probability of heart disease, the auxiliary task can be to analyze the relationship between specific blood pressure or cholesterol levels (structured features) and the description of specific symptoms in the medical record (medical text features).

[0159] Step S236: When the preprocessed historical medical data of multiple sample patients only contain the medical text feature x2 and the medical image feature x3, and the focus feature corresponding to the prior knowledge is the sixth related feature, the model training auxiliary task is determined to be the first feature transformation generation auxiliary task.

[0160] Specifically, in the present technical solution, when the historical medical data of multiple sample patients that have been preprocessed only contain features that are medical text features and medical image features, and the feature of interest corresponding to the prior knowledge is the sixth related feature, the multimodal assisted learning disease occurrence probability determination system can determine the model training auxiliary task as the first feature transformation to generate an auxiliary task. By performing the first feature transformation to generate auxiliary tasks, the disease occurrence probability prediction model can more effectively combine the visual patterns in the image data with the descriptive information in the text data, thereby enabling an in-depth understanding of complex medical situations. For example, if the main task is to predict the probability of occurrence of a particular cancer, the auxiliary task can be to combine the imaging tumor features with the clinical symptom description in the medical record to generate a clinical symptom description text that describes the imaging tumor features, so as to provide a more comprehensive disease portrait.

[0161] Step S237: When the preprocessed historical medical data of multiple sample patients only contain structural features x1 and medical imaging features x3, and the focus feature corresponding to the prior knowledge is the seventh related feature, the model training auxiliary task is determined to be the second feature transformation generation auxiliary task.

[0162] Specifically, in this technical solution, when the pre-processed historical medical data of multiple sample patients only contain structural features and medical imaging features, and the characteristic of interest corresponding to the prior knowledge is the seventh related characteristic, the multimodal assisted learning disease probability determination system can determine the model training auxiliary task as the second feature transformation to generate the auxiliary task. By performing the second feature transformation to generate the auxiliary task, the disease occurrence probability prediction model can more effectively combine the numerical information in the structured data and the visual pattern in the image data, and convert the image data into the numerical value in the structured data, so as to deeply understand the complex medical situation. For example, if the main task is to predict the probability of heart disease, the auxiliary task can be to combine the specific pattern of the electrocardiogram (ECG, a structured feature) with the features of the heart CT scan image (image feature) to identify early signs of heart disease.

[0163] It should be noted that if structural features, medical text features and medical image features are included at the same time, the auxiliary task is a combination of the aforementioned auxiliary tasks, so it will not be described in detail here.

[0164] In a possible implementation, the first related characteristic is to focus on the value of a specific test indicator, the second related characteristic is to focus on the changing trend of a specific test indicator, the third related characteristic is to focus on a preset part in the text, the fourth related characteristic is to focus on a preset part in the image result, the fifth related characteristic is to focus on whether the diagnosis and treatment conclusion is related to the specific test indicator, the sixth related characteristic is to focus on whether the medical imaging feature can obtain a specific conclusion description, and the seventh related characteristic is to focus on the size or area of ​​the medical imaging feature.

[0165] Specifically, in the present technical solution, the first relevant characteristic is to focus on the values ​​of specific laboratory indicators. In model training, the auxiliary task can be to reconstruct the features of the values ​​of specific laboratory indicators, extract the most informative features from a large amount of data, and remove redundant and irrelevant information to facilitate the identification and interpretation of the direct impact of these values ​​on the patient's health status.

[0166] For the second related characteristic, which is to pay attention to the changing trends of specific laboratory indicators, during model training, the auxiliary task can be to predict the future changing trends of these indicators to identify potential health risks.

[0167] For the third related feature, which is to focus on the preset parts in the text, in model training, the auxiliary task can be to use natural language processing technology to efficiently extract and identify the specific location and context of these key information to provide deeper insights into the patient's health status.

[0168] For the fourth related characteristic, focusing on the preset parts of the image results, in model training, the auxiliary task can be to use image processing technology to identify and interpret these image features.

[0169] The fifth related characteristic is to focus on whether the diagnosis and treatment conclusion is related to specific laboratory indicators. In model training, the auxiliary task can be to explore and analyze the relationship and influence between the two types of data: diagnosis and treatment conclusion and specific laboratory indicators.

[0170] The sixth related characteristic is to focus on whether the medical image features can obtain specific conclusion descriptions. In model training, auxiliary tasks can combine imaging and text descriptions to analyze the relationship and impact between the two types of data.

[0171] For the seventh related characteristic, which is the size or area of ​​the medical imaging features, in model training, auxiliary tasks can be generated for feature transformation to quantify these medical imaging features and evaluate their impact on disease status or progression.

[0172] In a possible implementation, the auxiliary task data set X includes a first reconstructed feature x1', a second reconstructed feature x3', and a medical text feature x2; the auxiliary task label set Y includes a first new label, a second new label, a third new label, a fourth new label, a fifth new label, a sixth new label, a seventh new label, and an eighth new label.

[0173] Specifically, in this technical solution, the multimodal assisted learning disease probability determination system aggregates the first reconstructed feature x1' and the second reconstructed feature x3', and the medical text feature x2 to form a comprehensive data set for further analysis and model training. By combining different types of reconstructed features, this auxiliary task data set can provide a more comprehensive perspective for the disease probability prediction model, thereby helping to capture subtle differences and patterns in complex medical data, and can improve the accuracy and generalization ability of the disease probability prediction model when processing diversified medical data, especially in complex medical scenarios that require the integration of multiple data sources to make accurate predictions.

[0174] The disease occurrence probability determination system of multimodal assisted learning aggregates the first new label, the second new label, the third new label, the fourth new label, the fifth new label, the sixth new label, the seventh new label, and the eighth new label to form a comprehensive label set, thereby providing a diversified training target for the model, thereby helping the model to better learn and understand complex medical data. By combining different types of new labels to provide multiple angles to understand and process medical data, the disease occurrence probability prediction model training is not limited to the labels of the main task, but covers a wider range of features and patterns. And by comprehensively using labels from different auxiliary tasks, the model can have better adaptability and accuracy when facing complex and changing medical scenarios.

[0175] Reference Figure 4 , Figure 4 This is the fourth flow chart of a method for determining the probability of disease occurrence based on multimodal assisted learning disclosed in an embodiment of the present application. Step S24 specifically includes steps S241-S248:

[0176] Step S241: reconstruct the auxiliary task based on the first feature to obtain a first reconstructed feature x1', a first newly created label y1 and a second newly created label y2, and obtain a first loss value based on the first reconstructed feature x1', the first newly created label y1 and the second newly created label y2.

[0177] Specifically, in the present technical scheme, the multimodal assisted learning disease occurrence probability determination system obtains the first reconstructed feature x1', the first newly created label y1, the second newly created label y2 and the first loss value based on the first feature reconstruction auxiliary task, that is, by reconstructing the structured features to analyze the characteristic test index values, extract deeper information from the structured features, and evaluate the effect of feature reconstruction through the first newly created label y1, the second newly created label y2 and the first loss value.

[0178] In subsequent embodiments, the specific method of reconstructing the auxiliary task based on the first feature to obtain the first reconstructed feature x1', the first new label y1 and the second new label y2, and obtaining the first loss value based on the first reconstructed feature x1', the first new label y1 and the second new label y2 will be described in detail.

[0179] Step S242: Based on the feature trend prediction auxiliary task, a third new label y3 is obtained, and a second loss value is obtained based on the third new label y3.

[0180] Specifically, in the present technical solution, the multimodal assisted learning disease occurrence probability determination system obtains the third new label y3 and the second loss value based on the feature trend prediction auxiliary task, that is, analyzes the changing trend of specific laboratory indicators, and evaluates the accuracy of the feature trend prediction through the third new label y3 and the second loss value.

[0181] In subsequent embodiments, a specific method for obtaining the third newly created label y3 based on the feature trend prediction auxiliary task and obtaining the second loss value based on the third newly created label y3 will be described in detail.

[0182] Step S243: Based on the feature position recognition auxiliary task, a fourth new label y4 is obtained, and a third loss value is obtained based on the fourth new label y4.

[0183] Specifically, in the present technical solution, the multimodal assisted learning disease occurrence probability determination system obtains the fourth new label y4 and the third loss value based on the feature position recognition auxiliary task, that is, by identifying the specific position and context of key information points in the medical text data, and evaluating the accuracy of feature position recognition through the fourth new label y4 and the third loss value.

[0184] In subsequent embodiments, a specific method for obtaining the fourth new label y4 based on the feature position recognition auxiliary task and obtaining the third loss value based on the fourth new label y4 will be described in detail.

[0185] Step S244: reconstruct the auxiliary task based on the second feature to obtain a second reconstructed feature x3' and a fifth newly created label y5, and obtain a fourth loss value based on the second reconstructed feature x3' and the fifth newly created label y5.

[0186] Specifically, in the present technical solution, the multimodal assisted learning disease occurrence probability determination system obtains the second reconstructed feature x3', the fifth new label y5 and the fourth loss value based on the second feature reconstruction auxiliary task, that is, by reconstructing the features of medical imaging data, identifying specific patterns or structures in medical images, and evaluating the effect of such reconstruction through the fifth new label y5 and the fourth loss value.

[0187] In subsequent embodiments, a specific method for reconstructing the auxiliary task based on the second feature, obtaining the second reconstructed feature x3' and the fifth newly created label y5, and obtaining the fourth loss value based on the second reconstructed feature x3' and the fifth newly created label y5 will be described in detail.

[0188] Step S245: Determine the auxiliary task based on the feature correlation, obtain the sixth new label y6, and obtain the fifth loss value based on the sixth new label y6.

[0189] Specifically, in the present technical scheme, the multimodal assisted learning disease occurrence probability determination system obtains the sixth new label y6 and the fifth loss value based on the feature correlation judgment auxiliary task, that is, by combining the relationship and correlation between structured features and medical text data to reveal the implicit connection between the data, improve the model's understanding ability when integrating multi-source data, and evaluate the effect of feature correlation judgment through the sixth new label y6 and the fifth loss value.

[0190] In subsequent embodiments, a specific method for determining the auxiliary task based on feature correlation, obtaining the sixth new label y6, and obtaining the fifth loss value based on the sixth new label y6 will be described in detail.

[0191] Step S246: Generate an auxiliary task based on the first feature transformation, obtain a seventh new label y7, and obtain a sixth loss value based on the seventh new label y7.

[0192] Specifically, in the present technical scheme, the multimodal assisted learning disease occurrence probability determination system generates an auxiliary task based on the first feature transformation to obtain the seventh new label y7 and the sixth loss value, that is, by combining the visual patterns in the image data with the descriptive information in the text data, the features in the medical image data are generated into text data to create a new and richer feature representation, and the effect of this feature change is evaluated by the seventh new label y7 and the sixth loss value.

[0193] In subsequent embodiments, a specific method for generating an auxiliary task based on the first feature transformation, obtaining the seventh new label y7, and obtaining the sixth loss value based on the seventh new label y7 will be described in detail.

[0194] Step S247: Generate an auxiliary task based on the second feature transformation, obtain the eighth new label y8, and obtain the seventh loss value based on the eighth new label y8.

[0195] Specifically, in the present technical solution, the multimodal assisted learning disease occurrence probability determination system generates an auxiliary task based on the second feature transformation to obtain the eighth new label y8 and the seventh loss value, that is, by quantifying the relationship between image features and structured data, and evaluating the effectiveness of this data fusion through the eighth new label y8 and the seventh loss value.

[0196] In subsequent embodiments, a specific method for generating an auxiliary task based on the second feature transformation, obtaining the eighth new label y8, and obtaining the seventh loss value based on the eighth new label y8 will be described in detail.

[0197] Step S248: Add the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, the sixth loss value and the seventh loss value to obtain the auxiliary task loss value.

[0198] Specifically, in the present technical solution, the disease occurrence probability determination system of multimodal assisted learning adds the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, the sixth loss value, and the seventh loss value to obtain the auxiliary task loss value. The auxiliary task loss value helps ensure that the model does not over-optimize a specific task and ignore other tasks, thereby improving the model's generalization ability for unseen data.

[0199] In a possible implementation, step S241 specifically includes the following steps:

[0200] Based on the first feature reconstruction auxiliary task, the specific test index x in all structured features x1 1' Add random noise disturbance to obtain the first reconstructed feature x1', as well as the first new label y1 and the second new label y2 corresponding to the first reconstructed feature x1'; where y1∈{0,1}, 0 represents the feature without adding random noise disturbance, and 1 represents the feature with adding random noise disturbance; y2 is a continuous value, indicating the specific test index x before adding random noise disturbance to the feature 1' The numerical value of .

[0201] Specifically, in this technical solution, the multimodal learning-assisted disease probability determination system determines the specific test index x in all structured features x1. 1'Add random noise disturbance to generate the first reconstructed feature x1'. The first reconstructed feature includes the feature without adding random noise disturbance and the feature with adding random noise disturbance. The addition of this random noise disturbance is intended to test the sensitivity and adaptability of the model to feature changes, which helps to improve the depth of the model's understanding of the data.

[0202] The first manually marked new label y1 is used to indicate whether random noise disturbance is added to the feature, where 0 represents no noise disturbance and 1 represents noise disturbance is added; the second newly created label y2 is used to indicate the actual value of the feature before random noise disturbance is added.

[0203] The function loss value of the first newly created label y1 is calculated based on the cross entropy loss function to obtain a first reconstruction loss value.

[0204] Specifically, in this technical solution, the disease occurrence probability determination system of multimodal assisted learning calculates the function loss value of the first new label y1 based on the cross entropy loss function to obtain the first reconstruction loss value. Among them, the cross entropy loss is suitable for evaluating binary classification problems, and is used here to measure the ability of the model to identify noise disturbance features. It should be noted that the first reconstruction loss value can be calculated in the following way:

[0205] ,

[0206] Among them, x represents the category, p represents the label value, and q represents the model prediction probability.

[0207] The function loss value of the second newly created label y2 is calculated based on the mean square error loss function to obtain a second reconstruction loss value.

[0208] Specifically, in this technical solution, the disease occurrence probability determination system of multimodal assisted learning calculates the function loss value of the second newly created label y2 based on the mean square error loss function to obtain the second reconstruction loss value. Among them, the mean square error loss is applicable to the prediction of continuous values ​​and is used to evaluate the accuracy of the model's prediction of the value before the noise disturbance. It should be noted that the second reconstruction loss value can be calculated in the following way:

[0209] ,

[0210] Among them, N represents the number of samples, y i Represents the tag value, Represents the predicted value.

[0211] The first reconstruction loss value is added to the second reconstruction loss value to obtain a first loss value.

[0212] Specifically, in the present technical solution, the disease occurrence probability determination system of multimodal assisted learning adds the above two loss values ​​to obtain an overall first loss value, which is used to measure the effect of the entire first feature reconstruction auxiliary task.

[0213] In a possible implementation, step S242 specifically includes the following steps:

[0214] Based on the feature trend prediction auxiliary task, the time series data in the structured feature x1 is determined; the time series data includes n time steps, where n is a positive integer greater than 1.

[0215] Specifically, in the present technical solution, the multimodal assisted learning disease occurrence probability determination system identifies time series data from the structured feature x1, and the time series data includes n time steps, where n is a positive integer greater than 1. Time series data can show the changes of specific laboratory indicators over time.

[0216] Based on the specific assay indicators corresponding to the first m time steps among the n time steps, the specific assay indicators corresponding to the m+1th time step are predicted.

[0217] Specifically, in the present technical solution, the multimodal assisted learning disease occurrence probability determination system predicts the specific laboratory indicator value at the m+1th time step based on the data of the first m time steps, so as to understand and predict the changing trend of the specific laboratory indicator over time.

[0218] The specific assay indicator corresponding to the m+1th time step is used as the third newly created label y3.

[0219] Specifically, in the present technical solution, the disease occurrence probability determination system of multimodal assisted learning defines the specific test index value at the m+1th time step as the third newly created label y3. The third newly created label y3 reflects the actual observation value and is used for subsequent prediction accuracy evaluation.

[0220] The function loss value of the third newly created label y3 is calculated based on the mean square error loss function to obtain a second loss value.

[0221] Specifically, in this technical solution, the mean square error loss function is used to calculate the loss value between the predicted value and the third newly created label y3. Among them, the mean square error loss function is applicable to continuous numerical prediction problems, and the difference between the predicted value and the actual value is quantified to evaluate the accuracy of the model in the feature trend prediction auxiliary task. It should be noted that the calculation method here is the same as the method for calculating the second reconstruction loss value in the aforementioned embodiment, which has been explained, so it will not be repeated here.

[0222] In a possible implementation, step S243 specifically includes the following steps:

[0223] Based on the feature position recognition auxiliary task, a fourth new label y4 is constructed; wherein y4∈{0,1,2}, 0 represents a non-starting or ending position, 1 represents a starting position, and 2 represents an ending position.

[0224] Specifically, in this technical solution, the multimodal assisted learning disease occurrence probability determination system constructs a fourth newly created label y4 based on the feature position recognition auxiliary task, and the fourth newly created label y4 is used to represent the position of a specific information point in the text (such as symptom description, diagnostic information, etc.). The fourth newly created label y4 can take the value of {0,1,2}, where 0 represents a non-starting or ending position, 1 represents a starting position, and 2 represents an ending position. Through this label design, the disease occurrence probability prediction model can recognize and learn the precise position of information in the text.

[0225] The function loss value of the fourth newly created label y4 is calculated based on the cross entropy loss function to obtain a third loss value.

[0226] Specifically, in the present technical solution, the disease occurrence probability determination system of multimodal assisted learning calculates the function loss value of the fourth newly created label y4 based on the cross entropy loss function to obtain a third loss value. Among them, the cross entropy loss function is suitable for classification tasks, and is used here to evaluate the model's ability to correctly identify the location of specific information points in the text. This loss value can quantify the accuracy of the model in position recognition tasks. It should be noted that the method for calculating the third reconstruction loss value here is consistent with the method for calculating the first reconstruction loss value in the aforementioned embodiment, so no further details will be given here.

[0227] In a possible implementation, step S244 specifically includes the following steps:

[0228] Based on the second feature reconstruction auxiliary task, a preset proportion of pixels in the target area corresponding to the preset target area in the medical image feature x3 are masked to obtain the second reconstructed feature x3' and the pixel values ​​of the masked replaced part.

[0229] Specifically, in this technical solution, the multimodal assisted learning disease occurrence probability determination system selects pixels in a preset target area in the medical image feature x3, and performs a mask replacement operation on a certain proportion of pixels in the area. The image data obtained after processing is called the second reconstructed feature x3'. The mask replacement can replace the value of the selected pixel with a specific value or pattern to simulate the situation of missing or interference of information.

[0230] The pixel value of the mask replacement part is used as the fifth new label y5.

[0231] Specifically, in the present technical solution, the disease occurrence probability determination system of multimodal assisted learning defines the original pixel value of the mask replacement part as the fifth newly created label y5. The fifth newly created label y5 reflects the actual pixel value of the replaced part in the original image data.

[0232] The function loss value of the fifth new label y5 is calculated based on the mean square error loss function to obtain a fourth loss value.

[0233] Specifically, in the present technical solution, the disease occurrence probability determination system of multimodal assisted learning adopts the mean square error loss function to calculate the loss value between the predicted pixel value and the fifth newly created label y5. Among them, the mean square error loss function is suitable for the prediction of continuous numerical values, and is used here to evaluate the accuracy of the model in reconstructing the image area replaced by the mask. This loss value can quantify the accuracy of the model in restoring or understanding the image features in the second feature reconstruction auxiliary task. It should be noted that the method for calculating the fourth reconstruction loss value here is consistent with the method for calculating the second reconstruction loss value in the aforementioned embodiment, so no further details will be given here.

[0234] In a possible implementation, step S245 specifically includes the following steps:

[0235] Based on the auxiliary task of feature correlation judgment, the sixth new label y6 is constructed according to the list of diagnosis and treatment conclusions corresponding to the prior knowledge, where y6∈{0,1}, 0 represents non-relevant indicators, and 1 represents relevant indicators.

[0236] Specifically, in this technical solution, the multimodal assisted learning disease probability determination system constructs the sixth new label y6 based on prior knowledge and a list of diagnosis and treatment conclusions. The sixth new label y6 is used to identify whether the structured feature is related to a specific diagnosis and treatment conclusion in the medical text data. The value range of the sixth new label y6 is {0,1}, where 0 represents an irrelevant indicator and 1 represents a relevant indicator. Through this label design, the disease probability prediction model can learn to identify which laboratory indicators are directly related to specific diagnosis and treatment conclusions.

[0237] The function loss value of the sixth new label y6 is calculated based on the cross entropy loss function to obtain the fifth loss value.

[0238] Specifically, in this technical solution, the cross entropy loss function is used to calculate the loss value between the predicted correlation label and the actual correlation label. Among them, the cross entropy loss function is suitable for binary classification problems, and is used here to evaluate the accuracy of the model in judging the correlation between specific laboratory indicators and diagnosis and treatment conclusions. It should be noted that the method for calculating the fifth reconstruction loss value here is consistent with the method for calculating the first reconstruction loss value in the aforementioned embodiment, so no further details will be given here.

[0239] In a possible implementation, step S246 specifically includes the following steps:

[0240] Based on the first feature transformation, an auxiliary task is generated, and the medical text feature x2 is used as the seventh new label y7.

[0241] Specifically, in this technical solution, the multimodal assisted learning disease occurrence probability determination system uses the medical text feature x2 as the seventh new label y7.

[0242] The function loss value of the seventh new label y7 is calculated based on the cross entropy loss function to obtain the sixth loss value.

[0243] Specifically, in the present technical solution, the disease occurrence probability determination system of multimodal assisted learning calculates the function loss value of the seventh new label y7 based on the cross entropy loss function to obtain the sixth loss value. The loss value is calculated by the cross entropy loss function to evaluate the effect of the disease occurrence probability prediction model in performing the first feature transformation to generate the auxiliary task. It should be noted that the method for calculating the sixth reconstruction loss value here is consistent with the method for calculating the first reconstruction loss value in the aforementioned embodiment, so no further details will be given here.

[0244] In a possible implementation, step S247 specifically includes the following steps:

[0245] Based on the second feature transformation, an auxiliary task is generated, and the structured feature x1 is used as the eighth new label y8.

[0246] Specifically, in the present technical solution, the multimodal assisted learning disease occurrence probability determination system uses the structured feature x1 as the eighth new label y8.

[0247] The function loss value of the eighth new label y8 is calculated based on the mean square error loss function to obtain the seventh loss value.

[0248] Specifically, in the present technical solution, the disease occurrence probability determination system of multimodal assisted learning calculates the function loss value of the eighth new label y8 based on the mean square error loss function to obtain the seventh loss value. Among them, the mean square error loss function is used for regression problems, which can measure the average square difference between the predicted value and the actual value. The seventh loss value is used to evaluate the effect of the disease occurrence probability prediction model in performing the second feature transformation to generate the auxiliary task. It should be noted that the method for calculating the seventh reconstruction loss value here is consistent with the method for calculating the second reconstruction loss value in the aforementioned embodiment, so it will not be repeated here.

[0249] In general, by introducing noise and performing data masking in the training of the disease probability prediction model, the robustness of the disease probability prediction model is improved. By using multiple loss functions such as cross entropy loss and mean square error loss, the disease probability prediction model can be effectively learned on different tasks. This improves the accuracy of predicting the disease probability using the disease probability prediction model.

[0250] Finally, in combination with the above embodiment, the output of warning information to the physician terminal when the first disease occurrence probability value is greater than a preset disease occurrence threshold value in step S14 is further illustrated.

[0251] The following will take acute pulmonary embolism (PTE) as an example to give two PTE risk assessment cases. The patients contain three modal data: structured, text, and image. The structured features include age and laboratory information, the medical text features include medical history and ultrasound impressions, and the medical image features include chest X-ray and electrocardiogram.

[0252] Patient A: 72 years old, D-dimer is 5.7mg / L, high-sensitivity C-reactive protein is 12mg / L; Past history text content: The patient suffered craniocerebral trauma at the age of 7 and had his first epileptic seizure at the age of 18; Ultrasound impression text: Biatrial and right ventricular enlargement, large amount of mitral regurgitation, and a small amount of pericardial effusion; Chest X-ray image; Electrocardiogram image. The disease occurrence probability prediction model predicts that the probability of PTE in this patient is 0.9. Assuming that the probability threshold is set to 0.5, the multimodal assisted learning disease occurrence probability determination system determines that the patient is a high-risk patient for PTE, and issues a high-risk PTE warning to the clinic.

[0253] Patient B: 28 years old, D-dimer is 0.5mg / L, high-sensitivity C-reactive protein is 5mg / L; Past history text content: denial of hypertension, diabetes, coronary heart disease, and cerebrovascular disease history; Ultrasound impression text: no obvious abnormalities in cardiac structure and blood flow; Chest X-ray image; Electrocardiogram image. The disease occurrence probability prediction model predicts that the probability of PTE in this patient is 0.2. Assuming that the probability threshold is set to 0.5, the multimodal assisted learning disease occurrence probability determination system determines that the patient is a low-risk patient for PTE and does not issue a high-risk PTE warning to the clinic.

[0254] The present application also provides a disease occurrence probability determination system based on multimodal assisted learning, referring to Figure 5, which shows a structural schematic diagram of a disease occurrence probability determination system based on multimodal assisted learning provided by an embodiment of the present application. The system includes: a first acquisition module 1, a data preprocessing module 2, a processing module 3 and an early warning module 4; the first acquisition module 1 is used to acquire the patient's real-time medical data, and the real-time medical data includes one or more of structured data features, medical text features and medical image features; the data preprocessing module 2 is used to preprocess the real-time medical data to obtain the medical data to be analyzed; the processing module 3 is used to input the medical data to be analyzed into the trained disease occurrence probability prediction model to obtain the first disease occurrence probability value, and the disease occurrence probability prediction model is trained based on the model training data set and the model training auxiliary task, the model training data set includes the preprocessed historical medical data of multiple sample patients and the patient labels of multiple sample patients, and the model training auxiliary task is constructed by the feature type contained in the preprocessed historical medical data of multiple sample patients and the prior knowledge corresponding to the sample patients, the prior knowledge is obtained based on the patient label, and the historical medical data includes one or more of structured data features, medical text features and medical image features; the early warning module 4 is used to output the early warning prompt information to the physician terminal when the first disease occurrence probability value is greater than the preset disease occurrence threshold.

[0255] In a possible implementation, the system further includes: a second acquisition module 5, a prediction module 6, a model building module 7 and an auxiliary task building module 8; the second acquisition module 5 is used to acquire historical medical data of multiple sample patients and patient labels of multiple sample patients; the prediction module 6 is used to determine the prediction endpoint based on the patient label; the second acquisition module 5 is also used to acquire prior knowledge corresponding to the prediction endpoint; the second acquisition module 5 is also used to build a basic data set based on historical medical data and patient labels; the data preprocessing module 2 is also used to acquire a model training data set; the model training data set includes preprocessed historical medical data x of multiple sample patients and patient labels y of multiple sample patients; wherein the historical medical data x of multiple sample patients includes one or more of structured features x1, medical text features x2 and medical imaging features x3; the patient labels y∈{0,1} of multiple sample patients, 0 represents no disease , 1 represents sick; the model building module 7 is also used to input the model training data set into the initial disease occurrence probability prediction model to obtain the main task loss value; the auxiliary task building module 8 is used to determine the model training auxiliary task based on the feature types and the attention characteristics corresponding to the prior knowledge contained only in the preprocessed historical medical data x of multiple sample patients; the auxiliary task building module 8 is also used to obtain the auxiliary task data set X and the auxiliary task label set Y according to the model training auxiliary task, and obtain the auxiliary task loss value based on the auxiliary task data set X and the auxiliary task label set Y; the model building module 7 is also used to add the main task loss value and the auxiliary task loss value to obtain the total loss value; the model building module 7 is also used to update the model parameters of the initial disease occurrence probability prediction model based on the total loss value using the back propagation algorithm, and train the initial disease occurrence probability prediction model for a preset number of rounds to obtain a trained disease occurrence probability prediction model.

[0256] It should be noted that the second acquisition module 5, prediction module 6, model construction module 7 and auxiliary task construction module 8 in the above embodiments are all part of the disease occurrence probability determination system based on multimodal assisted learning in this application, which can be applied before the processing module 3 to construct a preset disease occurrence probability prediction model.

[0257] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0258] The present application also discloses an electronic device. Figure 6 , Figure 6 The electronic device 600 may include: at least one processor 601 , at least one network interface 604 , a user interface 603 , a memory 605 , and at least one communication bus 602 .

[0259] The communication bus 602 is used to realize the connection and communication between these components.

[0260] The user interface 603 may include a display screen (Display) and a camera (Camera). The optional user interface 603 may also include a standard wired interface and a wireless interface.

[0261] The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0262] Among them, the processor 601 may include one or more processing cores. The processor 601 uses various interfaces and lines to connect the various parts of the entire multimodal assisted learning disease probability determination system, and executes various functions and processes data of the multimodal assisted learning disease probability determination system by running or executing instructions, programs, code sets or instruction sets stored in the memory 605, and calling the data stored in the memory 605. Optionally, the processor 601 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 601 can integrate one or more combinations of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 601, and it can be implemented by a single chip.

[0263] Among them, the memory 605 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 605 includes a non-transitory computer-readable storage medium. The memory 605 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 605 may optionally also be at least one storage device located away from the aforementioned processor 601. Refer to Figure 6 , the memory 605 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program.

[0264] exist Figure 6 In the electronic device 600 shown, the user interface 603 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 601 can be used to call an application stored in the memory 605, and when executed by one or more processors 601, the electronic device 600 executes one or more methods in the above-mentioned embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0265] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0266] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0267] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0268] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0269] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application can be essentially or partly or all or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a memory, including several instructions for a computer device (which can be a personal computer, a disease occurrence probability determination system for multimodal assisted learning, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks, or optical disks.

[0270] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0271] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for determining the probability of disease occurrence based on multimodal assisted learning, characterized in that: The method comprises: Acquire real-time medical data of the patient, wherein the real-time medical data includes one or more of structured data features, medical text features, and medical image features; Preprocessing the real-time medical data to obtain medical data to be analyzed; The medical data to be analyzed is input into a trained disease occurrence probability prediction model to obtain a first disease occurrence probability value, wherein the disease occurrence probability prediction model is trained based on a model training data set and a model training auxiliary task, wherein the model training data set includes preprocessed historical medical data of multiple sample patients and patient labels of multiple sample patients, and the model training auxiliary task is constructed by the feature types contained in the preprocessed historical medical data of multiple sample patients and the prior knowledge corresponding to the sample patients, wherein the prior knowledge is obtained based on the patient labels, and the historical medical data includes one or more of structured data features, medical text features, and medical imaging features; the disease occurrence probability prediction model is trained based on the model training data set and the model training auxiliary task, and specifically includes: Acquire the model training data set; the model training data set includes preprocessed historical medical data x of multiple sample patients and patient labels y of multiple sample patients; wherein the preprocessed historical medical data x of multiple sample patients includes one or more of structured features x1, medical text features x2, and medical image features x3; the patient labels y∈{0,1} of the multiple sample patients, 0 represents no disease, and 1 represents disease; Inputting the model training data set into the initial disease occurrence probability prediction model to obtain the main task loss value; Determine a model training auxiliary task based on the feature types contained in the preprocessed historical medical data x of the plurality of sample patients and the focus characteristics corresponding to the prior knowledge; Training the auxiliary task according to the model obtains an auxiliary task data set X and an auxiliary task label set Y, and obtains an auxiliary task loss value based on the auxiliary task data set X and the auxiliary task label set Y; Adding the main task loss value and the auxiliary task loss value to obtain a total loss value; Based on the total loss value, the model parameters of the initial disease occurrence probability prediction model are updated using a back propagation algorithm, and the initial disease occurrence probability prediction model is trained for a preset number of rounds to obtain the trained disease occurrence probability prediction model; When the first disease occurrence probability value is greater than a preset disease occurrence threshold, an early warning prompt message is output to the physician terminal.

2. The method according to claim 1, characterized in that Before acquiring the real-time medical data of the patient, the method further includes: Acquire historical medical data of multiple sample patients and patient labels of multiple sample patients; Based on the patient signature, determining a predicted endpoint; Acquiring prior knowledge corresponding to the predicted endpoint; Constructing a basic data set based on the historical medical data and the patient labels; The basic data set is preprocessed to obtain a model training data set.

3. The method according to claim 1, characterized in that The inputting of the model training data set into the initial disease occurrence probability prediction model to obtain the main task loss value specifically includes: Inputting the model training data set into an initial disease occurrence probability prediction model; the initial disease occurrence probability prediction model includes a bidirectional Transformer structure model and a unidirectional Transformer structure model; The bidirectional Transformer structure model is selected as an encoder to encode the non-time series data in the structured feature x1, the medical text feature x2, and the medical image feature x3, and the unidirectional Transformer structure model is selected as an encoder to encode the time series data in the structured feature x1, the medical text feature x2, and the medical image feature x3, and the data encoded by the bidirectional Transformer structure model and the unidirectional Transformer structure model are combined to output the second disease occurrence probability value; Obtaining a cross entropy loss value based on the second disease occurrence probability value and the patient labels y of the multiple sample patients; The cross entropy loss value is used as the main task loss value.

4. The method according to claim 1, characterized in that: The attention characteristics corresponding to the prior knowledge include a first related characteristic, a second related characteristic, a third related characteristic, a fourth related characteristic, a fifth related characteristic, a sixth related characteristic, and a seventh related characteristic; the determining of the model training auxiliary task based on the feature types only contained in the preprocessed historical medical data x of the plurality of sample patients and the attention characteristics corresponding to the prior knowledge specifically includes: When the preprocessed historical medical data of the plurality of sample patients only contain the structural feature x1, and the characteristic of interest corresponding to the prior knowledge is the first relevant characteristic, determining the model training auxiliary task to be the first feature reconstruction auxiliary task; When the preprocessed historical medical data of the plurality of sample patients only contain the structural feature x1, and the characteristic of interest corresponding to the prior knowledge is the second related characteristic, determining that the model training auxiliary task is a feature trend prediction auxiliary task; When the preprocessed historical medical data of the plurality of sample patients only contain the feature of the medical text feature x2, and the prior knowledge corresponds to the third related characteristic, determining the model training auxiliary task as a feature position recognition auxiliary task; When the preprocessed historical medical data of the plurality of sample patients only contain the medical image feature x3, and the characteristic of interest corresponding to the prior knowledge is the fourth related characteristic, determining that the model training auxiliary task is the second feature reconstruction auxiliary task; When the preprocessed historical medical data of the plurality of sample patients only contain the structural feature x1 and the medical text feature x2, and the characteristic of interest corresponding to the prior knowledge is the fifth related characteristic, determining that the model training auxiliary task is a characteristic correlation judgment auxiliary task; When the preprocessed historical medical data of the plurality of sample patients only contain the medical text feature x2 and the medical image feature x3, and the characteristic of interest corresponding to the prior knowledge is the sixth related characteristic, determining that the model training auxiliary task is the first feature transformation generation auxiliary task; When the preprocessed historical medical data of the multiple sample patients only contain the structural feature x1 and the medical image feature x3, and the focus characteristic corresponding to the prior knowledge is the seventh related characteristic, the model training auxiliary task is determined to be the second feature transformation generation auxiliary task.

5. The method according to claim 4, characterized in that The first related characteristic is to focus on the value of a specific test indicator, the second related characteristic is to focus on the changing trend of a specific test indicator, the third related characteristic is to focus on a preset part in the text, the fourth related characteristic is to focus on a preset part in the image result, the fifth related characteristic is to focus on whether the diagnosis and treatment conclusion is related to the specific test indicator, the sixth related characteristic is to focus on whether the medical imaging feature can obtain a specific conclusion description, and the seventh related characteristic is to focus on the size or area of ​​the medical imaging feature.

6. The method according to claim 4, characterized in that The auxiliary task dataset X includes a first reconstruction feature x ′ 1. Second reconstruction feature x ′ 3 and the medical text feature x2; the auxiliary task label set Y includes a first new label, a second new label, a third new label, a fourth new label, a fifth new label, a sixth new label, a seventh new label and an eighth new label; the auxiliary task data set X and the auxiliary task label set Y are obtained according to the model training auxiliary task, and the auxiliary task loss value is obtained based on the auxiliary task data set Y and the auxiliary task label set Y, specifically including: reconstructing the auxiliary task based on the first feature, obtaining the first reconstructed feature x ′ 1. The first newly created label y1 and the second newly created label y2 are based on the first reconstructed feature x ′ 1. The first newly created label y1 and the second newly created label y2 obtain a first loss value; Based on the feature trend prediction auxiliary task, obtaining the third newly created label y3, and obtaining a second loss value based on the third newly created label y3; Based on the feature position recognition auxiliary task, obtaining the fourth new label y4, and obtaining a third loss value based on the fourth new label y4; Based on the second feature, the auxiliary task is reconstructed to obtain the second reconstructed feature x ′ 3 and the fifth new label y5, and based on the second reconstructed feature x ′ 3 and the fifth newly created label y5 to obtain a fourth loss value; Determine the auxiliary task based on the feature correlation, obtain the sixth new label y6, and obtain the fifth loss value based on the sixth new label y6; Generate an auxiliary task based on the first feature transformation to obtain the seventh new label y7, and obtain a sixth loss value based on the seventh new label y7; Generate an auxiliary task based on the second feature transformation to obtain the eighth new label y8, and obtain a seventh loss value based on the eighth new label y8; The first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value and the sixth loss value are added to obtain the auxiliary task loss value.

7. The method according to claim 6, characterized in that The auxiliary task is reconstructed based on the first feature to obtain the first reconstructed feature x ′ 1. The first newly created label y1 and the second newly created label y2 are based on the first reconstructed feature x ′ 1. The first newly created label y1 and the second newly created label y2 obtain a first loss value, specifically including: Based on the first feature reconstruction auxiliary task, the specific test index x in all the structured features x1 is 1’ Add random noise disturbance to get the first reconstruction feature x ′ 1, and the first reconstruction feature x ′ 1 corresponds to the first new label y1 and the second new label y2; where y1∈{0,1}, 0 represents the feature without adding random noise disturbance, and 1 represents the feature with adding random noise disturbance; y2 is a continuous value, indicating the value of the feature before adding random noise disturbance; Calculate the function loss value of the first newly created label y1 based on the cross entropy loss function to obtain a first reconstruction loss value; Calculate the function loss value of the second newly created label y2 based on the mean square error loss function to obtain a second reconstruction loss value; The first reconstruction loss value is added to the second reconstruction loss value to obtain a first loss value.

8. The method according to claim 6, wherein obtaining the third new label y3 based on the feature trend prediction auxiliary task, and obtaining the second loss value based on the third new label y3, specifically comprises: Based on the feature trend prediction auxiliary task, determine the time series data in the structured feature x1; the time series data includes n time steps, where n is a positive integer greater than 1; Based on the specific assay indicators corresponding to the first m time steps among the n time steps, predict the specific assay indicators corresponding to the m+1th time step; The specific test index corresponding to the m+1th time step is used as the third newly created label y3; The function loss value of the third newly created label y3 is calculated based on the mean square error loss function to obtain a second loss value.

9. The method according to claim 6, characterized in that The step of obtaining the fourth new label y4 based on the feature position recognition auxiliary task, and obtaining the third loss value based on the fourth new label y4, specifically includes: Based on the feature position recognition auxiliary task, construct the fourth new label y4; wherein y4∈{0,1,2}, 0 represents a non-starting or ending position, 1 represents a starting position, and 2 represents an ending position; The function loss value of the fourth new label y4 is calculated based on the cross entropy loss function to obtain the third loss value.

10. The method according to claim 6, characterized in that The auxiliary task is reconstructed based on the second feature to obtain the second reconstructed feature x ′ 3 and the fifth new label y5, and based on the second reconstructed feature x ′ 3 and the fifth newly created label y5 to obtain a fourth loss value, specifically including: Based on the second feature reconstruction auxiliary task, a preset proportion of pixels in the target area corresponding to the preset target area in the medical image feature x3 are masked to obtain a second reconstructed feature x ′ 3 and the pixel value of the mask replacement part; Using the pixel value of the mask replacement part as the fifth newly created label y5; The function loss value of the fifth new label y5 is calculated based on the mean square error loss function to obtain the fourth loss value.

11. The method according to claim 6, characterized in that The determining the auxiliary task based on the feature correlation to obtain the sixth new label y6, and obtaining the fifth loss value based on the sixth new label y6, specifically includes: Based on the feature correlation judgment auxiliary task, and according to the diagnosis and treatment conclusion list corresponding to the prior knowledge, the sixth new label y6 is constructed, where y6∈{0,1}, 0 represents a non-relevant indicator, and 1 represents a relevant indicator; The function loss value of the sixth new label y6 is calculated based on the cross entropy loss function to obtain the fifth loss value.

12. The method according to claim 6, characterized in that The generating of the auxiliary task based on the first feature transformation to obtain the seventh new label y7, and obtaining the sixth loss value based on the seventh new label y7 specifically includes: Using the medical text feature x2 as the seventh newly created label y7; The function loss value of the seventh new label y7 is calculated based on the cross entropy loss function to obtain the sixth loss value.

13. The method according to claim 6, characterized in that The generating of the auxiliary task based on the second feature transformation to obtain the eighth new label y8, and obtaining the seventh loss value based on the eighth new label y8, specifically includes: Using the structured feature x1 as the eighth newly created label y8; The function loss value of the sixth new label y8 is calculated based on the mean square error loss function to obtain the seventh loss value.

14. A disease occurrence probability determination system based on multimodal assisted learning, used to implement the method described in any one of claims 1 to 13, characterized in that: The system comprises: a first acquisition module, a data preprocessing module, a processing module and an early warning module; The first acquisition module is used to acquire real-time medical data of the patient, wherein the real-time medical data includes one or more of structured data features, medical text features, and medical image features; The data preprocessing module is used to preprocess the real-time medical data to obtain medical data to be analyzed; The processing module is used to input the medical data to be analyzed into a trained disease occurrence probability prediction model to obtain a first disease occurrence probability value, wherein the disease occurrence probability prediction model is trained based on a model training data set and a model training auxiliary task, wherein the model training data set includes pre-processed historical medical data of multiple sample patients and patient labels of multiple sample patients, wherein the model training auxiliary task is constructed by the feature types contained in the pre-processed historical medical data of multiple sample patients and the prior knowledge corresponding to the sample patients, wherein the prior knowledge is obtained based on the patient labels, and wherein the historical medical data includes one or more of structured data features, medical text features, and medical image features; The early warning module is used to output early warning information to the physician terminal when the first disease occurrence probability value is greater than a preset disease occurrence threshold.

15. The system according to claim 14, characterized in that: The system further comprises: a second acquisition module, a prediction module, a model construction module and an auxiliary task construction module; The second acquisition module is used to acquire historical medical data of multiple sample patients and patient labels of multiple sample patients; The prediction module is used to determine a predicted endpoint based on the patient label; The second acquisition module is further used to acquire prior knowledge corresponding to the predicted endpoint; The model building module is used to build a basic data set based on the historical medical data and the patient labels; The preprocessing module is used to obtain the model training data set; the model training data set includes preprocessed historical medical data x of multiple sample patients and patient labels y of multiple sample patients; wherein the historical medical data x of the multiple sample patients includes one or more of structured features x1, medical text features x2 and medical image features x3; the patient labels y∈{0,1} of the multiple sample patients, 0 represents no disease, 1 represents disease; The model building module is further used to input the model training data set into the initial disease occurrence probability prediction model to obtain the main task loss value; The auxiliary task construction module is used to determine the model training auxiliary task based on the feature types only contained in the preprocessed historical medical data x of the plurality of sample patients and the focus characteristics corresponding to the prior knowledge; The auxiliary task construction module is further used to obtain an auxiliary task data set X and an auxiliary task label set Y according to the model training auxiliary task, and obtain an auxiliary task loss value based on the auxiliary task data set X and the auxiliary task label set Y; The model building module is also used to add the main task loss value and the auxiliary task loss value to obtain a total loss value; the model building module is also used to update the model parameters of the initial disease occurrence probability prediction model based on the total loss value using a back propagation algorithm, and train the initial disease occurrence probability prediction model for a preset number of rounds to obtain the trained disease occurrence probability prediction model.

16. An electronic device, characterized in that: The electronic device (600) comprises a processor (601), a memory (605), a user interface (603) and a network interface (604), wherein the memory (605) is used to store instructions, the user interface (603) and the network interface (604) are used to communicate with other devices, and the processor (601) is used to execute the instructions stored in the memory (605) so that the electronic device (600) executes the method according to any one of claims 1 to 13.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method steps according to any one of claims 1 to 13 are performed.

Images